Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make config updating safe in all object stores #533

Open
paraseba opened this issue Jan 3, 2025 · 4 comments
Open

Make config updating safe in all object stores #533

paraseba opened this issue Jan 3, 2025 · 4 comments

Comments

@paraseba
Copy link
Collaborator

paraseba commented Jan 3, 2025

GCS doesn't support put-if-match using etags. We may need to use an approach similar to branches.

This is necessary for 1.0 because the config on-disk format would have to change significantly.

@mpiannucci
Copy link
Contributor

@paraseba paraseba moved this to Ready in Icechunk 1.0 Jan 12, 2025
@rabernat
Copy link
Contributor

This is confusing to me. It certainly looks like it is implemented

If-Match header Applicable for requests that retrieve data. Request proceeds if the ETag of the target resource matches the value used in the precondition. If the values don't match, the request fails with a 412 Precondition Failed response.

@mpiannucci
Copy link
Contributor

mpiannucci commented Jan 23, 2025

On gcs if-match is only avaialable for Get or Head requests. For puts, you have to use the generation match functionality which currently we don't have a mechanism for in icechunk. Object store supports it but you have to track the version (generation) instead of the ETag which we don't do currently

@rabernat
Copy link
Contributor

So AWS S3 and GCS have mutually incompatible ways of achieving conditional put: etag (S3) vs generation number (GCS). And it's up to the application (e.g. Icechunk) to figure out which to use when.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Ready
Development

No branches or pull requests

3 participants