Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Zarr 3.0 compatibility to memh5 #288

Open
ljgray opened this issue Jan 9, 2025 · 2 comments
Open

Add Zarr 3.0 compatibility to memh5 #288

ljgray opened this issue Jan 9, 2025 · 2 comments
Assignees

Comments

@ljgray
Copy link
Contributor

ljgray commented Jan 9, 2025

Zarr version 3.0 was released, with quite a few breaking changes. See here. We should update memh5 to be compatible with these changes.

@ljgray ljgray self-assigned this Jan 13, 2025
@ljgray
Copy link
Contributor Author

ljgray commented Jan 23, 2025

@ketiltrout I've been looking into this and I think it's going to be a bit tricky, for a few reasons.

  1. It seems like zarr is deprecating compatibility with basic h5py methods, and will remove entirely in 3.1.0. This includes things like Group.create_dataset being renamed to Group.create_array, etc... We rely on this compatibility in various parts of memh5, so supporting this is going to require some thought.
  2. The documentation for Zarr 3, when compared to Zarr 2, seems pretty incomplete and overly simple. This, combined with the fact that they've made their code fully asynchronous, means that it's a huge pain to figure out all the subtle changes that they've made.
  3. There are some random things that were supported in Zarr 2 that aren't in Zarr 3, such as opening a Group as a context manager, or allowing dataset chunk size to be None. There's absolutely no mention of this in the current v3 documentation, but they do mention that they have a backlog of stuff from v2 that they're planning to add to v3, but haven't done yet.

Overall, version 3 as it stands seems a bit incomplete, so I propose keeping zarr pinned at <3 for a few months and then revisiting.

@ketiltrout
Copy link
Member

I think that's reasonable. There's no particular reason to rush, and it looks like supporting both zarr2 and 3 is going to be difficult.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants