consolidated metadata storage for Zarr v3

[grlee77]

In the current draft of the v3 spec in zarr-developers/zarr-python#898, consolidated metadata has been implemented similarly to v2. Currently this metadata gets stored in the group meta/root/consolidated, but this is not part of the spec. Likely we need to specify an extension for how consolidated metadata should be handled.

[ianliu]

Is the V3 considering metadata compression? I'm working with a dataset where the consolidated metadata (v2) has 1.2GB in size, and opening it from S3 takes a very long time to open.
After gzipping the metadata, it reduces to 20MB, so it would be nice to have compressed metadata.

[alimanfoo]

Hi @ianliu, thanks this is very useful to know. Adding a capability for compression into a v3 extension for consolidated metadata sounds like a good idea to me.

[rabernat]

@ianliu I'm curious what makes the metadata so big in your case? Do you have lots of arrays? Or very large custom attrs?

V3 allows for defining of custom metadata encoding, so a compressed format could be used.

Also, a major question for V3 is whether user metadata is a separate document or not (see #149 (comment)). If you have 1GB of custom attrs in a single array, that would be a strong motivation not to put user metadata together with the core array metadata.

[ianliu]

@rabernat in my particular case, I have many small arrays. It could be argued that there is a better data representation that would consolidate multiple arrays into a single array, thus reducing metadata, but the way our program is setup makes this a little bit trickier (mainly because our program produces results in parallel).

[joshmoore]

💯 for potential consolidated metadata features but

but this is not part of the spec. Likely we need to specify an extension for how consolidated metadata should be handled.

would this suggest stripping the current consolidated metadata support in V3 until there's an extension?