ZEP9 (phase 1): add clarifications for extension naming

[joshmoore]

This PR clarifies the extension mechanism concept in the v3 specification. Comments on any changes which will break existing implementations are STRONGLY encouraged. Please see zarr-developers/zeps#65 for background material.

TODOs:

• clarify the file numbering (currently 3.0.rst)
• move definitions to the appropriate location (core, subtype page or ext

Post-merge:

• remove preview label from zarr-extensions
• return to zarr-specs v3.1 follow-ons #329
• update sphinx to clearly show 3.0 vs 3.1

[rabernat]

@joshmoore - really glad you got this started! 🙌

My feedback is that the PR is hard to review. It touches 15 files, including a ton of minor, unrelated formatting changes to the core spec document.

If we want folks to engage and give meaningful feedback, we need to make it easier to review. I'd recommend starting fresh with a minimal PR in which the diffs are reflective exclusively of the actual proposed changes.

[joshmoore]

@rabernat
really glad you got this started! 🙌

👍

It touches 15 files

You're right. I've extracted out #331.

including a ton of minor, unrelated formatting changes to the core spec document.

I disagree that they are unrelated. Take a look. The sections I've modified were basically already un-parseable. Since I was adding sections, the outline was getting more convoluted.

I'd recommend starting fresh with a minimal PR in which the diffs are reflective exclusively of the actual proposed changes.

👍 Give it a look and let me know what you think.

[jbms]

Thanks for all of your work on this!

My current understanding of the practical effect of proposal is as follows:

-raw names will be granted fairly easily, e.g. zstd, bfloat16, and others I've proposed would be assigned to me, the ones that zarr-python has started using (string, bytes, vlen-utf8, etc.) would be assigned to someone from zarr-python. URL names will be used only for really experimental stuff, all commonly-used extensions will have raw names since they will be minimal effort. Therefore, the verbosity of the URLs is not really a problem in practice.

• the ZEP process, or really any mandatory review process at all, will not be used for proposing extensions that fit into any of the existing extension points, only for entirely new extension points. At most someone might ask around for comments informally before adopting something.

The lack of basically any review worries me a bit. But ultimately I'm in favor of this proposal because I think it reflects the reality that the ZEP process isn't working for the existing extension points, and it would be better to just rely on a less formal process.

[normanrz]

The lack of basically any review worries me a bit. But ultimately I'm in favor of this proposal because I think it reflects the reality that the ZEP process isn't working for the existing extension points, and it would be better to just rely on a less formal process.

I share your concerns to some degree. I think we can adapt the governance structure for extensions in the future, if we think that a more thorough review process would be necessary. We are thinking of forming a zarr specs team that could take on that responsibility.

[jakirkham]

Thank you Josh for spearheading this revision! ❤️

Also thanks to everyone on the ZSC who participated in the discussion and early drafts of this work 🙏

Think this provides a good path for developers to explore and expand the use cases Zarr can handle. With the work Norman has done on the first extensions, we can set an example future contributors can follow

[joshmoore]

Thanks, @jakirkham. With that approval, we now have unanimous support from @zarr-developers/steering-council and no major objections since the call on April 3rd.

I'll close all remaining comments above (per current GH settings), merge (and tag 3.1 -- this repo isn't tagged). I'd encourage interested reviewers to turn their attention to https://github.com/zarr-developers/zarr-extensions/pulls?q=sort%3Aupdated-desc+is%3Apr+is%3Aopen

Two final points:

• Apologies if it seemed like I ignored the zarr. prefix point, @mkitti. Somewhere above in the expanses, I stated how for this PR with its intent of being nonbreaking that's not an option because of the current state of prefixes in the community. We can certainly use the review process on zarr-extensions, though, to keep a prefix clear as those proposals start to appear.

• Thanks for taking the time for the write up, @indexzero! 🙏🏽 What you say resonates, especially the cost of fixing specs post hoc (see above) and the value of having implementations (staging process, etc.). This first step is about removing the requirement to have a PR against this repository in order to create an extension. I think we've achieved that in a fairly minimal manner here, but there is clearly a ways to go, and I imagine your advice will be hugely useful as we approach the question of making more significant changes.

[mkitti]

Somewhere above in the expanses, I stated how for this PR with its intent of being nonbreaking that's not an option because of the current state of prefixes in the

To clarify, what I mean overall is that we should reserve characters for later potential namespace or unregistered names in lieu of actually specifying that mechanism. That is we should reserve characters. I propose we at least copy the reservations of RFC 3986:

reserved = gen-delims / sub-delims

gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@"

sub-delims = "!" / "$" / "&" / "'" / "(" / ")" / "*" / "+" / "," / ";" / "=" 

Additionally, we should also reserve -, ., and ~.

This would allow us to adopt some kind of URI or namespace mechanism in the future if we so choose.

The propsective idea of a default zarr. prefix namespace would allow us to disambiguate these centrally registered extensions from potentially foreign registered extensions. This is meant to be backwards compatible because the unqualified lz4 would implicily be thought of as being zarr.lz4. This could be used to distinguish lz4 as implemented in Python numcodecs or numcodecs.js from the HDF5's incompatible lz4 codec in use with Zarr by Unidata's NetCDF, which we might call hdf5.lz4 in the future. The decision on the default namespace should be deferred, but we should make provisions so that we could adopt such a scheme in the future.

My proposition today is that we reserve . and other characters going forward except in the context of the grandfathered URI names. This would allow us to consider a variety of namespace and extension schemes in the future.

[mkitti]

I turned my comment into a pull request on zarr-extensions:
zarr-developers/zarr-extensions#7