The v3 spec defines an "optional" data type for raw bytes: the r* data type. This data type is parametrized by a length in bits, but unusually the r* data type stores the parametrization in its name: (r8, r16, ...). The fill value is a JSON array of ints in the range [0, 255], one per byte.
The spec emphasizes that this data type is not done:
We are explicitly looking for more feedback and prototypes of code using the r*,
raw bits, for various endianness and whether the spec could be made clearer.
Over in Zarr Python, we implemented a nearly identical data type. It uses the more conventional metadata form for storing the configuration:
{"name": "raw_bytes", "configuration": {"length_bytes": <int>}}
And the fill value is encoded as a base64 string, not an array of ints.
I am not happy that we duplicated functionality here. But I'm also not happy with the design of the r* data type -- to me it's quite bad to stringly type configuration parameters in the name of the data type, and I think encoding raw bytes scalars as JSON arrays of ints is worse than base64.
I would like to phase out the zarr-python-specific data type in favor of something community-standardized. I would publish a spec as a historical document, and use that document to refer people to the new community-standardized data type for fixed-length sequences of raw bytes. But I would rather not just use the r* data type as-is. I would like to fix its problems first. I understand this requires adaptation from other implementations, so if anyone is deeply opposed to this churn let me know and we can stick with the problematic r* data type as-is.
Either way, we need to update the core spec language about the r* data type -- if we aren't willing to change it, then it's simply not true that the spec authors are looking for feedback on the data type, and we should remove that language. And we are willing to change it, then we should change it.
The v3 spec defines an "optional" data type for raw bytes: the
r*data type. This data type is parametrized by a length in bits, but unusually ther*data type stores the parametrization in its name: (r8,r16, ...). The fill value is a JSON array of ints in the range[0, 255], one per byte.The spec emphasizes that this data type is not done:
Over in Zarr Python, we implemented a nearly identical data type. It uses the more conventional metadata form for storing the configuration:
And the fill value is encoded as a base64 string, not an array of ints.
I am not happy that we duplicated functionality here. But I'm also not happy with the design of the
r*data type -- to me it's quite bad to stringly type configuration parameters in the name of the data type, and I think encoding raw bytes scalars as JSON arrays of ints is worse than base64.I would like to phase out the zarr-python-specific data type in favor of something community-standardized. I would publish a spec as a historical document, and use that document to refer people to the new community-standardized data type for fixed-length sequences of raw bytes. But I would rather not just use the
r*data type as-is. I would like to fix its problems first. I understand this requires adaptation from other implementations, so if anyone is deeply opposed to this churn let me know and we can stick with the problematicr*data type as-is.Either way, we need to update the core spec language about the
r*data type -- if we aren't willing to change it, then it's simply not true that the spec authors are looking for feedback on the data type, and we should remove that language. And we are willing to change it, then we should change it.