Store versus array properties

[pvanlaake]

In v3 there is the array property chunk_key_encoding which, in its default form, requires that the separator is "." or "/". (In other incarnations that may presumably be any other (sequence of) token(s) that is inoffensive to the store. I personally like "αжنक日" very much as a chunk key separator.) One simple question:

• Shouldn't the chunk_key_encoding property be a store-level property rather than an array property?

If the answer to that question is negative, then the consequence is that any store should support any allowable chunk_key_encoding (sequence of) character(s). Including arrays in a single store using different encodings. Otherwise, how to specify this at the store level?

[jbms]

The spec defines the store as an abstract key-value store interface, where keys are just strings. The store interface doesn't know anything about chunks.

Can you give a more specific example of what you have in mind?

Icechunk is an example of a store that only supports a particular chunk_key_encoding because it imposes various assumptions on the specific form of keys and values that it can store. Attempting to use a different chunk_key_encoding will not work.

[pvanlaake]

The issue is obviously about the keys, not the chunks. The idiomatic structure of a key is determined by what the store will allow or by what is more natural to a store. For instance, a local file system store may use either a dot or a slash as both are allowed in a path name, but other stores may be more restrictive in what is allowed as a separator. In general, any requirements or restrictions on the separator are related to the store, rather than the array, so why is it an array metadata property?

Insofar as I am aware, there is no mechanism to convey store properties to a reader (I am happy to be corrected). Shouldn't there be such a mechanism?

[d-v-b]

if the dimension separator were a store concern, then it would not be possible to have multiple arrays in the same store using different dimension separators, but supporting arrays with varying dimension separators with the same store is a design goal.

[jbms]

It certainly would make sense for an implementation to take the store into account when choosing a default chunk key encoding for a new array. But that doesn't need to be in the spec.

If it doesn't get recorded in the array metadata then there are a lot of added complications just to save a small amount of space in the metadata.

[jbms]

It certainly would make sense for an implementation to take the store into account when choosing a default chunk key encoding for a new array. But that doesn't need to be in the spec.

If it doesn't get recorded in the array metadata then there are a lot of added complications just to save a small amount of space in the metadata.

[pvanlaake]

My point is more about a mechanism to convey store properties to the calling code, rather than saving space. Store properties could include a default chunk_key_encoding but there can also be other properties such as read_only or can_list. It would also be a natural home for consolidated_metadata. Would something like this not be useful for the root zarr.json of a store:

{
  "zarr_format": 3,
  "node_type": "group",
  "store_properties": [
    "read_only": FALSE,
    "can_list": TRUE,
    "chunk_key_encoding": {
      "name": "default",
      "configuration": {
        "separator": "."
      }
    },
    ...],
  "consolidated_metadata": {
    ...
  }
}

The properties and the consolidated_metadata element could all be made optional (so current v.3 stores remain valid). The array CKE can also be made optional but conditional on the presence of a store-level default CKE

if the dimension separator were a store concern, then it would not be possible to have multiple arrays in the same store using different dimension separators, but supporting arrays with varying dimension separators with the same store is a design goal.

The store-level default could still be overridden in the array zarr.json. My gut feeling is that the large majority of zarr stores use a single separator in all arrays, though.

[d-v-b]

arrays and groups are meant to be portable across stores. that means store details like listability should probably not be encoded in array or group metadata, because copying an array or group from an unlistable store to a listable store would invalidate this information.

also, it's helpful to keep in mind that readers and writers choose a store implementation before they can read / write array or group metadata. with that in mind, array / group metadata really doesn't seem like a good place for storage details.

[pvanlaake]

I'm not sure if I understand your reply correctly. Referring to portability, it seems that you are suggesting that copying an array or a group from one store to another should be possible at the object level, moving over JSON objects and chunks, without any inspection of any kind, especially at the receiving store. Is that what you mean? If so, what kind of a scenario are you referring to? It would seem to me that I/O latency, even on local disk drives, would leave ample time to inspect the JSON object and make any required changes before writing. And it sure doesn't sound like good software engineering to assume that "blindly" copying low-levels objects will preserve the semantics of higher-level objects. In GeoZarr, for instance, multiple arrays jointly form a "data variable" (borrowing CF terminology here). The only way to copy a data variable properly is to do so by identifying its constituent arrays which one does by reading the metadata. But perhaps I am misunderstanding what you mean by portability and array copying.

Similarly with your second statement, when accessing an existing store, perhaps an AWS or HTTP store, I do not get to "choose a store implementation" - I have to use the store that is available.

Perhaps more to the point, the only place I am suggesting that placing store properties might be useful is the root node. Copying an array from a listable store to an unlistable store, or vice-versa, would not invalidate the array or the store, especially not if the sensible approach of metadata inspection is used. Overwriting the root node would be problematic, but again, what scenario would allow that?

[d-v-b]

t seems that you are suggesting that copying an array or a group from one store to another should be possible at the object level, moving over JSON objects and chunks, without any inspection of any kind, especially at the receiving store. Is that what you mean? If so, what kind of a scenario are you referring to?

Yes, exactly. Blindly copying Zarr data from one store to another (i.e., just moving bytes) is common and useful. For example, many people copy Zarr data from local storage to cloud storage using zarr-ignorant tools like rclone or the aws cli. This is possible because the store is a separate abstraction layer from arrays / groups.

Similarly with your second statement, when accessing an existing store, perhaps an AWS or HTTP store, I do not get to "choose a store implementation" - I have to use the store that is available.

Zarr data can be stored on AWS or hosted over HTTP in multiple ways. Zarr data can be stored in Zip archives, which can be read over AWS or HTTP. Icechunk is another example. For a particular storage backend, there are multiple ways data can be organized and still be readable as Zarr. When you decide to open Zarr data, you have by definition already chosen a particular store implementation. So it would be useless to encode storage details inside array or group metadata documents, as you need a store in order to read them.

And for listing in particular: on cloud storage backends the ability to list objects isn't even a property of the storage backend -- it's a permission, which some users have, and other users may not have. So it would not be very meaningful to encode this in array or group metadata.