generic container data types

[d-v-b]

it would be useful to have some data types for expressing collections of of other data types.

• if we defined base byte and utf8 data types, then the existing bytes and string data types could be expressed as variable_length[<base type>]
• similarly, the existing r* data type, and other fixed-length bytes data types, could be expressed as fixed_length[<base type>]. This data type would require a length (in scalars) as part of its configuration.
• NumPy supports "subarray" data types, which are basically an N-dimensional generalization of a fixed-length container for some base data type. Users have requested support for this data type in Zarr-Python. I would like to have a spec in place before implementing anything.

[clbarnes]

+1 on generic varlen lists, which IMO overlaps with the label multiset proposal. These could be supported by different codecs like naive length-data, arrow-style run-end encoding, etc.

I suppose this goes for r* too, but how valuable is a fixed-length data type beyond just adding a dimension to your array and configuring the layout to ensure that that dimension is memory-contiguous? Same question if subarray data types are fixed-length. This feels like a flat-vs-nested trade-off. Could subarrays have subsubarrays?

Also +1 on not accepting any new extension to zarr-python without a spec here, as a rule; and recommending the same approach in other zarr implementations.

[d-v-b]

I suppose this goes for r* too, but how valuable is a fixed-length data type beyond just adding a dimension to your array and configuring the layout to ensure that that dimension is memory-contiguous?

Speculatively, one use case for a fixed-length data type would be to model fixed-size streams of heterogeneous types, e.g. a 72-byte-wide scalar made from packing a uint8 before a float64. This kind of data occurs in headers and footers of file formats. The "packing axis" wouldn't be natural to express as an array dimension in these cases. But that's speculative -- I have never seen the r* data actually used.

[clbarnes]

fixed-size streams of heterogeneous types, e.g. a 72-byte-wide scalar made from packing a uint8 before a float64.

Is this different to the struct type?

[d-v-b]

Is this different to the struct type?

it's a possible serialization of the struct type, and the struct type is probably the right choice here, now that that the data type has been defined. But r* is older than struct.

[d-v-b]

see #62

[d-v-b]

if #62 (generic fixed_size_list data type) is palatable, I would then propose three new data types:

• binary / byte: a single octet)
• unicode: a single code point)
• list[T]: a variable-sized collection of some inner data type T

r* becomes expressible as fixed_size_list[binary, Length]
string and bytes data types are expressible as list[unicode] and list[binary]

with some new codecs, I think we can fully deprecate the numpy-specific data types, and re-use the generic container types.

[clbarnes]

byte

Is this worth distinguishing from uint8? Rust doesn't bother.

unicode

The fixed_size_list proposal requires that the members of the list are also fixed-size, which means encoding unicode as UTF-32. Do you propose the same limitation for variable-sized lists? That would rule out UTF-8, which would be a shame given how much more common it is as a text encoding.

[d-v-b]

Is this worth distinguishing from uint8? Rust doesn't bother.

we already have types for raw bytes (r*). So I'd go with precedent and keep the bytes / int distinction. that also keeps us parallel with python and apache arrow, which treat bytes and ints separately.

The fixed_size_list proposal requires that the members of the list are also fixed-size, which means encoding unicode as UTF-32. Do you propose the same limitation for variable-sized lists?

that proposal doesn't actually require that fixed_sized_list[T] scalars have a constant encoded size in bytes. It just requires that the codec can round-trip the scalar. So utf8 and utf32 should both be viable text encodings for the array -> bytes step.

[jbms]

In general the variable-length list types make sense as generalizations but:

• it might be interesting to consider general ragged arrays rather than the more restricted form supported by list[T]

• I don't think unicode for representing a Unicode code point is helpful --- it is more likely to just lead users astray. You might sometimes deal with code points, grapheme clusters, etc. when processing text but zarr is for storage. I'd be more in favor of a type representing a utf-8 code unit, i.e. byte with extra semantic labeling, but still probably doesn't pull it's weight.

• The string data type is already defined as a variable-length utf8 sequence, and normally we don't want to store utf32 at any point, even in memory. To make list[unicode] just pass through utf8 somehow, without even storing the number of code points, would seem to add a lot of implementation complexity.

[d-v-b]

it might be interesting to consider general ragged arrays rather than the more restricted form supported by list[T]

I'm also interested, but I confess I don't know how a general ragged array differs from list[T]. Could you explain in more detail?

I don't think unicode for representing a Unicode code point is helpful --- it is more likely to just lead users astray. You might sometimes deal with code points, grapheme clusters, etc. when processing text but zarr is for storage. I'd be more in favor of a type representing a utf-8 code unit, i.e. byte with extra semantic labeling, but still probably doesn't pull it's weight.

The string data type is already defined as a variable-length utf8 sequence, and normally we don't want to store utf32 at any point, even in memory. To make list[unicode] just pass through utf8 somehow, without even storing the number of code points, would seem to add a lot of implementation complexity.

that works for me, we can scratch the primitive unicode dtype

[jbms]

it might be interesting to consider general ragged arrays rather than the more restricted form supported by list[T]

I'm also interested, but I confess I don't know how a general ragged array differs from list[T]. Could you explain in more detail?

As I see it, a ragged array would allow any of the dimensions to be variable-length, not just the last one, and potentially could allow chunking along variable-length dimensions, and would allow efficient slicing/indexing of variable-length dimensions just as you can do with fixed-length dimensions in zarr.

As a further extension, you might also have several related variables and want to tie their variable-length dimensions together (and thereby also avoid having to redundantly store the sizes).

I don't think unicode for representing a Unicode code point is helpful --- it is more likely to just lead users astray. You might sometimes deal with code points, grapheme clusters, etc. when processing text but zarr is for storage. I'd be more in favor of a type representing a utf-8 code unit, i.e. byte with extra semantic labeling, but still probably doesn't pull it's weight.
The string data type is already defined as a variable-length utf8 sequence, and normally we don't want to store utf32 at any point, even in memory. To make list[unicode] just pass through utf8 somehow, without even storing the number of code points, would seem to add a lot of implementation complexity.

that works for me, we can scratch the primitive unicode dtype

[d-v-b]

As I see it, a ragged array would allow any of the dimensions to be variable-length, not just the last one, and potentially could allow chunking along variable-length dimensions, and would allow efficient slicing/indexing of variable-length dimensions just as you can do with fixed-length dimensions in zarr.

As a further extension, you might also have several related variables and want to tie their variable-length dimensions together (and thereby also avoid having to redundantly store the sizes).

Would the semantics of this data type align with / resemble https://arrow.apache.org/docs/format/CanonicalExtensions.html#variable-shape-tensor?

And how would you see the data type definition interacting with the chunk grid?

[jbms]

As I see it, a ragged array would allow any of the dimensions to be variable-length, not just the last one, and potentially could allow chunking along variable-length dimensions, and would allow efficient slicing/indexing of variable-length dimensions just as you can do with fixed-length dimensions in zarr.
As a further extension, you might also have several related variables and want to tie their variable-length dimensions together (and thereby also avoid having to redundantly store the sizes).

Would the semantics of this data type align with / resemble https://arrow.apache.org/docs/format/CanonicalExtensions.html#variable-shape-tensor?

I think it isn't that related although indeed you can have ragged arrays by nesting lists and/or variable shape tensors.

And how would you see the data type definition interacting with the chunk grid?

As I imagined it, "ragged array" would not be a data type --- it would be a more general zarr extension. You would just use any regular data type to specify the element type.

Basically instead of the metadata shape being a fixed array of numbers, you would specify for each dimension either a fixed size, or the set of dimensions on which the size depends, e.g.

"shape": [5, {"variable": [0]}, {"variable": [0]}, 3, {"variable": [1, 2]}]

would mean that:

• dimension 0 has a fixed size of 5,
• dimension 1 has variable size that depends on dimension 0, meaning we have to store a 1-d array of size 5 to represent the size of dimension 1,
• dimension 2 also has variable size that depends on dimension 0, meaning we have to also store a 1-d array of size 5 to represent the size of dimension 2
• dimension 3 has fixed size of 3
• dimension 4 has variable size that depends on dimensions 1 and 2, and since those dimensions are themselves variable-size the size array for dimension 4 is itself a 2-d ragged array

I suppose in principle the chunk size could also be variable in the same way as the shape --- if chunking of variable-length dimensions is allowed then it would also make sense for the chunking to be variable-length.

I haven't thought through all of the details though.

[clbarnes]

that proposal doesn't actually require that fixed_sized_list[T] scalars have a constant encoded size in bytes

Perhaps I misunderstood this part of #62 :

| base_data_type | The JSON representation of a Zarr v3 data type | yes | The data type of each element. MUST have a fixed encoded size.