You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Support the fixed_length_utf32 Zarr V3 string dtype
Summary
zarrita 0.7.3 cannot open arrays whose data_type is the fixed_length_utf32 Zarr V3 extension dtype (spec: zarr-developers/zarr-extensions ›
data-types/fixed_length_utf32).
Attempting to zarr.open.v3 such an array throws TypeError: dataType.match is not a function from inside src/util.ts's getCtr
because getCtr assumes dataType is a string and calls .match(...)
on it, while v3 extension dtypes are object descriptors of the form { name: "fixed_length_utf32", configuration: { length_bytes: N } }.
#329 added support for the related "string" (vlen-utf8) extension but
explicitly scoped out fixed_length_utf32, even though the original
request in #305 mentioned both. As a result, NumPy/xarray-produced v3
stores whose string coordinates land as fixed-length unicode (the
NumPy <U / >U dtype, very common in xarray output) still can't be
opened by zarrita.
Real-world example
Fields of The World (FTW)'s alpha global
predictions store on Source Cooperative
(https://data.source.coop/ftw/global-data/predictions/zarr/alpha/global.zarr)
uses fixed_length_utf32 for its band coordinate (3 class labels,
80-byte-wide strings).
import*aszarrfrom"zarrita";conststore=newzarr.FetchStore("https://data.source.coop/ftw/global-data/predictions/zarr/alpha/global.zarr",);constgroup=awaitzarr.open.v3(store,{kind: "group"});constband=awaitzarr.open.v3(group.resolve("band"),{kind: "array"});// → TypeError: dataType.match is not a function// at getCtr (zarrita/dist/src/util.js:57:24)// at createContext (zarrita/dist/src/hierarchy.js:45:21)// at new Array (zarrita/dist/src/hierarchy.js:97:32)// at _openV3 (zarrita/dist/src/open.js:112:23)
Tested with zarrita 0.7.3 (current latest on npm).
Each element of an array with this data type is a Unicode string with
a fixed maximum number of code points, where each code point is
encoded as a 4-byte UTF-32 code unit.
A Zarr V2 dtype such as "<U12" corresponds to the Zarr V3 fixed_length_utf32 data type with a length_bytes of 48.
So a chunk for shape [N], length_bytes=B is N × B raw bytes of
little-endian UTF-32. Shorter strings are zero-padded to the full
width.
Why I care / why this matters in the wild
fixed_length_utf32 is the canonical Zarr V3 representation of the
NumPy <U / >U dtype that xarray uses for string coordinates by
default — anyone converting a NumPy/xarray dataset to Zarr V3 with the
"keep dtypes faithful" option will produce stores like the FTW one
above. The "string" (vlen) dtype added in #329 is a separate, newer
representation that requires the producer to opt in.
In the FTW viewer I'm building, I worked around the issue by fetching
the chunk bytes directly with fetch() and running them through
zarrita's bundled zstd codec + a 5-line UTF-32-LE decoder:
constzstdFactory=zarr.registry.get("zstd")!;constZstdClass=awaitzstdFactory();constcodec=awaitZstdClass.fromConfig({level: 0,checksum: false});constraw=awaitcodec.decode(compressedBytes);// Then read N × `length_bytes`-wide slots as utf-32-le, stopping at U+0000.
That confirms the decoding path itself is straightforward — the
missing piece is teaching zarr.open.v3 to accept the object
descriptor and route it to the right TypedArray-like.
In src/util.ts's getCtr, before the dataType.match(...) line,
add an isObjectDataType(dataType) branch that dispatches on dataType.name. For "fixed_length_utf32", return a constructor
producing a string-like array of shape.length elements, each
decoded from length_bytes UTF-32-LE bytes (stop at the first U+0000).
In src/metadata.ts, extend the discriminated union for DataType
to include the fixed_length_utf32 object form (parallel to
whatever shape "string" got in Add support for type "string" #329).
In src/codecs/bytes.ts, the BytesCodec just needs to know the
element width — length_bytes from the config gives that directly.
Tests: a v3 fixture array 1d.contiguous.fixed_length_utf32 plus a
chunked variant, mirroring the 1d.contiguous.string.* fixtures
added in Add support for type "string" #329.
If a maintainer is amenable, I'm happy to put up a PR following that
shape — wanted to file the issue first to confirm scope.
Support the
fixed_length_utf32Zarr V3 string dtypeSummary
zarrita 0.7.3 cannot open arrays whose
data_typeis thefixed_length_utf32Zarr V3 extension dtype (spec:zarr-developers/zarr-extensions ›
data-types/fixed_length_utf32).
Attempting to
zarr.open.v3such an array throwsTypeError: dataType.match is not a functionfrom insidesrc/util.ts'sgetCtrbecause
getCtrassumesdataTypeis a string and calls.match(...)on it, while v3 extension dtypes are object descriptors of the form
{ name: "fixed_length_utf32", configuration: { length_bytes: N } }.#329 added support for the related
"string"(vlen-utf8) extension butexplicitly scoped out
fixed_length_utf32, even though the originalrequest in #305 mentioned both. As a result, NumPy/xarray-produced v3
stores whose string coordinates land as fixed-length unicode (the
NumPy
<U/>Udtype, very common in xarray output) still can't beopened by zarrita.
Real-world example
Fields of The World (FTW)'s alpha global
predictions store on Source Cooperative
(
https://data.source.coop/ftw/global-data/predictions/zarr/alpha/global.zarr)uses
fixed_length_utf32for itsbandcoordinate (3 class labels,80-byte-wide strings).
The variable's
band/zarr.json:{ "shape": [3], "data_type": { "name": "fixed_length_utf32", "configuration": { "length_bytes": 80 } }, "chunk_grid": { "name": "regular", "configuration": { "chunk_shape": [3] } }, "codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "zstd", "configuration": { "level": 0, "checksum": false } } ], "dimension_names": ["band"], "fill_value": "", "zarr_format": 3, "node_type": "array" }Minimal reproduction:
Tested with zarrita 0.7.3 (current
lateston npm).Spec summary
From the extension
README:
So a chunk for shape
[N],length_bytes=BisN × Braw bytes oflittle-endian UTF-32. Shorter strings are zero-padded to the full
width.
Why I care / why this matters in the wild
fixed_length_utf32is the canonical Zarr V3 representation of theNumPy
<U/>Udtype that xarray uses for string coordinates bydefault — anyone converting a NumPy/xarray dataset to Zarr V3 with the
"keep dtypes faithful" option will produce stores like the FTW one
above. The
"string"(vlen) dtype added in #329 is a separate, newerrepresentation that requires the producer to opt in.
In the FTW viewer I'm building, I worked around the issue by fetching
the chunk bytes directly with
fetch()and running them throughzarrita's bundled zstd codec + a 5-line UTF-32-LE decoder:
That confirms the decoding path itself is straightforward — the
missing piece is teaching
zarr.open.v3to accept the objectdescriptor and route it to the right TypedArray-like.
Suggested implementation sketch
Roughly mirrors what #329 did for
"string":src/util.ts'sgetCtr, before thedataType.match(...)line,add an
isObjectDataType(dataType)branch that dispatches ondataType.name. For"fixed_length_utf32", return a constructorproducing a string-like array of
shape.lengthelements, eachdecoded from
length_bytesUTF-32-LE bytes (stop at the firstU+0000).src/metadata.ts, extend the discriminated union forDataTypeto include the
fixed_length_utf32object form (parallel towhatever shape
"string"got in Add support for type "string" #329).src/codecs/bytes.ts, theBytesCodecjust needs to know theelement width —
length_bytesfrom the config gives that directly.1d.contiguous.fixed_length_utf32plus achunked variant, mirroring the
1d.contiguous.string.*fixturesadded in Add support for type "string" #329.
If a maintainer is amenable, I'm happy to put up a PR following that
shape — wanted to file the issue first to confirm scope.
References
fixed_length_utf32andStringDType; was closed by Add support for type "string" #329 which only added"string")