Skip to content

Support the fixed_length_utf32 Zarr V3 string dtype #428

Description

@tylere

Support the fixed_length_utf32 Zarr V3 string dtype

Summary

zarrita 0.7.3 cannot open arrays whose data_type is the
fixed_length_utf32 Zarr V3 extension dtype (spec:
zarr-developers/zarr-extensions ›
data-types/fixed_length_utf32
).
Attempting to zarr.open.v3 such an array throws
TypeError: dataType.match is not a function from inside
src/util.ts's getCtr
because getCtr assumes dataType is a string and calls .match(...)
on it, while v3 extension dtypes are object descriptors of the form
{ name: "fixed_length_utf32", configuration: { length_bytes: N } }.

#329 added support for the related "string" (vlen-utf8) extension but
explicitly scoped out fixed_length_utf32, even though the original
request in #305 mentioned both. As a result, NumPy/xarray-produced v3
stores whose string coordinates land as fixed-length unicode (the
NumPy <U / >U dtype, very common in xarray output) still can't be
opened by zarrita.

Real-world example

Fields of The World (FTW)'s alpha global
predictions store on Source Cooperative
(https://data.source.coop/ftw/global-data/predictions/zarr/alpha/global.zarr)
uses fixed_length_utf32 for its band coordinate (3 class labels,
80-byte-wide strings).

The variable's band/zarr.json:

{
  "shape": [3],
  "data_type": {
    "name": "fixed_length_utf32",
    "configuration": { "length_bytes": 80 }
  },
  "chunk_grid": { "name": "regular", "configuration": { "chunk_shape": [3] } },
  "codecs": [
    { "name": "bytes", "configuration": { "endian": "little" } },
    { "name": "zstd", "configuration": { "level": 0, "checksum": false } }
  ],
  "dimension_names": ["band"],
  "fill_value": "",
  "zarr_format": 3,
  "node_type": "array"
}

Minimal reproduction:

import * as zarr from "zarrita";

const store = new zarr.FetchStore(
  "https://data.source.coop/ftw/global-data/predictions/zarr/alpha/global.zarr",
);
const group = await zarr.open.v3(store, { kind: "group" });
const band = await zarr.open.v3(group.resolve("band"), { kind: "array" });
// → TypeError: dataType.match is not a function
//     at getCtr (zarrita/dist/src/util.js:57:24)
//     at createContext (zarrita/dist/src/hierarchy.js:45:21)
//     at new Array (zarrita/dist/src/hierarchy.js:97:32)
//     at _openV3 (zarrita/dist/src/open.js:112:23)

Tested with zarrita 0.7.3 (current latest on npm).

Spec summary

From the extension
README
:

Each element of an array with this data type is a Unicode string with
a fixed maximum number of code points, where each code point is
encoded as a 4-byte UTF-32 code unit.

A Zarr V2 dtype such as "<U12" corresponds to the Zarr V3
fixed_length_utf32 data type with a length_bytes of 48.

So a chunk for shape [N], length_bytes=B is N × B raw bytes of
little-endian UTF-32. Shorter strings are zero-padded to the full
width.

Why I care / why this matters in the wild

fixed_length_utf32 is the canonical Zarr V3 representation of the
NumPy <U / >U dtype that xarray uses for string coordinates by
default — anyone converting a NumPy/xarray dataset to Zarr V3 with the
"keep dtypes faithful" option will produce stores like the FTW one
above. The "string" (vlen) dtype added in #329 is a separate, newer
representation that requires the producer to opt in.

In the FTW viewer I'm building, I worked around the issue by fetching
the chunk bytes directly with fetch() and running them through
zarrita's bundled zstd codec + a 5-line UTF-32-LE decoder:

const zstdFactory = zarr.registry.get("zstd")!;
const ZstdClass = await zstdFactory();
const codec = await ZstdClass.fromConfig({ level: 0, checksum: false });
const raw = await codec.decode(compressedBytes);
// Then read N × `length_bytes`-wide slots as utf-32-le, stopping at U+0000.

That confirms the decoding path itself is straightforward — the
missing piece is teaching zarr.open.v3 to accept the object
descriptor and route it to the right TypedArray-like.

Suggested implementation sketch

Roughly mirrors what #329 did for "string":

  1. In src/util.ts's getCtr, before the dataType.match(...) line,
    add an isObjectDataType(dataType) branch that dispatches on
    dataType.name. For "fixed_length_utf32", return a constructor
    producing a string-like array of shape.length elements, each
    decoded from length_bytes UTF-32-LE bytes (stop at the first
    U+0000).
  2. In src/metadata.ts, extend the discriminated union for DataType
    to include the fixed_length_utf32 object form (parallel to
    whatever shape "string" got in Add support for type "string" #329).
  3. In src/codecs/bytes.ts, the BytesCodec just needs to know the
    element width — length_bytes from the config gives that directly.
  4. Tests: a v3 fixture array 1d.contiguous.fixed_length_utf32 plus a
    chunked variant, mirroring the 1d.contiguous.string.* fixtures
    added in Add support for type "string" #329.

If a maintainer is amenable, I'm happy to put up a PR following that
shape — wanted to file the issue first to confirm scope.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions