v3 structured dtype: per-field endianness still lost on metadata roundtrip (#4142 covers the write path only; breaks virtual/reference stores)

[emfdavid]

Follow-up to #4141 / #4142

First — thank you @d-v-b for the incredibly fast turnaround on #4142! It fixed the case I hit first (zarr-written big-endian structured arrays now round-trip correctly).

While validating it end-to-end I found that the underlying issue #4141 described — the serialization itself dropping per-field endianness — is still present. #4142 resolved the symptom on the write path by byte-swapping structured data to native on encode, but the metadata serialization still loses per-field byte order. That leaves one important case broken: stores whose chunk bytes are not written by zarr — i.e. virtual / reference stores (VirtualiZarr referencing big-endian FITS binary tables → Icechunk). There is no write to swap, so the reopened-native metadata misreads the big-endian bytes, silently.

The serialization still drops per-field endianness (pure zarr, no extras)

import numpy as np
from zarr.core.dtype import parse_data_type
from zarr.core.metadata.v3 import ArrayV3Metadata

be = np.dtype([("flux", ">f4"), ("ivar", ">f4")])
meta = ArrayV3Metadata(
    shape=(3,), data_type=parse_data_type(be, zarr_format=3),
    chunk_grid={"name": "regular", "configuration": {"chunk_shape": (3,)}},
    chunk_key_encoding={"name": "default"}, fill_value=np.zeros((), be)[()],
    codecs=[{"name": "bytes", "configuration": {"endian": "little"}}],
    attributes={}, dimension_names=None,
)
print(meta.data_type)                                          # Struct(... Float32(endianness='big'))
print(ArrayV3Metadata.from_dict(meta.to_dict()).data_type)     # Struct(... Float32(endianness='little'))  ← dropped

The serialized form has nowhere to record byte order — each field is a bare type name:

"data_type": {"name": "struct", "configuration": {"fields": [
    {"name": "flux", "data_type": "float32"}, {"name": "ivar", "data_type": "float32"}]}}

so from_dict rebuilds every field as native. parse_data_type itself preserves >f4; the loss is purely in to_dict → from_dict.

Why #4142 covers the write path but not virtual/reference reads

check	zarr 3.2.1	post-#4142 (main)
to_dict → from_dict preserves struct endianness	❌	❌ (unchanged)
reopened data_type label	little	little
create_array(dtype=">f4 struct") → reopen values correct	❌	✅

On main the reopened label is still little, but the values are correct because encode swapped the bytes to native on the way to disk — write-swap and native-label are symmetric. A virtual/reference store never encodes through zarr (it points at the original big-endian FITS bytes in place), so that swap never runs, and the read is wrong. The only recourse for a consumer today is to reinterpret each field with .view('>f4') at the numpy boundary.

Why this one is harder (and needs a backward-compatible representation)

In the v3 model, byte order for scalars lives in the BytesCodec (endian), not the data type. A structured array has a single BytesCodec for all fields, so it cannot express per-field byte order — and the struct's serialized field spec ({"name", "data_type"}) has no slot for it either. So per-field endianness is currently unrepresentable in the serialized form.

A fix likely needs the structured-dtype field representation to carry per-field byte order in a backward-compatible way — e.g. an optional endianness (or a richer per-field data_type object) that defaults to native when absent, so existing serialized structs keep reading as they do today. Since the v3 structured dtype is still flagged unstable/unspecified (the UnstableSpecificationWarning points at zarr-extensions), this may also warrant a small spec decision on where per-field byte order lives. Happy to help test whatever shape you land on.

Real-world impact

Reading archival big-endian FITS binary tables (astronomy) in place via VirtualiZarr → Icechunk, with no reshard. The bytes are referenced, never rewritten, so the write-side swap from #4142 doesn't apply and consumers get silently wrong values without the .view('>f4') workaround.

Versions

Reproduced on released zarr==3.2.1 and on main at the #4142 merge commit (bbe27d7). numpy 2.x, Python 3.12.

[d-v-b]

thanks for this report, it might be helpful here to establish that the zarr V3 format uses a different model of data types than zarr V2 used. In Zarr V2, like NumPy, the byte order of a data type was part of its definition. So a data type could be little- or big-endian. In Zarr V3, the data type itself contains no byte order information. In fact, Zarr V3 gives implementations complete freedom to use whatever in-memory data structure they want for decoded scalars of a given data type. That's because the decoded form is confined to a particular implementation. What the spec requires is that implementations agree on the encoded representation of data. This is why the bytes codec takes an endian parameter. It defines the encoded representation of scalars with byte-order semantics, but it doesn't define the decoded representation! That's an implementation concern. So we won't be adding byte order information to the basic struct metadata (that's a stored-representation-concern), but we can change what zarr python does.

In Zarr Python, our current practice is to decode zarr data into numpy arrays that use the platform endianness. We could expose this via configuration as a knob users can adjust, and it would allow decodes of big-endian data to be a bit more efficient because a copy could be avoided. We could also allow users to declare the numpy data type they want to read into when they access the array (probably better than a global config). But before doing that, it would be good to know why the byte order of the numpy arrays is important for your application.

[emfdavid]

Thanks @d-v-b — that's a clear explanation, and it resolves this issue. Agreed on the model: byte
order describes the encoded representation, it belongs in the bytes codec, and the struct
metadata shouldn't carry it. The fix I proposed in the description is the wrong one — please
disregard it. I updated my test and #4142 does fix the issue I was seeing with the FITS data.

The remaining issue was mine via VirtualiZarr, not zarr-python. VirtualiZarr picks the codec with
if dtype.byteorder == ">", but numpy reports np.dtype([("flux", ">f4")]).byteorder == "|"
for a structured dtype (it's the field that is >), so struct arrays silently got BytesCodec()
— little-endian.

Confirmed end-to-end on main (FITS BinTableHDU → kerchunk → VirtualiZarr → Icechunk → read).
Patching only VirtualiZarr's byte-order detection:

	codec endian written	native read matches astropy
VirtualiZarr as-is	little	❌
with the fix	big	✅

No zarr change needed, and no .view('>f4') workaround on the consumer side once it lands. I'm
filing that on VirtualiZarr.

on your byte-order question — do we want the encoded (big-endian) form?

Short version: no — we want decode-to-native, so zarr's current default is right for us. The only
thing that was ever wrong here was silent corruption, and that's resolved. What follows is a
performance observation about how the native decode allocates, not a correctness ask.

Context: we stream archival big-endian FITS in place into a training loader via VirtualiZarr →
Icechunk, no reshard. zarr never writes the pixels (the chunk bytes are byte-range references into the
original files), so the encode path never runs and decode is the whole story — exactly what your fix
covers. Our framework boundary is DLPack, which rejects non-native byte order (and structured dtypes),
so something on our side has to land native before a tensor crosses it.

I went looking at whether that swap needs a full-size copy. Measured on a 256 MiB >f4 block,
coarsened 4×4 and handed to torch (numpy 2.4, median of 7):

When a vectorized transform follows (our common case):

	time	peak extra
astype('<f4') then coarsen	463 ms	+272 MiB
coarsen directly on the >f4	325 ms	+16 MiB
byteswap(inplace=True).view() then coarsen	344 ms	+16 MiB

The direct read is cheapest: numpy's reduction runs through its internal buffer (NPY_BUFSIZE) and
swaps each ~8 K-element block in cache as part of the pass it was already making, so there's never a
full-size native temporary — only the small output. An explicit astype/byteswap() copy forces
exactly the temporary we're trying to avoid (the +256 MiB). byteswap(inplace=True) recovers the
memory but mutates the buffer, so we only reach for it where we own the data outright.

Pure pass-through (no transform — a swap is unavoidable):

	time	peak extra
astype('<f4')	161 ms	+256 MiB
byteswap(inplace=True).view(native)	38 ms	+0 MiB

So I had one thing backwards earlier: pass-through is not "a wash." An in-place swap makes it ~4×
faster and zero-alloc versus the astype copy.

Which reframes what I'd actually ask for, and it's narrower than a dtype knob. #4142's decode does the
swap as chunk_array.astype(...) — a fresh full-size allocation. zarr owns that just-decoded buffer,
so if it's exclusively held at that point, an in-place swap there would deliver the identical
native result at ~zero extra allocation (the 38 ms / +0 row) — no encoded-form exposure, nothing that
fights the v3 model. That's a performance nicety, not a correctness need; happy to open a separate
issue and test a patch if it's worth tracking.

Thanks again for the fast turnaround on #4142, and sorry for sending you a follow-up that your own fix
had already covered.

_{Versions: zarr 3.2.1 (released) vs main @ d916f759; numpy 2.4.6; Python 3.12. Timings are one
transform shape on one box — indicative, not a benchmark.}