Skip to content

Fix bytes codec endian for big-endian structured dtypes#1047

Open
emfdavid wants to merge 1 commit into
zarr-developers:mainfrom
emfdavid:fix/struct-codec-endianness
Open

Fix bytes codec endian for big-endian structured dtypes#1047
emfdavid wants to merge 1 commit into
zarr-developers:mainfrom
emfdavid:fix/struct-codec-endianness

Conversation

@emfdavid

@emfdavid emfdavid commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

What

convert_to_codec_pipeline chose the array→bytes codec's byte order from dtype.byteorder.
numpy reports "|" for every structured dtype regardless of what its fields hold, so
big-endian record data fell through to the little-endian BytesCodec() default. zarr then
faithfully decodes the referenced bytes in the byte order it was handed — the wrong one — so a
virtual reference to a FITS BinTableHDU (e.g. an SDSS spectrum) reads back silently wrong
values
. No error, just garbage.

Simple scalar arrays were unaffected (np.dtype('>f4').byteorder == '>'), which is why this
only ever surfaced on structured/record dtypes.

Fix

Derive the byte order from the fields rather than from the dtype itself:

  • walk dtype.fields, looking through subarray fields (whose order hides behind .base) and
    nested structs (hence the recursion);
  • ignore fields that carry no byte order at all — single-byte numbers, strings — instead of
    letting them imply little-endian;
  • a structured dtype whose fields mix byte orders now raises ValueError: a Zarr V3 array
    has a single bytes codec and so cannot express per-field byte order. FITS is uniformly
    big-endian by standard, so this doesn't bite the motivating case; happy to downgrade to a
    warning if you'd prefer.

numpy gotchas the naive dtype.byteorder == ">" check misses, all covered by the new helper and
tests:

dtype .byteorder
np.dtype([('flux','>f4')]) `'
np.dtype('<f4') '=' — not '<'
('>f4', (3,)) subarray field `'
nested struct `'
S8, i1 `'

Companion zarr fix

This PR is one half of the story. It makes VirtualiZarr write endian: big for a big-endian
struct; zarr then has to honor it when decoding the referenced bytes. That decode-side branch
for structured dtypes landed in zarr-developers/zarr-python#4142 (merged; ships in zarr
3.2.2). Before that fix, zarr ignored the codec's endian for structs, so even a
correctly-tagged array read back wrong. Both halves are needed for a virtual big-endian struct to
read correctly end-to-end.

Testing

  • new test_codecs.py cases covering the dtypes above (incl. the mixed-order raise); the codec
    pipeline now emits BytesCodec(endian="big") for a big-endian struct. These pass on released
    zarr 3.2.1 (what CI pins);
  • end-to-end (FITS BinTableHDU → kerchunk → VirtualiZarr → Icechunk → native read
    byte-identical to astropy, no .view('>f4') consumer workaround) additionally requires zarr's
    decode fix, i.e. a zarr carrying fix: byte-order handling for structured dtypes in the bytes codec zarr-python#4142 (main today, 3.2.2 on
    release).

Sibling to #1037 (the list/base64 struct-ref fix) — same FITS-binary-table motivation.

Heads-up: a separate, pre-existing break (not this PR)

On zarr main, virtualizarr/utils.py:130 does endian.value, but zarr changed
BytesCodec.endian from an Endian enum to a plain str, giving 4 test_kerchunk.py
failures. CI is green only because it pins released zarr; this will surface when zarr 3.2.2
releases. Flagging separately — glad to file an issue or fold a fix in if that's useful.

Acceptance criteria

  • Tests added — new cases in virtualizarr/tests/test_codecs.py
  • Tests passing — codec suite + FITS BinTableHDU end-to-end green locally on released zarr 3.2.1; CI here is authoritative
  • No test coverage regression — this only adds tests
  • Full type hint coverage — the new _byte_orders / _is_big_endian helpers are annotated
  • Changes are documented in docs/about/releases.md
  • New functions/methods listed under docs/api — n/a, _byte_orders / _is_big_endian are private internal helpers
  • New functionality has documentation — n/a, this is a bugfix, not new functionality

🤖 Generated with Claude Code

The byte order of the array-to-bytes codec was taken from dtype.byteorder,
which numpy reports as "|" for every structured dtype no matter what its
fields hold. Big-endian record data therefore fell through to the
little-endian default, and zarr faithfully decoded the referenced bytes in
the byte order it had been told -- the wrong one -- so virtual references to
a FITS BinTableHDU (e.g. an SDSS spectrum) yielded silently wrong values.

Read the byte order from the fields instead, looking through subarray fields
(whose order hides behind .base) and nested structs, and ignoring fields that
carry no byte order at all (single-byte numbers, strings) rather than letting
them imply little-endian. A structured dtype whose fields mix byte orders now
raises: a Zarr V3 array has a single bytes codec and cannot express it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@codecov

codecov Bot commented Jul 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.41%. Comparing base (fbe8afc) to head (be8f65d).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1047      +/-   ##
==========================================
+ Coverage   90.36%   90.41%   +0.05%     
==========================================
  Files          37       37              
  Lines        2242     2254      +12     
==========================================
+ Hits         2026     2038      +12     
  Misses        216      216              
Files with missing lines Coverage Δ
virtualizarr/codecs.py 94.36% <100.00%> (+1.14%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant