Chunk-range temporal reads via hidefix (drop whole-granule fetch, eventually h5py)

[espg]

🤖 from Claude

Follow-up to #220/#221 (which put xarray + h5netcdf + h5py in the Lambda layer to unblock the temporal worker): replace the temporal reader's whole-granule fetch with chunk-range reads, most likely via hidefix — which would also let h5py/libhdf5 leave the layer again.

Why

zagg.temporal.open_dataset's non-.zarr S3 branch GETs the entire object into memory before opening (obstore.get(...).bytes() → xr.open_dataset(BytesIO)). Correct, but each temporal worker pulls full MERRA-2 daily granules (hundreds of MB) to read one storm's bbox sliver. The pre-zagg orchestrator (jbbutler/antarctic_AR_dataset#1) averaged ~7 MB/storm with range reads; the first post-#221 slice run will measure what whole-file fetching costs by comparison — that number sizes the payoff here.

What hidefix already provides (verified against the local checkout, v0.13.0)

• src/reader/s3.rs: async chunk-range S3 reader — ranged GETs, up to 16 concurrent, adjacent chunks coalesced into single requests, any S3-compatible endpoint with caller-supplied credentials. Exactly the read model needed.
• Codecs: gzip/deflate + shuffle — covers MERRA-2 .nc4.
• Serializable chunk indexes — the same machinery zagg's spatial sidecar index backend already consumes through h5coro-hidefix.

The gaps (all in the Python/xarray glue, python/hidefix/xarray.py)

1. The xarray entrypoint is local-path only; the S3 reader is unreachable from engine="hidefix".
2. No prebuilt-index injection — upstream has the exact TODO in place: # TODO: allow take an existing index, maybe from a serialized object.
3. HidefixDataStore opens netCDF4.Dataset alongside the hidefix index for metadata/attributes — adopting the engine as shipped would put libnetcdf and libhdf5 back in the layer (heavier than the h5py it replaces).

Two routes (design fork — hence discuss)

A. Upstream the glue (three well-scoped PRs to gauteh/hidefix): accept (s3 url, serialized index) in the engine; implement the serialized-index TODO; derive metadata from hidefix's own index instead of netCDF4-python. Cleanest long-term (one engine, everyone benefits); waits on upstream review.

B. zagg-side adapter: a thin temporal reader over the Rust S3Reader — per-granule chunk indexes built orchestrator-side and distributed alongside the granule URLs, workers do coalesced range GETs. Mirrors the spatial sidecar design one-to-one; waits on nobody; adds a zagg-owned read path to maintain.

A hybrid is plausible: (B) now behind the READERS registry (reader: hidefix_s3), fold into (A) as upstream lands.

Open questions

1. Route A, B, or hybrid?
2. Index placement: per-granule chunk-index sidecars hung off the temporal granule catalog (the downstream repo's phase-B index is the natural host) vs. inline in the event payload?
3. Payoff gate: wait for the slice-run bytes-per-storm measurement before committing, or proceed on the PR-1 ~7 MB/storm prior?
4. Layer end-state: once chunk-range reads land, do h5py/h5netcdf leave the layer (mask files could also ship as indexed NetCDF or Zarr), or stay as the fallback engine?

Refs: #220, #221 (layer state + measured sizes), #218/#219 (shared 250 MB budget; async-tiff precedent for lean Rust readers on the fleet), gauteh/hidefix (S3Reader + the serialized-index TODO).

---

Amendments (maintainer-ratified; see thread)

• Route A ratified — three upstream PRs to gauteh/hidefix: (1) wire the merged S3Reader (Add reader for S3 (picks up #26) gauteh/hidefix#47 — Rust-level only today) through the Python bindings + xarray entrypoint; (2) implement the serialized-index TODO; (3) capture attributes/dimension metadata in the Rust index so the engine drops netCDF4-python entirely (it is a metadata stand-in, not a requirement). Route B is dropped — it was the h5coro-hidefix spatial pattern cloned onto the temporal path, and serves only as design precedent now.
• Index placement ratified — mirror the spatial pattern: inline and sidecar options through zagg's existing index-backend mechanism and config templates.
• Remaining open question: the payoff gate (question 3) — the post-Add xarray/h5netcdf/h5py to the Lambda layer (issue #220) #221 slice run's bytes-per-storm measurement lands soon either way.

[espg]

The xarray entrypoint is local-path only; the S3 reader is unreachable from engine="hidefix".

...didn't we already add this in gauteh/hidefix#47 ?

No prebuilt-index injection — upstream has the exact TODO in place: # TODO: allow take an existing index, maybe from a serialized object.

sounds like something we should add / do

HidefixDataStore opens netCDF4.Dataset alongside the hidefix index for metadata/attributes — adopting the engine as shipped would put libnetcdf and libhdf5 back in the layer (heavier than the h5py it replaces).

this sounds like the biggest issue to me... does it need to do this? Could we modify it to be better / lighter??

A. Upstream the glue (three well-scoped PRs to gauteh/hidefix): accept (s3 url, serialized index) in the engine; implement the serialized-index TODO; derive metadata from hidefix's own index instead of netCDF4-python. Cleanest long-term (one engine, everyone benefits); waits on upstream review.

this is clearly what we want to do; last upstream review was very quick.

B. zagg-side adapter: a thin temporal reader over the Rust S3Reader — per-granule chunk indexes built orchestrator-side and distributed alongside the granule URLs, workers do coalesced range GETs. Mirrors the spatial sidecar design one-to-one; waits on nobody; adds a zagg-owned read path to maintain.

don't we already do this?? isn't that what h5coro-hidefix is??

Index placement: per-granule chunk-index sidecars hung off the temporal granule catalog (the downstream repo's phase-B index is the natural host) vs. inline in the event payload?

match zagg's existing pattern-- options for both inline and sidecar. We already have the mechanism in setup for the yaml templates...

[espg]

🤖 from Claude

Point-by-point, after verifying against upstream gauteh/main (the local checkout is current with it):

gauteh/hidefix#47 — partially. It merged (2026-07-02) the Rust S3Reader (src/reader/s3.rs + Rust tests); its src/python.rs change is only GIL-initialization build plumbing (prepare_freethreaded_python), not an S3 export. On upstream main today, neither the Python module nor the xarray entrypoint can reach the S3 reader — python/hidefix/xarray.py still normalizes a local path. So gap 1 stands, but narrowed and easier: the hard part (the reader) is merged; what's left is wiring it through the Python bindings and the entrypoint.

Serialized-index TODO — agreed, we implement it upstream.

Does the engine need netCDF4-python? No — and the fix is the right kind of light. The dependence exists because hidefix's Rust index parses chunk layout but not attributes (no attribute handling anywhere in src/idx/), so xarray.py needs some metadata source and reaches for nc.Dataset (global attrs, per-variable attrs via getncattr). The indexer already walks the HDF5 object headers where attributes live, so capturing attrs/dimension metadata during that walk is incremental Rust work — after which the engine is fully self-contained: no libnetcdf, no libhdf5, no h5py anywhere. That's upstream PR three, and yes, it's the one that matters most.

"isn't B just h5coro-hidefix?" — it's the same pattern, different pipeline: h5coro-hidefix is pure-Python metadata parsing (h5coro) + hidefix chunk decode behind the spatial sidecar index backend. The temporal/xarray path never goes through h5coro, so route B was "clone that pattern for the temporal reader." With route A ratified, B drops — its only remaining value is as the design precedent the upstream work imitates.

Ratified (body amended)

• Route A: three upstream PRs to gauteh/hidefix — (1) expose the merged S3Reader through the Python bindings + xarray entrypoint; (2) serialized-index injection (their TODO); (3) attributes/dims into the Rust index, dropping netCDF4-python from the engine.
• Index placement: mirror zagg's existing spatial pattern — inline and sidecar options through the same index-backend mechanism already wired into the config/templates.

Next step on my side: prepare the three upstream PRs as branches + draft PR text for maintainer review before anything posts to gauteh/hidefix (per the outbound-review practice for non-espg repos).

[espg]

and the fix is the right kind of light. The dependence exists because hidefix's Rust index parses chunk layout but not attributes (no attribute handling anywhere in src/idx/), so xarray.py needs some metadata source and reaches for nc.Dataset (global attrs, per-variable attrs via getncattr). The indexer already walks the HDF5 object headers where attributes live, so capturing attrs/dimension metadata during that walk is incremental Rust work — after which the engine is fully self-contained: no libnetcdf, no libhdf5, no h5py anywhere. That's upstream PR three, and yes, it's the one that matters most.

fantastic.

[espg]

🤖 from Claude

Upstream plan (maintainer-directed): one PR to gauteh/hidefix, three phases. Grounded in the current source: the indexer walks files through the statically-linked hdf5-metno crate (src/idx/index.rs), so attribute capture happens exactly where the existing walk is; index serialization already exists at the Rust level (src/idx/serde.rs), so phase 2 is largely Python exposure; the S3 reader merged in gauteh/hidefix#47 is Rust-only, so phase 3 is wiring, not reader work.

Phase 1 — metadata into the index; engine drops netCDF4-python

• Rust: during Index::index_file's group walk, capture global + per-dataset attributes and NetCDF dimension metadata (DIMENSION_LIST/dimension-scale attrs) into the Index/Dataset structures, serde-derived so they ride the serialized form.
• Python: expose attrs/dims through the bindings; python/hidefix/xarray.py builds variables and attributes from the index alone — import netCDF4 deleted, dependency dropped from pyproject.toml.
• Tests: engine-equality vs the netcdf4 engine over the existing test corpus (values, attrs, dims, CF decode incl. time units).
• This phase alone makes the engine self-contained (no libnetcdf/libhdf5 at read time) — the gap that matters most for consumers with size-gated deployments.

Phase 2 — serialized-index save/load in Python (the in-code TODO)

• Expose the existing Rust serde through PyO3: Index.save(path) / Index.load(path_or_bytes), and open_dataset(..., index=...) on the entrypoint (resolving the # TODO: allow take an existing index in xarray.py).
• A version + source-file fingerprint (size/mtime or checksum) in the serialized form so a stale index fails loudly, not wrongly.
• Tests: round-trip, open-with-prebuilt-index equality vs fresh index, stale-index rejection.

Phase 3 — S3 through Python and the xarray entrypoint

• Expose the ATL03 template + region benchmark + hierarchical-read wiring (Refs #30, #43) #47 S3Reader via PyO3; the entrypoint accepts s3:// URIs with a required prebuilt index (indexing itself needs the hdf5 crate against a local file — the orchestrator-builds/worker-reads split is the intended model) plus bucket config (region, endpoint, anonymous/ambient credentials).
• guess_can_open handles s3://…​.nc4; local-path behavior unchanged.
• Tests: the minio docker harness from ATL03 template + region benchmark + hierarchical-read wiring (Refs #30, #43) #47 (tests/docker-compose.minio.yml), extended through the xarray engine; concurrency/coalescing exercised on a chunked file.

Validation gate for the whole PR

Bit-identical reads vs the netcdf4 engine on the corpus plus one real MERRA-2 .nc4 granule (deflate+shuffle, the zagg temporal workload), local and via minio-S3.

Process

Implemented on a branch of the espg/hidefix fork; PR title/body and any upstream comments are drafted for maintainer review before anything posts to gauteh/hidefix, per standing practice for external repos. zagg-side adoption (a hidefix temporal reader, index distribution through the ratified inline/sidecar mechanism, and eventually reversing #221's h5py addition) is follow-on work here, out of the upstream PR's scope.

[espg]

🤖 from Claude

The payoff-gate measurement (open question 3) is in. First successful fleet slice (10 storms × 2 configs, post-#225/#226 subset-and-load workers): ar_attributes 9 events / 173 s wall / ~$0.0363 (2720 GB-s); ar_precip 9 events / 108 s / ~$0.0275 (2065 GB-s) — ≈ $0.0071 per storm across both runs, extrapolating to ≈ $60 for the full 8,714-storm catalog. The pre-zagg orchestrator's range-read baseline (jbbutler/antarctic_AR_dataset#1) did the full catalog for $11.73 — so whole-granule fetching costs roughly 5× even after #225 bounds the memory (workers still download whole granules; they just don't hold them). That is the concrete case for chunk-range reads: parity in capability at ~$12 vs ~$60 per full-catalog pass, growing with every hyperparameter variant swept (19 catalogs shipped). Slice storms skew short, so treat 5× as a floor estimate.

[espg]

🤖 from Claude

Cost addendum — the memory knob changes the calculus. CloudWatch REPORT lines from the successful slice show post-#226 workers peaking at 182–230 MB on a 4096 MB function — ~94% of the GB-seconds bill is unused headroom sized for the pre-#225 whole-granule-holding behavior. Right-sizing the function (operator knob; suggest an A/B slice at 1024 and 2048 MB, expecting the sweet spot near 1769 MB where Lambda grants a full vCPU) should cut the ~$60/full-catalog estimate to roughly $15–25 without any code change — near the range-read baseline's $11.73. The chunk-range work here then targets what memory sizing cannot: ~5× transfer volume, wall time, the double-fetch of M2T1NXINT across the two-config design, and the h5py layer exit — compounding across the 19 hyperparameter catalogs.