hidefix adoption: upstream index save/load PR + dependency discussion (compiled reads, #148 offset cache)

[espg]

🤖 from Claude

Opened at espg's direction on #149 (comment). Decision recorded there: hidefix is the intended mechanism for both the compiled read step and #148's phase (2b) chunk-offset cache — this issue is the dependency discussion (per the repo's add-a-dependency convention) plus the upstream work it's contingent on.

Evidence (PR #150, bench/h5coro/results/REPORT.md + hidefix_spike_notes.md)

• Byte-identical on the full captured worker workload (sha256 gate, o9 + o10, incl. the 2-D shuffled int8 signal_conf_ph, native hyperslices): 15.5 s (o10) / 20.8 s (o9) vs 93.5 / 129.0 baseline, at ~55–60% of baseline RSS.
• Those rows include in-process chunk-index construction (~10.6 s of the o10 15.5 s, ≈0.2 s/granule). With a pre-built serialized index the replay is ~4.9 s / ~7.9 s → ~19× / 16×, faster than the C++ shim ceiling (10.3 / 14.2 s) because the shim re-parses HDF5 metadata every run. The hidefix rows are a floor, not a cap.
• The serialized index (~600 KB per ~1.9 GB granule, bincode/flexbuffers) carries exactly the per-chunk (addr, size, offset) + dtype/shape/filter payload add 88 south stress test shard + benchmark #148's offset cache needs, as public documented crate types.

Blocker: the Python binding surface

The binding today cannot save, load, or enumerate the index (pickle raises TypeError; no save/load methods) — the crate itself supports serialization (the hfxidx CLI proves it). Upstream PR needed: expose index serialize/deserialize (+ ideally chunk enumeration) in the Python binding. Small and self-contained.

Dependency discussion (expanded here, per the #149 thread)

1. License — ambiguous upstream; needs clarification (see the correction comment below): the repo's LICENSE file is MIT (added 2023, what GitHub shows), while Cargo.toml/crates.io/PyPI metadata declare LGPL-3.0-or-later on every release incl. 0.12.0. Ask upstream to reconcile. If MIT, this item evaporates; if LGPL, dynamically-linked Python extension use is the normal-case-fine reading, but the Lambda-layer redistribution posture should be confirmed deliberately.
2. Transitive py-deps — the pip package hard-depends on xarray/netcdf4/pandas. pandas is already pinned in the layer; xarray + netcdf4 are new weight that must be measured against the 250 MB dual-arch gate (or upstream could loosen them to extras).
3. No linux-aarch64 wheel on PyPI — source build with pinned Rust in the layer pipeline (working recipe: bench/h5coro/Containerfile.hidefix, --locked, maturin pinned) or vendor a wheel the way the spherely fork is handled.
4. S3 read story — the binding is local-file only. Options: (1) upstream a range-reader/S3 driver, (2) an obstore-backed reader feeding hidefix's decode (obstore already ships in the layer), (3) staged/presigned local reads. Needs a spike; this is the main unknown between "benchmarks well locally" and "works in the Lambda worker".
5. Maintenance posture — single-maintainer crate (gauteh), MIT-era activity is healthy; upstream receptivity to the save/load PR is the cheapest early signal.

Proposed sequence

1. Upstream index save/load PR (unblocks everything; also de-risks (5))
2. S3 read spike against the captured-workload harness (PR h5coro read-path benchmark: profile + 4-variant comparison (issue #149) #150's bench/h5coro/)
3. Dependency sign-off in this thread
4. Wire into add 88 south stress test shard + benchmark #148 phase (2b): extraction produces per-granule indices at catalog time; workers read chunks with zero HDF5 metadata I/O

Refs #149, #148; evidence in PR #150.

(Left unlabeled for espg to triage.)

[espg]

🤖 from Claude

Correction on item (1), prompted by espg checking the repo: the license situation is ambiguous upstream, not cleanly LGPL.

• The repo's LICENSE file is MIT (added 2023-01-17, 81b198a, never modified) — this is what GitHub's sidebar shows.
• The Cargo.toml license field is LGPL-3.0-or-later on every published release from 0.1.0 (2020-08) through 0.12.0 (2024-10) — including releases made after the MIT file landed — and that's what crates.io and the PyPI wheel metadata declare (where this issue's original "LGPL-3.0" came from).

So the distributed artifacts self-describe as LGPL while the repo carries an MIT license text. Plausibly a 2023 relicense to MIT with a stale metadata field, but that's inference, not license. Resolution path: ask upstream to reconcile the two (a one-line Cargo.toml fix in whichever direction is intended) — natural to raise alongside, or ahead of, the index save/load PR, and it doubles as the maintenance-receptivity check from item (5). If the answer is MIT, item (1) evaporates; if LGPL, the dynamic-linking analysis stands as written.

[espg]

🤖 from Claude

Folding in the packaging discussion with espg. Ground rule established there, from the spherely-fork experience: off-PyPI distribution is disqualifying — hard to build in CI/CD, hard to distribute, hard to use; the spherely fork sits unused, gated on an upstream merge. Everything below assumes pip install from PyPI or it doesn't count.

Key reframe: ownership without vendoring

hidefix is a published crates.io crate. A binding package declares hidefix = "0.12" in Cargo.toml (pinned by Cargo.lock) the way Python pins a PyPI dep — no source is vendored. And the save/load gap needs no upstream changes at all: the index's serde impls are public crate API (that's precisely how the hfxidx CLI serializes it — the spike used it), so Index.save()/.load() plus chunk enumeration for #148 land in our binding layer.

Measured for scale (hidefix main @ 0.12.x): the whole crate is ~4.8k LOC of Rust (idx 1,659 / reader 666 / extent 538 / filters 468 / python bindings 295 / tests+benches ~1,050). We would own none of it — the owned surface is:

piece	size
pyo3 bindings: open/read→numpy, hyperslabs, save/load (bincode), chunk enumeration	~300–500 LOC Rust (upstream's python.rs is 295)
pyproject.toml + Cargo.toml (maturin, abi3-py39 → one wheel per platform)	~60 lines
CI: maturin-action matrix — manylinux x86_64 + aarch64, macOS arm64 → PyPI	~80 lines (bench/h5coro/Containerfile.hidefix in PR #150 already proved the aarch64 build, static libhdf5 included)
h5coro-side optional-import glue + h5coro[turbo] extra	~100–200 LOC pure Python

Under 1k owned lines, everything pip-installable.

Recommended shape: companion package in the SlideRuleEarth org

h5coro lives in the SlideRuleEarth org, so the companion package would be created there (and, if it ever comes to vendoring, vendored there too): a small h5coro-hidefix (name TBD) repo/package next to h5coro. h5coro itself stays pure-Python noarch — its stated identity and its conda-forge feedstock are untouched — and gains an optional import plus an extra. espg has commit/release access in the org, so unlike spherely nothing is gated on an external maintainer, while the artifact is a normal PyPI wheel the Lambda layer and CI consume directly.

Rejected shapes, for the record:

1. Compile the extension into the h5coro package itself — flips h5coro from one noarch wheel to a platform-wheel matrix for every user, disrupts the conda-forge feedstock, and puts a Rust subsystem inside a project whose README defines it as the pure-Python re-implementation — an identity change needing org buy-in, all to avoid a second tiny package.
2. Hard fork of hidefix — same wheel-CI burden as the companion, plus ~3.5k LOC of ownership and crates.io divergence, for nothing the binding crate doesn't already give. cargo vendor remains the one-command escape hatch if upstream ever dies — an escape hatch the spherely situation never had.

Upstream PRs to gauteh/hidefix (aarch64 wheels, xarray/netcdf4→extras, the MIT-vs-LGPL metadata fix above) stay worth filing — but nothing gates on them.

Updated sequence

1. Prototype the SRE-org binding crate (gated on nothing): pyo3 layer incl. save/load, abi3 wheels for manylinux x86_64/aarch64 + macOS, validated against PR h5coro read-path benchmark: profile + 4-variant comparison (issue #149) #150's captured-workload harness (byte-equality gate) — cached-index replay should land ~5–8 s on the o10 workload (~19× effective).
2. S3 read spike (the remaining unknown; same options as before — upstream driver / obstore-fed buffers / staged reads).
3. Dependency sign-off here (license clarification folds into the upstream engagement).
4. Wire into add 88 south stress test shard + benchmark #148 phase (2b): extraction emits serialized indices at catalog time.

The C++ alternative — still fastest uncached at 9.1× — is now outlined separately in #156 for comparison; its structural costs (vendoring with no dependency mechanism, manylinux static-linking, per-version wheels, no external-index entry point) are why this route stays primary.

[espg]

🤖 from Claude

Closing at espg's direction — every substantive item here resolved, and the structural decision moved to #160:

1. License — resolved upstream as MIT (LICENSE file (MIT) and Cargo.toml license field (LGPL-3.0-or-later) disagree gauteh/hidefix#48, metadata fixed in Update Cargo.toml license metadata to MIT gauteh/hidefix#49, merged).
2. Binding surface — shipped in espg/h5coro-hidefix (save/load, chunk enumeration, read_plan/read_from_buffers, from_chunks; abi3 wheels incl. manylinux-aarch64, CI green).
3. S3 read story — solved twice over: the obstore-fed buffer path in the binding, and the upstream S3 reader (Add reader for S3 (picks up #26) gauteh/hidefix#47, merged, with the aarch64 wheel matrix riding it).
4. Dependency posture — superseded by Virtual index entry point: pluggable chunk-index backends for the read path #160's entry-point design: the sidecar backend ships in h5coro-hidefix, so zagg core never takes the dependency; the layer either includes the wheel or the backend isn't discoverable.

Still tracked elsewhere: PyPI namespace registration + first public release (espg, in the binding repo), and the #148 (2b) wiring (#160).