examples: SDSS spectral reconstruction (spPlate, train in place, no reshard)

[emfdavid]

An astroML cross-domain example: reconstruct SDSS galaxy spectra streamed in place from real
spPlate FITS, with no reshard. The decode-amortization/streaming companion to the Hubble example.

What it does

astroML's spectral-PCA workflow (fetch_sdss_corrected_spectra → compute_sdss_pca) first
downloads the raw archive and resamples every spectrum onto a common grid into one spec4000.npz —
the download-and-reshard step insitubatch argues against. examples/sdss instead indexes real
spPlate frames on data.sdss.org (HTTPS) as virtual references: one plate holds all ~640
fibers on a common log-wavelength grid as a single (fiber, wave) image, which is virtually
re-chunked along the fiber axis by pure byte arithmetic — no pixels moved.

The task mirrors astroML's spectral reconstruction: a 1-D convolutional autoencoder, trained by
SGD over the streamed mini-batches, beats a PCA baseline at the same latent dim — because galaxy
spectra are translation-structured (varying redshift shifts the lines, which a fixed-dim linear
basis reconstructs poorly). The offline synthetic default (redshift-shifted spectra) carries the
beats-PCA claim and backs the drift test; the real run is a same-pipeline, real-archive demo.

Two build modes (--plates N) — both rewrite only the chunk manifest, no pixels moved

• one plate → full wave width, many fibers per chunk (the O(chunks) decode-amortization regime).
• many plates → every SDSS plate shares dloglam and their COEFF0 start wavelengths differ by
  whole bins, so cropping each to the shared window aligns them onto one grid exactly (no
  resampling) and folds (plate, fiber) into one flat sample axis, one fiber per chunk (the
  streaming regime — the only no-reshard way to concat the ragged-width plates, scaling to the
  full ~2800-plate archive). Verified: 3 plates → 1920 spectra byte-identical to astropy.

The modes are mutually exclusive by construction: cropping to the common window breaks the
multi-fiber contiguity the single-plate mode rides on — an honest boundary of the FITS byte layout.

FITS binary tables

SDSS spectra also ship as FITS binary tables (structured big-endian dtypes).
tests/test_fits_bintable.py validates that path end-to-end through insitubatch (kerchunk →
VirtualiZarr → Icechunk → insitubatch) against astropy, and the astronomy extra pins the
VirtualiZarr fix it needs (VirtualiZarr#1037,
open on a fork) — released VirtualiZarr raises on the list-valued kerchunk field specs. Endianness
is dropped in the VZ/icechunk struct metadata, so consumers reinterpret each field with .view('>f4')
at the numpy boundary (the zarr #4142 codec fix does not engage in this path, so it is not pinned).

Tests / gates

tests/test_sdss.py (synthetic drift guard, AE beats PCA), tests/test_sdss_build.py (both build
modes offline, exact cross-plate alignment, fail-fast on non-bin-aligned grids), and
tests/test_fits_bintable.py. Full suite 178 passed / 10 skipped; ruff + format + mypy clean.

Note: the astronomy extra pins the VirtualiZarr fork via [tool.uv.sources] (dev-environment
only — stripped from the published wheel). Repoint to VirtualiZarr main once #1037 merges, then to
a release. SDSS is HTTPS-only (no S3 mirror), so throughput benchmarking co-location is still TBD.

🤖 Generated with Claude Code

[martindurant]

Since you pick a concrete example with real data, this is clearly what at least some astronomers do :)
My experience is pretty outdated here, and I was only ever looking at individual objects anyway.

Is there an intermediate representation of the chunk/manifest that can be persisted in a public place? If the workflow were structured as 1) scan 2) use, then interested people could just do the latter with any benchmarking they care to do. I understand its not quite just a vzarr manifest if you need to do extra work like numpy .view() calls.

Will you make this a published blog or something?

[emfdavid]

Good question, and it's the right shape — scan-once / use-many is already how the example is structured, and Icechunk is the persistence layer that makes the "scan" output publishable.

The one thing I want to keep central is that this stays anchored to the tools astronomers already validate with. astropy is the ground truth here: the build checks byte-identical against astropy.io.fits (3 plates → 1920 spectra, exact), and that comparison — not the manifest — is how you know a streamed batch is correct. A persisted Icechunk manifest accelerates the use step; it never replaces astropy as the reference. (The .view('>f4') you flagged is part of staying byte-exact to astropy at the numpy boundary — a shim, not a divergence. VZ #1047 + #1048, both mergeable, remove the need for it without changing the astropy-validated result — and it only affects the FITS binary-table path; the spPlate image path is already clean.)

On persistence itself: the committed Icechunk store isn't an extra step — it's a one-time side effect of the first build (kerchunk → VirtualiZarr → Icechunk → insitubatch). The scan writes it once — virtual refs, no pixels moved — and everything downstream already reads from it. So "publish the manifest" just means dropping that artifact in a public bucket: a consumer opens it and trains/benchmarks with no re-scan and no astropy in the loop. The raw FITS is HTTPS-only (no S3 mirror), but the manifest is tiny and location-independent.

Where it's headed makes the "what for" concrete: the eventual target is a PR to astroML adding insitubatch recipes that hook into its existing SDSS workflow (fetch_sdss_corrected_spectra / compute_sdss_pca) — train-in-place as an option inside the ecosystem astronomers already use, validated against the same astropy/astroML baselines, rather than a stack off to the side.

And yes — blog post committed, kicking off with a Pangeo Showcase talk (~September), with this SDSS case as one of the cross-domain examples.