examples: Hubble FITS denoising (train in place on a non-zarr archive)

[emfdavid]

What

A cross-domain example that trains in place on an archival format that never was zarr: real Hubble WFC3/IR frames of M16 (the Eagle Nebula) on MAST's public S3 bucket (s3://stpubdata, anonymous), indexed as virtual references (VirtualiZarr → kerchunk.fits → Icechunk) and streamed by insitubatch with no reshard.

• examples/hubble/data.py — build the virtual-reference store, the denoise_dataset, the chunk/batch transform stages, and the median-filter baseline.
• examples/hubble/train_torch.py — a tiny residual CNN denoiser.
• tests/test_hubble.py — a drift guard on the insitubatch-facing wiring.

How it maps onto the engine

• sample_axis=0 → each FITS frame is one sample (one image = one chunk).
• Chunk stage (clean_normalize → block-mean Coarsen) runs vectorized on the decode pool; per-sample Gaussian noise lives in the AddNoise batch stage — per the transform-cost contract.
• Honest framing: because a FITS image is one chunk, this is the streaming-in-place / no-reshard value, not the many-samples-per-chunk decode-amortization of the chunked-zarr examples.
• The build-time index stack (virtualizarr/kerchunk/astropy) is needed only to build the store; the train hot path is icechunk + numpy (kerchunk never runs at train time).

Result

A short CPU run has the trained denoiser beat the median-filter baseline (real ≈26.4 vs ≈24.7 dB PSNR).

Testing / offline default

--source synthetic (the default) writes a local zarr with the same SCI (frame, y, x) geometry — sharp point sources a median filter can't preserve, plus NaN bad-pixels to exercise the NaN-scrub — so the example runs with no network or FITS stack and backs tests/test_hubble.py. The test asserts the geometry (one frame per chunk), the (noisy, clean) batch labels + NaN scrubbing, the baseline denoises, and the CNN beats the baseline. ruff, mypy, and the tests are green.

Note: MAST's anonymous bucket throttles (HTTP 503) under heavy concurrent read-ahead, so max_inflight is capped low by default; an authenticated/retrying path on AWS would raise it.

🤖 Generated with Claude Code