research: bootstrap PyAutoReduce — HST/ACS reduction design + spike

[Jammy2211]

Overview

Bootstrap PyAutoReduce — a new PyAutoLabs library that reduces archival HST imaging into the modeling-ready datasets PyAutoLens/PyAutoGalaxy load via al.Imaging.from_fits (data.fits, noise_map.fits, psf.fits, psf_full.fits + provenance). Phase 1 targets ACS/WFC at SLACS quality: drizzled 0.05″/pix mosaics, tiered PSF estimation, RMS noise maps with Poisson counts, and a disk-frugal download→reduce→evict flow. This is a research task: the deliverables are the repo, the design docs, and a de-risking spike — production implementation follows as a separate prompt.

Plan

• Create and register the PyAutoLabs/PyAutoReduce repo (package autoreduce) in the organism.
• Research SLACS / weak-lensing / AGN literature and STScI tooling to ground the design.
• Write design doc A — the thorough HST/ACS pipeline plan (acquire / align / drizzle / noise / PSF / package / validation), defaults-first with documented lensing deviations.
• Write design doc B — roadmap skeleton (WFC3, JWST, per-exposure _flt products, instrument adapters).
• Run a spike: one SLACS lens end-to-end (MAST → AstroDrizzle → noise map → PSF → cutout) compared against the existing modeling dataset.
• File one follow-up implementation prompt when the design is validated.

Detailed implementation plan

Affected Repositories

• PyAutoReduce (primary, new)
• PyAutoMind (registry: repos.yaml, active.md)

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoReduce	main (new repo)	clean
./PyAutoMind	main	clean

Suggested branch: work lands on main directly — the repo is new, pre-release, with no downstream consumers; feature-branch/PR conventions begin with the phase-1 implementation task.

Implementation Steps

1. gh repo create PyAutoLabs/PyAutoReduce (public, MIT); clone to ~/Code/PyAutoLabs/PyAutoReduce.
2. Register in PyAutoMind/repos.yaml (category: library) and run scripts/repos_sync.py --write so routing tables include the repo.
3. Package skeleton: autoreduce/{acquire,align,drizzle,noise,psf,package,instruments}/, test_autoreduce/ (numpy-only import tests), pyproject.toml (core deps astropy/astroquery/photutils; heavy STScI stack behind an [hst] extra; PSF back-ends behind [psf]), README, AGENTS.md/CLAUDE.md.
4. docs/design/hst_acs_pipeline.md: per-stage default-vs-deviation design — astroquery.mast _flc acquisition with transient size-capped cache + HAPCut path for survey mosaics; a-priori WCS with TweakReg as a triggered fallback; AstroDrizzle (final_scale=0.05, north-up, final_wht_type=IVM, pixfrac/kernel fixed by parity study); noise map σ=√(N/W+σ²_sky) with correlated-noise correction (Bayer et al. 1803.05952, DrizzlePac handbook); tiered PSF (photutils ePSF → TinyTim+focus per Rhodes et al. 2007 → STARRED/PSFr optional), always drizzle-consistent; packaging with intact WCS/provenance headers.
5. docs/design/roadmap.md: instrument-adapter obligation, WFC3 (hst-lens notebooks as checklist), JWST (calwebb + STPSF), per-exposure frame products, Euclid explicitly out of scope.
6. Spike prototypes/slacs_f814w_spike.py: one SLACS lens end-to-end; data/noise/PSF ratio comparison against /mnt/c/.../slacs/<lens>/; resolves the units question (e-/s vs e-) and the correlated-noise question empirically.
7. Review of design docs + spike results, then file feature/pyautoreduce/hst_acs_phase1.md as the single follow-up prompt.

Key Files

• docs/design/hst_acs_pipeline.md — authoritative phase-1 design
• docs/design/roadmap.md — later phases skeleton
• autoreduce/instruments/ — adapter boundary all later instruments depend on
• prototypes/slacs_f814w_spike.py — de-risking spike

Original Prompt

Click to expand starting prompt

I want to make a new project, PyAutoData (but feel free to suggest better names), which performs all data reductions required for PyAutoLens modeling (and also PyAutoGalaxy). First, the focus will be on HST data reduction, making images comparable to the data I am used to modeling (e.g. /mnt/c/Users/Jammy/Science/subhalo/dataset/slacs). It should therefore drizzle the individual exposures to a common mosaic, do PSF estimation, have an RMS noise map including Poisson counts. Future work will include reducing other HST data (e.g. other wavelengths, WFC3), JWST data and other instruments, and producing individual frames for modeling (e.g. _flt images with cosmic rays). My understanding is you will have to download large HST tiles, but ultimately we only need cut-outs of each strong lens, so the project should have tools which exploit this to avoid too much hard disk space being used to store lots of tiles where possible. Read some strong lensing literature to work out how HST data reduction is performed; this github repo may give guidance albeit I think their reduction pipeline is not as high quality as SLACS (https://github.com/ajshajib/hst-lens) and it may also not do ACS. Do deep research in planning this, make a thorough plan for the HST steps and then a follow-up which we will flesh out later to do JWST (and other instruments). Where possible, we should stick to the default data reduction pipeline of each instrument and only add changes or deviations or customizations when required for specific lensing calculations. You will also benefit from trying to find science publications on weak lensing or AGN modeling which describe detailed PSF modeling.

(Name decided in intake: PyAutoReduce.)

[Jammy2211]

Milestone 1 — repo bootstrapped, design docs in.

The repo now exists with the autoreduce package skeleton (stage subpackages acquire/align/drizzle/noise/psf/package + an instruments/ adapter boundary) and the two design docs:

• docs/design/hst_acs_pipeline.md — the thorough phase-1 plan. Defaults-first (astroquery.mast → AstroDrizzle → photutils), with every lensing deviation tabulated and justified: 0.05″/pix north-up mosaics, IVM weights, the Bayer et al. (1803.05952) noise recipe σ=√(N/W+σ²_sky) plus correlated-noise correction, and a three-tier PSF strategy (ePSF stars → TinyTim+focus per COSMOS weak lensing → STARRED/PSFr for quasars later). The drizzled-PSF invariant is called out explicitly.
• docs/design/roadmap.md — WFC3/JWST/per-exposure-frame skeleton (to flesh out later, one prompt at a time).

Registered in the organism (repos.yaml + routing tables). Next: the de-risking spike — reduce slacs0008-0004 from MAST end-to-end and compare data/noise pixel-for-pixel against the legacy modeling dataset, which settles the units and correlated-noise questions empirically.

[Jammy2211]

Milestone 2 — spike ran end-to-end; the parity numbers close.

Reduced slacs0008-0004 from MAST (7 FLC exposures, prog 10886, 3132s) through CRDS sync → AstroDrizzle (0.05″/pix, north-up, IVM, pixfrac 0.8, cps) → noise map → 281×281 cutout, and compared pixel-for-pixel against the legacy modeling dataset:

• Units: e-/s confirmed — bright-pixel data ratio 0.934 (an electrons mismatch would have read ~3132×).
• Noise recipe: matches once correlated noise is applied. Raw noise ratio 0.678; × R=1.364 (Casertano factor for pixfrac 0.8) → 0.924 ≈ the data ratio. The legacy SLACS noise maps behave as if correlated-noise-corrected, so stage 4 applies R as designed.
• Residual ~7% global flux offset on both data and noise — suspects are the exposure set (the spike stacked neighbouring-pointing exposures the legacy reduction surely excluded) and sky subtraction; chased in phase 1 with the exact legacy exposure set + sub-pixel WCS registration.
• Tier-1 ePSF plausible here (236 point-like >10σ detections mosaic-wide, before strict star selection).
• Two acquire-stage lessons folded into the design doc: CRDS reference files (jref$) are part of acquisition (IVM needs the flats), and coordinate queries must filter HAP skycell products (31 product rows → 7 real files, plus unwanted neighbouring pointings).

Design doc updated with a parity appendix (commit 4abd7e1). Environment note: the reduction stack (drizzlepac/astroquery/photutils/crds) now lives in the main PyAuto venv, installed under constraints so the astropy/scipy/numpy pins didn't move.

Next: design-doc review, then the single phase-1 implementation prompt.

[Jammy2211]

Milestone 3 — design approved; research task complete.

Design docs signed off. Two late literature findings made it in first: SLACS V never drizzled its snapshot data (bilinear ACSPROC rectification + L.A. Cosmic + identically-rectified TinyTim PSF — Bolton et al. 2008 §2), and published pixfrac choices span 0.6–1.0 across follow-ups, so final_pixfrac/final_kernel are now user-facing dials with the WHT-uniformity diagnostic and correlated-noise factor R reported on every run (spike: RMS/median 0.066 at pixfrac 0.8).

The phase-1 implementation prompt is filed as PyAutoMind/feature/pyautoreduce/hst_acs_phase1.md — scope: productionize the validated pipeline into the autoreduce stage modules behind the ACS/WFC instrument adapter, with the parity acceptance test (2–3 SLACS lenses, sub-pixel registration, exact legacy exposure sets, PyAutoLens fits agreeing on both reductions; the spike's ~7% flux offset gets grounded there).

This issue tracked the research/bootstrap task and can close when the maintainer is happy; phase 1 gets its own issue via /start_dev.