feat: weak lensing real-data support — catalog IO, reduced shear, A2744 (weak series step 7)

[Jammy2211]

Overview

Step 7, the final leg of the weak-lensing home straight: make WeakDataset consume real shear catalogs and reproduce a published cluster analysis on Abell 2744. Phased because the external-data availability is unresolved: the Harvey & Massey 2024 pyRRG-JWST paper says catalogs live in the pyRRG GitHub repo, but the top level shows code only; Medezinski et al. 2016 (Subaru) is the alternative.

Plan

Phase 7a — library (no external data needed, ships regardless):

• WeakDataset.to_csv / from_csv via the autoconf.csvable machinery, mirroring PointDataset (columns: name,y,x,gamma_2,gamma_1,noise).
• WeakDataset.from_fits — binary-table loader with configurable column names (real catalogs: RA/Dec or x/y, e1/e2, weights).
• weights= convenience on the loaders (inverse-variance weights → σ = w^(-1/2)).
• Reduced shear: real catalogs measure g = γ/(1−κ). WeakDataset gains is_reduced metadata; SimulatorShearYX gains a reduced-shear mode; FitWeak computes the model reduced shear (via the LensCalc Hessian convergence) when the dataset declares it. Round-trip tests: reduced simulator + reduced fit = zero residuals; γ-vs-g differences grow toward the Einstein radius.
• Per-galaxy source redshift storage (optional column) — the demo uses the standard single-effective-source-plane approximation; full per-galaxy Σ_crit scaling is documented as the extension point, not implemented.

Phase 7b — workspace real-data example (data acquisition first):

• Obtain a public A2744 shear catalog: (1) pyRRG repo subdirs/releases, (2) UNCOVER/MAST HLSP products, (3) Medezinski 2016; document provenance in the script.
• scripts/weak/real_data/a2744.py: load via the new interface, fit an NFW(+substructure) mass model with AnalysisWeak, compare the tangential shear profile and mass to the published analysis, show the KS map against the known merger structure.
• Fallback if no catalog is genuinely retrievable: the example becomes a realistic mock in the published catalog's exact format (same columns, depths, n_gal, geometry), exercising the identical loader path, with the swap-in-real-data step documented — and a question parks on this issue for the human to source the data.

Branch: feature/weak-real-data; worktree PyAutoLens + autolens_workspace. Autonomy: --auto continuation 2026-07-09, effective supervised; 7a proceeds now, 7b's data-acquisition outcome is reported here before the example is written.

Original Prompt

Click to expand

Weak lensing real data example on Abell 2744. Reproduce a not-too-complex published cluster weak-lensing analysis end to end on a public shear catalog, as the first PyAutoLens weak-lensing result on real data. Target: Abell 2744 (HST Frontier Fields cluster) — public catalogs include the Medezinski et al. 2016 Subaru/Suprime-Cam analysis (arXiv:1507.03992) and the JWST UNCOVER pyRRG-JWST shear catalog; pick whichever is cleanest to obtain and document provenance. Requires: (1) a WeakDataset catalog loader (from_fits / from_csv: positions, e1/e2, weights, optional source redshifts); (2) reduced shear support — real catalogs measure g = gamma/(1-kappa), not shear, so FitWeak/simulator need a reduced-shear mode, plus sigma_crit / lensing-efficiency scaling for a source redshift distribution; (3) an autolens_workspace/scripts/weak/real_data example fitting an NFW (or NFW+substructures) mass model and reproducing the published tangential shear profile and mass within errors (Oguri 2012 SGAS-style analysis). Depends on AnalysisWeak from feature/weak/4_modeling.md.

[Jammy2211]

Phase 7a ship sign-off + phase 7b data decision (supervised checkpoint — run parked here)

7a — done, committed locally (47b8888b, not pushed), requesting ship approval

WeakDataset.from_arrays / to_csv / from_csv / from_fits (explicit column mapping, weights= → σ=w^(−1/2), optional per-galaxy redshifts stored), and full reduced-shear support: SimulatorShearYX(reduced=True), WeakDataset.is_reduced, FitWeak computes g = γ/(1−κ) via the same Hessian primitive. Loaders default is_reduced=True (real catalogs measure ellipticities); simulator defaults plain γ (matches the earlier tutorials). 7 new tests incl. reduced round-trip and a mislabel guard; suite 364 green. Heart YELLOW, same acknowledged set.

7b — data acquisition outcome (the plan said report before writing the example)

Found and inspected a real A2744 pyRRG shape catalog: trainStarGalClass/TrainingData/abell2744_galaxies.fits on the pyRRG repo's jwst branch (6585 rows; RA≈3.588, Dec≈−30.397 ✓; SExtractor photometry + RRG moments + e1/e2 with per-component errors + star/galaxy prob). Caveats: it ships as star/galaxy-classification training data, and the raw e1/e2 are uncut (range ±167 — needs the standard |e|, error and prob cuts); I cannot verify from the repo alone that these are the final calibrated shears of Harvey & Massey 2024.

Question — which 7b path?

• (a) Recommended: use this catalog with standard quality cuts, framed honestly as "fitting a real A2744 pyRRG shape catalog" (cites Harvey & Massey 2024 for context; compares the NFW mass/profile to published values as a consistency check, not a metrology reproduction). Real data, in hand, exercises from_fits + the quality-cut workflow end to end.
• (b) Hunt the UNCOVER/MAST HLSP products for the published DR1 shear catalog first; fall back to (a).
• (c) Medezinski 2016 Subaru catalog (VizieR); ground-based σ, different systematics.
• (d) Published-format mock (the plan's fallback) — weakest, only if you want zero provenance risk.

On approval of 7a (+ a/b/c/d choice): push + PR the library half immediately; 7b proceeds per your pick, with the workspace example + any follow-up loader tweaks as the second PR.

[Jammy2211]

Shipped — the weak-lensing series is complete end to end 🎉

• 7a feat: WeakDataset catalog IO + reduced shear (weak series step 7a) #589 (merged): WeakDataset catalog IO (from_arrays/to_csv/from_csv/from_fits), weights=, per-galaxy redshift storage, and full reduced-shear support (simulator → dataset flag → FitWeak model g = γ/(1−κ)).
• 7b feat: real-data A2744 weak lensing example (weak series step 7b) autolens_workspace#253 (merged): scripts/weak/real_data/a2744.py — the first PyAutoLens weak-lensing analysis of real sky data, per the human's option (a): the public pyRRG A2744 catalogue (commit-pinned download, provenance documented), standard quality cuts (6585 → 382 usable shapes), reduced-shear dataset, Kaiser-Squires map, NFW Nautilus fit. Framed honestly: the training-data-grade sample yields a marginal (~1.5σ) tangential signal with the correct radial concentration and a null B-mode — the demonstrated workflow transfers unchanged to deep survey catalogues.

Series status: steps 0–9 all shipped (docs, simulator, visualization, fit, modeling, likelihood guide, profiles+KS maps, combined strong+weak, smoke support, real data). Closing.