From 6adacdd86712936773944295caa6c73cfb2775fb Mon Sep 17 00:00:00 2001 From: Jammy2211 Date: Thu, 9 Jul 2026 17:04:44 +0100 Subject: [PATCH 1/2] docs: complete wiki/core workspace coverage + refresh stale API citations for release Coverage (new sections): LOS-halo simulation machinery (LOSSampler, negative-kappa sheets), sky background via DatasetModel, point-source deblending, custom Analysis subclasses, double-Einstein-ring multi-plane pointer; weak-lensing page rewritten for the shipped FitWeak/AnalysisWeak API (reduced shear, A2744 real-data capstone, strong+weak joint series). Staleness (release audit): removed-Plane-class prose fixed (galaxy_and_plane, tracer, stack pages -> Galaxies); sampler module->package citation moves (nautilus/, dynesty/, emcee/, ...); over_sampling, sensitivity, mass/abstract, basis, csv-api, priors/visualize config layouts re-grounded against installed source; slam_pipeline fictional import replaced with the inline-functions reality; index.md gains missing operations/external entries; llms.txt reference bullets point at the wiki indexes. Validated: audit_skill_apis.py 0 missing/broken (117 symbols); provenance re-stamped (analysis_objects); all Project:path citations + relative links resolve. Issue: https://github.com/PyAutoLabs/autolens_assistant/issues/40 Co-Authored-By: Claude Fable 5 --- llms.txt | 3 +- skills/README.md | 2 +- skills/_style.md | 2 +- skills/al_build_imaging_model.md | 2 +- skills/al_cluster_csv_api.md | 3 +- skills/al_configure_search.md | 6 +- skills/al_custom_profile.md | 2 +- skills/al_debug_fit_failure.md | 2 +- skills/al_prepare_imaging_data.md | 2 +- skills/al_sensitivity_mapping.md | 2 +- skills/al_subhalo_detect.md | 2 +- skills/al_update_wiki.md | 12 ++-- wiki/core/README.md | 4 +- wiki/core/api/analysis_objects.md | 27 +++++++- wiki/core/api/configuration.md | 60 +++++++++--------- wiki/core/api/csv_api.md | 11 ++-- wiki/core/api/light_profile_catalog.md | 4 +- wiki/core/api/searches.md | 32 +++++----- wiki/core/concepts/cosmology_and_units.md | 4 +- .../extra_galaxies_and_noise_scaling.md | 27 +++++++- wiki/core/concepts/galaxy_and_plane.md | 36 ++++++----- wiki/core/concepts/grids_and_masks.md | 6 +- .../concepts/group_and_cluster_lensing.md | 7 +-- .../concepts/inversions_and_pixelizations.md | 4 +- wiki/core/concepts/lensing_basics.md | 6 +- wiki/core/concepts/point_source.md | 26 +++++++- wiki/core/concepts/sensitivity_mapping.md | 6 +- wiki/core/concepts/slam_pipeline.md | 17 ++--- .../concepts/substructure_and_subhalos.md | 26 +++++++- wiki/core/concepts/tracer.md | 33 +++++++--- wiki/core/concepts/weak_lensing.md | 62 +++++++++++++++---- wiki/core/index.md | 29 ++++++--- wiki/core/operations/installation.md | 2 +- wiki/core/stack/autoarray.md | 4 +- wiki/core/stack/autogalaxy.md | 8 +-- wiki/core/stack/autolens.md | 4 +- 36 files changed, 329 insertions(+), 156 deletions(-) diff --git a/llms.txt b/llms.txt index 944f7b6..9d0e6e7 100644 --- a/llms.txt +++ b/llms.txt @@ -26,7 +26,8 @@ Read order: [AGENTS.md](./AGENTS.md) → [skills/README.md](./skills/README.md) ## Reference -- [wiki/](./wiki): curated PyAuto* reference (`wiki/core/`) plus the strong-lensing science wiki (`wiki/literature/`). +- [wiki/core/index.md](./wiki/core/index.md): curated PyAuto* reference — stack, lensing concepts, API catalogues, operations. +- [wiki/literature/index.md](./wiki/literature/index.md): strong-lensing science wiki — concepts, named entities (surveys, lenses), bibliography. ## Runnable examples & tutorials (elsewhere) diff --git a/skills/README.md b/skills/README.md index fc5abfc..a610941 100644 --- a/skills/README.md +++ b/skills/README.md @@ -17,7 +17,7 @@ configured) via symlinks; the canonical files live here. + the understanding to evolve it. Project-workflow skills may instead drive `rsync`, `cp`, or other repo-level operations. - Source citations use the project-name + repo-relative-path form, - e.g. `PyAutoFit:autofit/non_linear/search/nest/nautilus.py`, resolved via + e.g. `PyAutoFit:autofit/non_linear/search/nest/nautilus/`, resolved via [`../sources.yaml`](../sources.yaml). - Wiki references use workspace-relative paths, e.g. `wiki/core/concepts/non_linear_search.md`. diff --git a/skills/_style.md b/skills/_style.md index c6fff17..acd32b0 100644 --- a/skills/_style.md +++ b/skills/_style.md @@ -151,7 +151,7 @@ project's repo root**, resolvable via [`../sources.yaml`](../sources.yaml). Good: -> See `PyAutoFit:autofit/non_linear/search/nest/nautilus.py` for the search's default +> See `PyAutoFit:autofit/non_linear/search/nest/nautilus/` for the search's default > settings, and `wiki/core/api/searches.md#nautilus` for when to pick it. Bad: diff --git a/skills/al_build_imaging_model.md b/skills/al_build_imaging_model.md index 5c366c7..f9ffaae 100644 --- a/skills/al_build_imaging_model.md +++ b/skills/al_build_imaging_model.md @@ -117,7 +117,7 @@ for g in gaussian_per_basis[1:]: lens_bulge = af.Model(al.lp_basis.Basis, profile_list=gaussian_per_basis) ``` -See `PyAutoGalaxy:autogalaxy/profiles/light_linear/` for the linear-profile classes. +See `PyAutoGalaxy:autogalaxy/profiles/light/linear/` for the linear-profile classes. ## Branch — pixelised source diff --git a/skills/al_cluster_csv_api.md b/skills/al_cluster_csv_api.md index 84a5f7a..28cf571 100644 --- a/skills/al_cluster_csv_api.md +++ b/skills/al_cluster_csv_api.md @@ -35,7 +35,8 @@ ranges. > TODO: recipe. Pattern: `model = al.csv.model_from(mass_csv=Path("mass.csv"), > light_csv=Path("light.csv"), point_csv=Path("point.csv"))` (verify -> exact API in `PyAutoLens:autolens/csv/...` — module name may differ). +> exact API in `PyAutoGalaxy:autogalaxy/galaxy/galaxy_model_csv.py`; the +> workspace walkthrough is `autolens_workspace:scripts/cluster/csv_api.py`). ## Branch — adding scaling-tier members diff --git a/skills/al_configure_search.md b/skills/al_configure_search.md index 5babf70..82efba5 100644 --- a/skills/al_configure_search.md +++ b/skills/al_configure_search.md @@ -46,7 +46,7 @@ search = af.Nautilus( ) ``` -Source: `PyAutoFit:autofit/non_linear/search/nest/nautilus.py`. +Source: `PyAutoFit:autofit/non_linear/search/nest/nautilus/`. Knobs to know: - `n_live` — more = more accurate posterior, slower. Start at 200; go to 400+ only if @@ -68,7 +68,7 @@ search = af.DynestyStatic( ) ``` -Source: `PyAutoFit:autofit/non_linear/search/nest/dynesty.py`. +Source: `PyAutoFit:autofit/non_linear/search/nest/dynesty/`. ## Branch — Emcee (MCMC) @@ -85,7 +85,7 @@ search = af.Emcee( ) ``` -Source: `PyAutoFit:autofit/non_linear/search/mcmc/emcee.py`. +Source: `PyAutoFit:autofit/non_linear/search/mcmc/emcee/`. ## Branch — Other searches diff --git a/skills/al_custom_profile.md b/skills/al_custom_profile.md index 691b8c3..38873af 100644 --- a/skills/al_custom_profile.md +++ b/skills/al_custom_profile.md @@ -117,7 +117,7 @@ class CustomNFW(ag.mp.MassProfile): raise NotImplementedError("Wire up your deflection integral here.") ``` -Source: `PyAutoGalaxy:autogalaxy/profiles/mass/abstract.py` — interface. +Source: `PyAutoGalaxy:autogalaxy/profiles/mass/abstract/` — interface. `PyAutoGalaxy:autogalaxy/profiles/mass/dark/nfw.py` — canonical NFW for shape. For the physics of what each method represents (convergence = surface mass density, diff --git a/skills/al_debug_fit_failure.md b/skills/al_debug_fit_failure.md index e71056e..f64b7ed 100644 --- a/skills/al_debug_fit_failure.md +++ b/skills/al_debug_fit_failure.md @@ -86,7 +86,7 @@ Symptoms: parameter posteriors are tight but at unphysical values (einstein radi The inversion has over-fitted background noise into the source plane. - **Regularisation too low.** Bump the regularisation coefficient range — see - `PyAutoLens:autolens/inversion/regularization/`. The `ConstantSplit` regularisation + `PyAutoArray:autoarray/inversion/regularization/`. The `ConstantSplit` regularisation with sensible coefficients usually works. - **No positions penalty.** Add a `PositionsLH` to prevent the inversion from demagnifying the source into infinity. diff --git a/skills/al_prepare_imaging_data.md b/skills/al_prepare_imaging_data.md index 23fec9e..1337875 100644 --- a/skills/al_prepare_imaging_data.md +++ b/skills/al_prepare_imaging_data.md @@ -103,7 +103,7 @@ Source citations: - `PyAutoArray:autoarray/dataset/imaging/dataset.py` — `Imaging.from_fits`, `apply_mask`, `apply_over_sampling`. - `PyAutoArray:autoarray/mask/mask_2d.py` — `Mask2D.circular`. -- `PyAutoArray:autoarray/operators/over_sampling.py` — `over_sample_size_via_radial_bins_from`. +- `PyAutoArray:autoarray/operators/over_sampling/over_sample_util.py` — `over_sample_size_via_radial_bins_from`. Read [`wiki/core/concepts/grids_and_masks.md`](../wiki/core/concepts/grids_and_masks.md) for *why* over-sampling matters (steep light profiles in pixels near the centre alias diff --git a/skills/al_sensitivity_mapping.md b/skills/al_sensitivity_mapping.md index 7418069..55f6392 100644 --- a/skills/al_sensitivity_mapping.md +++ b/skills/al_sensitivity_mapping.md @@ -31,7 +31,7 @@ Workspace path: > TODO: recipe. The pattern: define a `SensitivityMapping` job that takes > the base fit + grid of perturber parameters; per cell, simulate, fit > base + fit perturbed, store Δlog-evidence. Aggregate into a map. See -> `PyAutoFit:autofit/non_linear/grid/sensitivity.py` for the framework +> `PyAutoFit:autofit/non_linear/grid/sensitivity/` for the framework > and `PyAutoLens:autolens/lens/sensitivity.py` (if present) for the > lensing wrapper. diff --git a/skills/al_subhalo_detect.md b/skills/al_subhalo_detect.md index 888732a..d05b5bc 100644 --- a/skills/al_subhalo_detect.md +++ b/skills/al_subhalo_detect.md @@ -33,7 +33,7 @@ The workspace path is > a `Grid2D` of subhalo positions, refitting at each cell with an added > `al.mp.NFW` (or similar). Compare per-cell log evidence to the base > evidence; significant gains flag detections. See -> `PyAutoLens:autolens/lens/subhalo/...` for the subhalo helper module +> `PyAutoLens:autolens/lens/subhalo.py` for the subhalo helper module > if one exists, otherwise compose by hand. ## Branch — followup on a detection candidate diff --git a/skills/al_update_wiki.md b/skills/al_update_wiki.md index 0228a3a..294d273 100644 --- a/skills/al_update_wiki.md +++ b/skills/al_update_wiki.md @@ -31,11 +31,11 @@ title: Non-linear searches sources: - project: PyAutoFit paths: - - autofit/non_linear/search/nest/nautilus.py - - autofit/non_linear/search/nest/dynesty.py - - autofit/non_linear/search/mcmc/emcee.py - - autofit/non_linear/search/mle/bfgs.py - - autofit/non_linear/search/mle/pyswarms.py + - autofit/non_linear/search/nest/nautilus/ + - autofit/non_linear/search/nest/dynesty/ + - autofit/non_linear/search/mcmc/emcee/ + - autofit/non_linear/search/mle/bfgs/ + - autofit/non_linear/search/mle/pyswarms/ pinned_commit: last_updated: 2026-05-22 --- @@ -93,7 +93,7 @@ unaffected prose just because the page was touched — small, reviewable diffs. When rewriting: -- Cite source code as `:` (e.g. `PyAutoFit:autofit/non_linear/search/nest/nautilus.py`). +- Cite source code as `:` (e.g. `PyAutoFit:autofit/non_linear/search/nest/nautilus/`). - Update tables of classes / functions / parameters to match what's actually exported. - Add new entries when the source has new public items in `__all__` or top-level imports. diff --git a/wiki/core/README.md b/wiki/core/README.md index 6f086da..6269f5e 100644 --- a/wiki/core/README.md +++ b/wiki/core/README.md @@ -28,7 +28,7 @@ title: sources: - project: PyAutoFit paths: - - autofit/non_linear/search/nest/nautilus.py + - autofit/non_linear/search/nest/nautilus/ pinned_commit: last_updated: 2026-05-22 --- @@ -48,7 +48,7 @@ Inside a wiki page, code references use the **project name + path relative to th project's repo root**, identical to the skill convention: ``` -See `PyAutoFit:autofit/non_linear/search/nest/nautilus.py` for the implementation. +See `PyAutoFit:autofit/non_linear/search/nest/nautilus/` for the implementation. ``` Resolve project names via [`../../sources.yaml`](../../sources.yaml). Never embed diff --git a/wiki/core/api/analysis_objects.md b/wiki/core/api/analysis_objects.md index d454a62..fdf3131 100644 --- a/wiki/core/api/analysis_objects.md +++ b/wiki/core/api/analysis_objects.md @@ -13,8 +13,8 @@ sources: - autofit/graphical/declarative/collection.py - autofit/graphical/declarative/abstract.py pinned_commit: ce2baa2b6611de99922e04d44b272de1be3ceb8e -last_updated: 2026-06-22 -content_sha256: 8c415d521a9a0e4789eeb9ef69cb9bc9371ae5e7131e4035506554ed443d765b +last_updated: 2026-07-09 +content_sha256: 4d96296dca40cf01909f7426f1639417b9fc0af02124e6389e649bae26baa21b --- # Analysis objects @@ -144,6 +144,29 @@ loglike = analysis.log_likelihood_function(instance=instance) This evaluates the model at a specific parameter vector and returns the log-likelihood. Useful for prior-bound sanity checks. +## Custom `Analysis` subclasses + +When a dataset doesn't fit any built-in analysis, write your own: subclass +`af.Analysis`, store the dataset in `__init__`, and implement +`log_likelihood_function(instance)` — the instance is the lens model at one +parameter vector (galaxies, profiles, all as concrete objects), and the return +value is the log likelihood the non-linear search samples. Everything else +(searches, priors, samples, the aggregator) works unchanged, because the +`Analysis` class is PyAutoFit's only contract between model and data. + +The built-in analyses are the reference implementations — they live in the +`imaging/model/`, `interferometer/model/`, `point/model/` packages of +PyAutoGalaxy/PyAutoLens, and a shortened `AnalysisImaging` is walked through in +the workspace guide. Weak lensing is the instructive precedent: it began life +as exactly this kind of custom analysis before graduating into the built-in +`AnalysisWeak` (see [`../concepts/weak_lensing.md`](../concepts/weak_lensing.md)). + +Workspace guide: `autolens_workspace:scripts/guides/advanced/custom_analysis.py`. +PyAutoFit's analysis cookbook +(https://pyautofit.readthedocs.io/en/latest/cookbooks/analysis.html) is the +concise API reference. This is the ground the +[`al_custom_analysis`](../../../skills/al_custom_analysis.md) skill covers. + ## Visualisations during fit Analyses also produce diagnostic plots that PyAutoFit writes to `output/.../image/` diff --git a/wiki/core/api/configuration.md b/wiki/core/api/configuration.md index 5c7dc0f..d84eec4 100644 --- a/wiki/core/api/configuration.md +++ b/wiki/core/api/configuration.md @@ -16,7 +16,7 @@ sources: - project: PyAutoLens paths: [autolens/config/] pinned_commit: main -last_updated: 2026-05-22 +last_updated: 2026-07-09 --- # Configuration files @@ -28,20 +28,21 @@ loads them (with override layering); the other packages query them via You rarely write code that calls `Conf.instance` directly — but you will edit YAMLs when you want to change default priors, plotting defaults, or output paths. -## The five `general.yaml` / `notation.yaml` / `output.yaml` triads +## What each package's `config/` ships -Each package's `config/` has a `general.yaml` (runtime flags), `notation.yaml` (plot -label conventions and unit strings), and `output.yaml` (where results are written). +The file set differs per package — not every package has every YAML: -- `PyAutoArray:autoarray/config/general.yaml` — grid layout defaults, numba flags. -- `PyAutoArray:autoarray/config/notation.yaml` — `y, x` arcsecond axis labels. -- `PyAutoArray:autoarray/config/output.yaml` — array-output filename conventions. -- `PyAutoFit:autofit/config/general.yaml` — search runtime defaults. -- `PyAutoFit:autofit/config/logging.yaml` — log levels. -- `PyAutoFit:autofit/config/notation.yaml` — parameter symbol/LaTeX strings. -- `PyAutoFit:autofit/config/output.yaml` — `output/` folder structure. -- `PyAutoGalaxy:autogalaxy/config/...` — galaxy + profile defaults. -- `PyAutoLens:autolens/config/...` — lensing-specific overrides. +- `PyAutoArray:autoarray/config/` — `general.yaml` (grid/runtime flags), + `logging.yaml`, `visualize/`. +- `PyAutoFit:autofit/config/` — `general.yaml` (search runtime defaults), + `logging.yaml`, `notation.yaml` (parameter symbol/LaTeX strings), + `output.yaml` (`output/` folder structure), `non_linear/` (per-search + defaults), `priors/`, `visualize/`. +- `PyAutoGalaxy:autogalaxy/config/` — `general.yaml`, `latent.yaml`, + `notation.yaml`, `output.yaml`, `priors/`, `visualize/`. +- `PyAutoLens:autolens/config/` — `general.yaml`, `latent.yaml`, + `non_linear.yaml`, `output.yaml`, `visualize/` (its priors ride on + PyAutoGalaxy's — there is no `autolens/config/priors/`). ## Priors @@ -52,15 +53,19 @@ log-uniform between 0.01 and 8 arcseconds by default". Lives in `PyAutoFit:autofit/config/priors/` ships the meta-templates (`template.yaml`, `model.yaml`, `profile.yaml`). -`PyAutoGalaxy:autogalaxy/config/priors/` ships per-class priors: +`PyAutoGalaxy:autogalaxy/config/priors/` ships per-class priors, organised by +family: -- `Sersic.yaml` — defaults for all Sersic-family classes. -- `Isothermal.yaml` / `PowerLaw.yaml` / `NFW.yaml` — defaults per mass profile. +- `light/` and `mass/` — one YAML per profile class (Sersic family, + Isothermal, PowerLaw, NFW, …). +- `galaxy/`, `ellipse/`, `point_sources.yaml` — composite-object defaults. - `cosmology.yaml` — cosmology constructor defaults. - `mesh/` — pixelisation mesh defaults. -- `basis.yaml` — basis-expansion defaults. +- `basis.yaml` — basis-expansion defaults; `dataset_model.yaml` — the + `DatasetModel` nuisance parameters (sky level, grid offset). -`PyAutoLens:autolens/config/priors/` adds lensing-specific overrides. +PyAutoLens no longer ships its own `config/priors/` — lensing classes take +their defaults from the PyAutoGalaxy priors above. When you compose `af.Model(al.lp.Sersic)`, the constructor parameters get their default priors from these YAMLs. @@ -77,12 +82,9 @@ sersic.intensity = af.UniformPrior(lower_limit=0.01, upper_limit=10.0) ## Visualize `/config/visualize/` controls plot styling — colour maps, axis labels, output -formats. Two sub-flavours per package: - -- `plots.yaml` / `plots_settings.yaml` — per-figure defaults (title, axis labels, - whether to log-scale). -- `mat_wrap_1d/` and `mat_wrap_2d/` — matplotlib wrapper defaults (cmap, figure - size, etc.). +formats — via a `plots.yaml` (per-figure defaults) and, in PyAutoGalaxy, a +`general.yaml`. The old `mat_wrap_1d/` / `mat_wrap_2d/` wrapper folders belonged +to the removed object-oriented plot system and no longer exist. Edit these to change plotting defaults workspace-wide. @@ -104,10 +106,12 @@ defaults — extend or override entries there rather than editing the library YA ## Common tweaks -- **Tighten a default prior** — edit the relevant YAML under `/config/priors/`, - workspace-overlay or in-place. -- **Change the default colormap** — edit `/config/visualize/mat_wrap_2d/Cmap.yaml`. -- **Change output filenames** — edit `/config/output.yaml`. +- **Tighten a default prior** — edit the relevant YAML under + `PyAutoGalaxy:autogalaxy/config/priors/` (or `PyAutoFit:autofit/config/priors/` + for the templates), workspace-overlay or in-place. +- **Change plot styling / colormaps** — edit `/config/visualize/plots.yaml`. +- **Change output filenames** — edit `/config/output.yaml` (PyAutoFit, + PyAutoGalaxy, PyAutoLens). - **Suppress noisy logs** — edit `PyAutoFit:autofit/config/logging.yaml`. ## See also diff --git a/wiki/core/api/csv_api.md b/wiki/core/api/csv_api.md index db1b5ed..dfa142d 100644 --- a/wiki/core/api/csv_api.md +++ b/wiki/core/api/csv_api.md @@ -1,11 +1,12 @@ --- title: CSV API — spreadsheet-driven cluster model composition sources: - - project: PyAutoLens + - project: PyAutoGalaxy paths: - - autolens/csv + - autogalaxy/galaxy/galaxy_model_csv.py + - autogalaxy/galaxy/galaxy_table.py pinned_commit: main -last_updated: 2026-05-22 +last_updated: 2026-07-09 --- # CSV API — cluster-scale model composition @@ -32,7 +33,9 @@ worth it when catalog size dominates the cognitive load. ## Schema — `mass.csv`, `light.csv`, `point.csv` -The exact schema lives in `PyAutoLens:autolens/csv/`, but the conceptual +The exact schema lives in `PyAutoGalaxy:autogalaxy/galaxy/galaxy_model_csv.py` +(with `PyAutoGalaxy:autogalaxy/galaxy/galaxy_table.py` for the +`output_to_csv` / `list_from_csv` table helpers), but the conceptual layout is stable: - one row per galaxy or point-source component diff --git a/wiki/core/api/light_profile_catalog.md b/wiki/core/api/light_profile_catalog.md index 1336b45..8cfd3b6 100644 --- a/wiki/core/api/light_profile_catalog.md +++ b/wiki/core/api/light_profile_catalog.md @@ -7,7 +7,7 @@ sources: - autogalaxy/profiles/light/linear/ - autogalaxy/profiles/light/operated/ - autogalaxy/profiles/light/linear_operated/ - - autogalaxy/profiles/light/basis.py + - autogalaxy/profiles/basis.py pinned_commit: 2547ca175a82f365a64af261923e0ac7232655ac last_updated: 2026-05-22 --- @@ -91,7 +91,7 @@ mge = af.Model(al.lp_basis.Basis, profile_list=gaussians) Multi-Gaussian Expansion is the canonical use. Other basis sets (Shapelet expansions) work the same way. -Source: `PyAutoGalaxy:autogalaxy/profiles/light/basis.py`. +Source: `PyAutoGalaxy:autogalaxy/profiles/basis.py`. ## Operated profiles diff --git a/wiki/core/api/searches.md b/wiki/core/api/searches.md index e5b475a..ee8c71f 100644 --- a/wiki/core/api/searches.md +++ b/wiki/core/api/searches.md @@ -3,16 +3,16 @@ title: Non-linear search catalogue sources: - project: PyAutoFit paths: - - autofit/non_linear/search/nest/nautilus.py - - autofit/non_linear/search/nest/dynesty.py - - autofit/non_linear/search/nest/ultranest.py - - autofit/non_linear/search/mcmc/emcee.py - - autofit/non_linear/search/mcmc/zeus.py - - autofit/non_linear/search/mle/bfgs.py - - autofit/non_linear/search/mle/pyswarms.py - - autofit/non_linear/search/mle/drawer.py + - autofit/non_linear/search/nest/nautilus/ + - autofit/non_linear/search/nest/dynesty/ + - autofit/non_linear/search/nest/ultranest/ + - autofit/non_linear/search/mcmc/emcee/ + - autofit/non_linear/search/mcmc/zeus/ + - autofit/non_linear/search/mle/bfgs/ + - autofit/non_linear/search/mle/pyswarms/ + - autofit/non_linear/search/mle/drawer/ pinned_commit: main -last_updated: 2026-05-22 +last_updated: 2026-07-09 --- # Non-linear search catalogue @@ -37,7 +37,7 @@ af.Nautilus( ) ``` -Source: `PyAutoFit:autofit/non_linear/search/nest/nautilus.py`. Optional dep: +Source: `PyAutoFit:autofit/non_linear/search/nest/nautilus/`. Optional dep: `nautilus-sampler` (pinned to 1.0.5 in the stack). Reference: Lange (2023), arXiv:2306.16923 — see @@ -53,7 +53,7 @@ af.DynestyStatic(path_prefix=..., name=..., nlive=150) af.DynestyDynamic(path_prefix=..., name=..., nlive_init=100) ``` -Source: `PyAutoFit:autofit/non_linear/search/nest/dynesty.py`. Pinned: `dynesty==2.1.4`. +Source: `PyAutoFit:autofit/non_linear/search/nest/dynesty/`. Pinned: `dynesty==2.1.4`. Reference: Speagle (2020), arXiv:1904.02180 — see [`wiki/literature/sources/bayesian-inference-methods.md`](../../literature/sources/bayesian-inference-methods.md#speagle-2020--dynesty). @@ -68,7 +68,7 @@ UltraNest is not currently exposed as a public `autofit` search class in the af.Nautilus(path_prefix=..., name=..., n_live=200) ``` -Source: `PyAutoFit:autofit/non_linear/search/nest/ultranest.py`. Optional dep. +Source: `PyAutoFit:autofit/non_linear/search/nest/ultranest/`. Optional dep. Reference: Buchner — algorithmic foundation in [`wiki/literature/concepts/nested-sampling.md`](../../literature/concepts/nested-sampling.md) @@ -86,7 +86,7 @@ around a known mode, not for finding the mode in the first place. af.Emcee(path_prefix=..., name=..., nwalkers=50, nsteps=10000) ``` -Source: `PyAutoFit:autofit/non_linear/search/mcmc/emcee.py`. Pinned: `emcee>=3.1.6`. +Source: `PyAutoFit:autofit/non_linear/search/mcmc/emcee/`. Pinned: `emcee>=3.1.6`. Reference: Foreman-Mackey et al. (2013), arXiv:1202.3665 — see [`wiki/literature/sources/bayesian-inference-methods.md`](../../literature/sources/bayesian-inference-methods.md#foreman-mackey-2013--emcee). @@ -100,7 +100,7 @@ runtime per step. af.Zeus(path_prefix=..., name=..., nwalkers=50, nsteps=10000) ``` -Source: `PyAutoFit:autofit/non_linear/search/mcmc/zeus.py`. Optional dep. +Source: `PyAutoFit:autofit/non_linear/search/mcmc/zeus/`. Optional dep. Reference: Karamanis, Beutler & Peacock (2021), arXiv:2105.03468 — see [`wiki/literature/sources/bayesian-inference-methods.md`](../../literature/sources/bayesian-inference-methods.md#karamanis-2021--zeus). @@ -116,7 +116,7 @@ starting point for an MCMC chain or as a quick mode-finding pass. af.BFGS(path_prefix=..., name=...) ``` -Source: `PyAutoFit:autofit/non_linear/search/mle/bfgs.py`. +Source: `PyAutoFit:autofit/non_linear/search/mle/bfgs/`. ### Particle Swarm / MLE Searches @@ -138,7 +138,7 @@ prior gives reasonable models. af.Drawer(path_prefix=..., name=..., total_iterations=100) ``` -Source: `PyAutoFit:autofit/non_linear/search/mle/drawer.py`. +Source: `PyAutoFit:autofit/non_linear/search/mle/drawer/`. ## Picking a search at a glance diff --git a/wiki/core/concepts/cosmology_and_units.md b/wiki/core/concepts/cosmology_and_units.md index 6b9a6f7..6e03fe5 100644 --- a/wiki/core/concepts/cosmology_and_units.md +++ b/wiki/core/concepts/cosmology_and_units.md @@ -5,7 +5,7 @@ sources: paths: - autogalaxy/cosmology/ pinned_commit: main -last_updated: 2026-05-22 +last_updated: 2026-07-09 --- # Cosmology and units @@ -19,7 +19,7 @@ Source: `PyAutoGalaxy:autogalaxy/cosmology/`. ## Default cosmology -Each `Tracer` (or `Plane`, or analysis object) carries a cosmology. The default is +Each `Tracer` (or analysis object) carries a cosmology. The default is Planck 2018 via astropy: ```python diff --git a/wiki/core/concepts/extra_galaxies_and_noise_scaling.md b/wiki/core/concepts/extra_galaxies_and_noise_scaling.md index 8513098..f8c0040 100644 --- a/wiki/core/concepts/extra_galaxies_and_noise_scaling.md +++ b/wiki/core/concepts/extra_galaxies_and_noise_scaling.md @@ -4,9 +4,10 @@ sources: - project: PyAutoArray paths: - autoarray/dataset/imaging/dataset.py + - autoarray/dataset/dataset_model.py - autoarray/mask/mask_2d.py pinned_commit: main -last_updated: 2026-05-29 +last_updated: 2026-07-09 --- # Extra galaxies and noise scaling @@ -62,6 +63,30 @@ manual script (`.../data_preparation/examples/optional/mask_extra_galaxies.py`). Source: `PyAutoArray:autoarray/dataset/imaging/dataset.py` (`apply_noise_scaling`). +## Sky background + +The other light in every image that is not the strong lens is the **sky +background** — sky glow, zodiacal light and unresolved field sources. Data +reduction normally subtracts it, but the subtraction is never perfect, and its +residual is degenerate with the faint outskirts of the lens galaxy's light: an +over-subtracted sky can masquerade as a smaller effective radius or steeper +Sersic index, and the formal errors on those parameters will be underestimated +if the sky is assumed perfect. + +When low-surface-brightness structure matters, fit the sky as a model +component: `al.DatasetModel(background_sky_level=...)` adds the sky level as a +free parameter of the non-linear search (one extra dimension), so the +posterior on every light-profile parameter marginalises over the sky +uncertainty. The same `DatasetModel` object also carries `grid_offset` and +`grid_rotation_angle`, the nuisance parameters used for astrometric offsets in +multi-dataset fitting (see +[`multi_wavelength`](./multi_wavelength.md)). + +Source: `PyAutoArray:autoarray/dataset/dataset_model.py`. Workspace example: +`autolens_workspace:scripts/imaging/features/advanced/sky_background/modeling.py` +(fits a dataset simulated *without* sky subtraction, so the image outskirts sit +at the sky level rather than zero). + ## Galaxy-scale vs group-scale The strategy split mirrors the lens scale. At **galaxy scale** (one main lens), extra galaxies diff --git a/wiki/core/concepts/galaxy_and_plane.md b/wiki/core/concepts/galaxy_and_plane.md index dc7a5c2..e403026 100644 --- a/wiki/core/concepts/galaxy_and_plane.md +++ b/wiki/core/concepts/galaxy_and_plane.md @@ -1,22 +1,24 @@ --- -title: Galaxy and Plane +title: Galaxy and Galaxies (redshift planes) sources: - project: PyAutoGalaxy paths: - autogalaxy/galaxy/galaxy.py - - autogalaxy/plane/plane.py + - autogalaxy/galaxy/galaxies.py pinned_commit: main -last_updated: 2026-05-22 +last_updated: 2026-07-09 --- -# Galaxy and Plane +# Galaxy and Galaxies (redshift planes) The two structural objects between profiles (`al.lp.*`, `al.mp.*`) and a `Tracer`. -A `Galaxy` bundles light + mass profiles at one redshift; a `Plane` (used internally -by `Tracer`) groups galaxies at the same redshift. +A `Galaxy` bundles light + mass profiles at one redshift; a `Galaxies` collection +(used internally by `Tracer`) groups galaxies at the same redshift — what lensing +theory calls a *plane*. (Older PyAutoLens versions had a dedicated `Plane` class; +it no longer exists — the redshift slice is now just a `Galaxies` collection.) Source: `PyAutoGalaxy:autogalaxy/galaxy/galaxy.py` and -`PyAutoGalaxy:autogalaxy/plane/plane.py`. +`PyAutoGalaxy:autogalaxy/galaxy/galaxies.py`. ## Galaxy @@ -55,22 +57,24 @@ defl = galaxy.deflections_yx_2d_from(grid) # sum of mass profile deflections These delegate to the profiles. For lensing-specific operations (ray tracing, critical curves, magnification), wrap the galaxies in a `Tracer`. -## Plane +## Galaxies — the redshift-plane grouping -A `Plane` groups galaxies at the *same* redshift. Most users never construct one -manually — `Tracer` builds them internally from the input galaxy list: +`al.Galaxies` is a list-like collection of galaxies that computes summed +quantities (`image_2d_from`, `convergence_2d_from`, `deflections_yx_2d_from`) +over its members. `Tracer` groups its input galaxy list by redshift into one +`Galaxies` per plane: ```python tracer = al.Tracer(galaxies=[lens, source]) -tracer.planes # [Plane(z=0.5, [lens]), Plane(z=1.0, [source])] +tracer.planes # [Galaxies([lens]), Galaxies([source])] — ascending redshift ``` -You'd build a `Plane` directly only when you have multiple galaxies at exactly the -same redshift and want them as a single unit (e.g. a cluster lens with member -galaxies). The standard workflow is to pass the full list to `Tracer` and let it -group by redshift. +You'd build a `Galaxies` directly only when you want several same-redshift +galaxies treated as a single unit outside a tracer (e.g. plotting a cluster +plane's total convergence). The standard workflow is to pass the full list to +`Tracer` and let it group by redshift. -Source: `PyAutoGalaxy:autogalaxy/plane/plane.py`. +Source: `PyAutoGalaxy:autogalaxy/galaxy/galaxies.py`. ## Galaxies in `af.Model` diff --git a/wiki/core/concepts/grids_and_masks.md b/wiki/core/concepts/grids_and_masks.md index ae3e149..76a5da3 100644 --- a/wiki/core/concepts/grids_and_masks.md +++ b/wiki/core/concepts/grids_and_masks.md @@ -5,9 +5,9 @@ sources: paths: - autoarray/structures/grids/uniform_2d.py - autoarray/mask/mask_2d.py - - autoarray/operators/over_sampling.py + - autoarray/operators/over_sampling/over_sample_util.py pinned_commit: main -last_updated: 2026-05-22 +last_updated: 2026-07-09 --- # Grids and masks @@ -93,7 +93,7 @@ dataset = dataset.apply_over_sampling(over_sample_size_lp=over) `over_sample_size_lp` controls light-profile over-sampling. Mass-profile evaluation uses a separate `over_sample_size_pixelization` for inversions. -Source: `PyAutoArray:autoarray/operators/over_sampling.py`. +Source: `PyAutoArray:autoarray/operators/over_sampling/over_sample_util.py`. When to over-sample more aggressively: diff --git a/wiki/core/concepts/group_and_cluster_lensing.md b/wiki/core/concepts/group_and_cluster_lensing.md index f2d5a06..497ead3 100644 --- a/wiki/core/concepts/group_and_cluster_lensing.md +++ b/wiki/core/concepts/group_and_cluster_lensing.md @@ -4,12 +4,9 @@ sources: - project: PyAutoGalaxy paths: - autogalaxy/galaxy/galaxy.py + - autogalaxy/galaxy/galaxy_model_csv.py pinned_commit: main - - project: PyAutoLens - paths: - - autolens/csv - pinned_commit: main -last_updated: 2026-05-22 +last_updated: 2026-07-09 --- # Group- and cluster-scale lensing diff --git a/wiki/core/concepts/inversions_and_pixelizations.md b/wiki/core/concepts/inversions_and_pixelizations.md index 7b74815..54f02e8 100644 --- a/wiki/core/concepts/inversions_and_pixelizations.md +++ b/wiki/core/concepts/inversions_and_pixelizations.md @@ -12,7 +12,7 @@ sources: - autolens/lens/to_inversion.py - autolens/analysis/positions.py pinned_commit: a91febcb1aa12797f9d5ece54c1cbbac528cd087 -last_updated: 2026-05-22 +last_updated: 2026-07-09 --- # Inversions and pixelisations @@ -46,7 +46,7 @@ pixelization = al.Pixelization( source = al.Galaxy(redshift=1.0, pixelization=pixelization) ``` -Source: `PyAutoArray:autoarray/inversion/pixelization/` and +Source: `PyAutoArray:autoarray/inversion/pixelization.py` and `PyAutoLens:autolens/lens/to_inversion.py`. ## Mesh choices diff --git a/wiki/core/concepts/lensing_basics.md b/wiki/core/concepts/lensing_basics.md index 3714376..aa368fe 100644 --- a/wiki/core/concepts/lensing_basics.md +++ b/wiki/core/concepts/lensing_basics.md @@ -7,9 +7,9 @@ sources: pinned_commit: main - project: PyAutoGalaxy paths: - - autogalaxy/profiles/mass/abstract.py + - autogalaxy/profiles/mass/abstract/ pinned_commit: main -last_updated: 2026-05-22 +last_updated: 2026-07-09 --- # Lensing basics @@ -48,7 +48,7 @@ kappa = tracer.convergence_2d_from(grid=grid) A galaxy-cluster region typically has `κ > 1`; a small foreground halo `κ ≪ 1`. Source: `PyAutoLens:autolens/lens/tracer.py` (`convergence_2d_from`) and -`PyAutoGalaxy:autogalaxy/profiles/mass/abstract.py` (`convergence_2d_from`). +`PyAutoGalaxy:autogalaxy/profiles/mass/abstract/` (`convergence_2d_from`). ## Magnification diff --git a/wiki/core/concepts/point_source.md b/wiki/core/concepts/point_source.md index edd0094..87f1eb7 100644 --- a/wiki/core/concepts/point_source.md +++ b/wiki/core/concepts/point_source.md @@ -8,9 +8,9 @@ sources: pinned_commit: main - project: PyAutoGalaxy paths: - - autogalaxy/profiles/point/abstract.py + - autogalaxy/profiles/point_sources.py pinned_commit: main -last_updated: 2026-05-22 +last_updated: 2026-07-09 --- # Point-source lensing @@ -53,6 +53,28 @@ profile. Sources: `PyAutoLens:autolens/point/model/analysis.py` and `PyAutoLens:autolens/point/fit/dataset.py`. +## Where the positions come from — deblending + +The `PointDataset` positions must first be *measured* from imaging. +Reading off the brightest 2 or 4 pixels (ds9, a GUI) is often good +enough, but it gives no sub-pixel precision, ignores the PSF, and blends +the point-source fluxes with the lens galaxy's light. + +The **deblending** workflow fixes this by fitting the `Imaging` data +directly, modeling each multiple image as an independent PSF-convolved +light profile **in the image plane** — deliberately *without* a +source-plane point source or ray tracing. The reason is microlensing: +stars in the lens galaxy boost or suppress each image's magnification in +a way no smooth mass model can capture, so tying the image fluxes +together through a source-plane model would bias them. Leaving each +image's `intensity` free absorbs the microlensing, which is also why +fluxes are usually down-weighted or omitted in point-source fits. + +The fit simultaneously models the lens galaxy's light, so its output is +sub-pixel image positions, PSF-deblended fluxes *and* lens light +properties — ready to feed into a `PointDataset`. Workspace example: +`autolens_workspace:scripts/point_source/features/deblending/modeling.py`. + ## Solving the lens equation for image positions Unlike extended-source fitting, point-source fitting must explicitly diff --git a/wiki/core/concepts/sensitivity_mapping.md b/wiki/core/concepts/sensitivity_mapping.md index 7b1c551..2d0c63e 100644 --- a/wiki/core/concepts/sensitivity_mapping.md +++ b/wiki/core/concepts/sensitivity_mapping.md @@ -3,13 +3,13 @@ title: Sensitivity mapping for substructure detection sources: - project: PyAutoFit paths: - - autofit/non_linear/grid/sensitivity.py + - autofit/non_linear/grid/sensitivity/ pinned_commit: main - project: PyAutoLens paths: - autolens/lens pinned_commit: main -last_updated: 2026-05-22 +last_updated: 2026-07-09 --- # Sensitivity mapping @@ -24,7 +24,7 @@ recording the evidence gain. ## The framework PyAutoFit provides the generic engine in -`PyAutoFit:autofit/non_linear/grid/sensitivity.py`. The key abstraction is +`PyAutoFit:autofit/non_linear/grid/sensitivity/`. The key abstraction is that you define: - a `base_model` that represents the smooth lens fit diff --git a/wiki/core/concepts/slam_pipeline.md b/wiki/core/concepts/slam_pipeline.md index dba38a6..6c16ad5 100644 --- a/wiki/core/concepts/slam_pipeline.md +++ b/wiki/core/concepts/slam_pipeline.md @@ -4,9 +4,8 @@ sources: - project: autolens_workspace paths: - scripts/guides/modeling/slam_start_here.py - - slam/ pinned_commit: main -last_updated: 2026-05-22 +last_updated: 2026-07-09 --- # SLaM pipeline — Source, Light, Mass @@ -95,12 +94,16 @@ production tooling, not an introductory workflow. ## How to actually run it -The SLaM driver functions live in the workspace at `autolens_workspace:slam/`, not in -PyAutoLens itself. From a workspace clone: +The SLaM driver functions live in the workspace, not in PyAutoLens itself — defined +**inline** in `autolens_workspace:scripts/guides/modeling/slam_start_here.py` (the +canonical reference for pipeline structure and function signatures; there is no +separate `slam/` package). Per-topic SLaM examples live under each topic's +`features/slam/` folder. From a workspace clone: -```python -from autolens_workspace.slam import slam_main -slam_main(dataset=dataset, redshift_lens=0.5, redshift_source=1.0, ...) +```bash +# The pipeline stages (source_lp -> source_pix -> light -> mass) are plain +# functions defined inline in the script; copy it and adapt the stages. +python scripts/guides/modeling/slam_start_here.py ``` See [`../../../skills/al_run_slam_pipeline.md`](../../../skills/al_run_slam_pipeline.md). diff --git a/wiki/core/concepts/substructure_and_subhalos.md b/wiki/core/concepts/substructure_and_subhalos.md index 3168f7d..9ec6fff 100644 --- a/wiki/core/concepts/substructure_and_subhalos.md +++ b/wiki/core/concepts/substructure_and_subhalos.md @@ -4,12 +4,13 @@ sources: - project: PyAutoLens paths: - autolens/lens + - autolens/lens/los.py pinned_commit: main - project: PyAutoGalaxy paths: - autogalaxy/profiles/mass/dark pinned_commit: main -last_updated: 2026-05-22 +last_updated: 2026-07-09 --- # Substructure and subhaloes @@ -99,6 +100,29 @@ tracing machinery. Inference-wise, the practical lesson is that the population model should not collapse everything into "subhaloes" unless line-of-sight contributions have been modeled or marginalized over. +### Simulating LOS halo populations + +PyAutoLens ships dedicated machinery for generating realistic LOS +populations, in `PyAutoLens:autolens/lens/los.py`: + +- **`LOSSampler`** draws haloes from a cosmological halo mass function + within a light-cone geometry around the lens, converts them to + `al.mp.NFWTruncatedSph` profiles, and distributes them over multiple + redshift planes between the observer and the source. +- **`los_planes_from`** assembles the sampled haloes into the per-plane + galaxy lists a multi-plane `Tracer` consumes. +- Each plane also receives an `al.mp.MassSheet` with **negative + convergence**, compensating the mean convergence of the halo + population so total mass is conserved (following He et al. 2022, + MNRAS 511, 3046). Without these sheets the LOS population + systematically over-lenses: only the *fluctuations* about the mean + density should perturb the images, not the mean itself. + +The workspace example is +`autolens_workspace:scripts/imaging/features/advanced/los_halos/simulator.py` +(with a JAX variant alongside), which also writes the halo sample list +and per-plane sheet values out as diagnostics. + ## Subhalo mass parameterisations The dark-profile catalog in diff --git a/wiki/core/concepts/tracer.md b/wiki/core/concepts/tracer.md index 9c8eff0..52b9ad8 100644 --- a/wiki/core/concepts/tracer.md +++ b/wiki/core/concepts/tracer.md @@ -4,16 +4,16 @@ sources: - project: PyAutoLens paths: - autolens/lens/tracer.py - - autolens/lens/plane.py + - autolens/lens/tracer_util.py pinned_commit: main -last_updated: 2026-05-22 +last_updated: 2026-07-09 --- # Tracer — multi-plane ray tracing `al.Tracer` is the central lensing object. It composes one or more `Galaxy` objects -(or `Plane` objects, each grouping galaxies at the same redshift) and knows how to -ray-trace a grid of image-plane coordinates back to each subsequent plane. +(grouped internally into one `Galaxies` collection per redshift plane) and knows how +to ray-trace a grid of image-plane coordinates back to each subsequent plane. Source: `PyAutoLens:autolens/lens/tracer.py`. @@ -28,9 +28,9 @@ source = al.Galaxy(redshift=1.0, bulge=al.lp.SersicCore(...)) tracer = al.Tracer(galaxies=[lens, source]) ``` -`Tracer` reads the redshifts off the galaxies and constructs `Plane` objects -internally — one per unique redshift, in ascending order. With two redshifts you get -two planes; with three or more you get a multi-plane system. +`Tracer` reads the redshifts off the galaxies and groups them into `Galaxies` +collections internally — one per unique redshift, in ascending order. With two +redshifts you get two planes; with three or more you get a multi-plane system. ## What a tracer computes @@ -56,7 +56,7 @@ For physical-unit equivalents, see [`cosmology_and_units`](./cosmology_and_units ```python tracer.galaxies # list of Galaxy objects, by redshift order -tracer.planes # list of Plane objects, each a redshift slice +tracer.planes # list of Galaxies collections, one per redshift slice tracer.galaxies[0].image_2d_from(grid) # image-plane image of just the first galaxy tracer.planes[-1].image_2d_from(grid) # image of the source plane ``` @@ -84,7 +84,20 @@ The relative angular-diameter distances `D_ij` between every pair of planes are computed from a cosmology (default Planck 2018; set explicitly on the tracer if you need otherwise). -Source: `PyAutoLens:autolens/lens/plane.py`. +The showcase multi-plane application is the **double Einstein ring** — two +sources at different redshifts behind one lens, where the inner source is +itself a deflector of the outer one. Because the ring radii depend on distance +ratios between *three* planes, these systems constrain cosmological parameters +in a way single-ring lenses cannot. Fitting them is a hard initialisation +problem (the tutorial `modeling.py` only converges by starting at the truth): +the practical recipe is the chained two-search script, and SLaM for production. +Workspace: +`autolens_workspace:scripts/imaging/features/advanced/double_einstein_ring/` +(`modeling.py` → `chaining.py` → `slam.py`). LOS-halo populations (see +[`substructure_and_subhalos`](./substructure_and_subhalos.md)) are the other +routine multi-plane use. + +Source: `PyAutoLens:autolens/lens/tracer_util.py`. ## Pixelised tracer output @@ -101,7 +114,7 @@ max-log-likelihood lens model. ## See also - [`lensing_basics`](./lensing_basics.md) — the physics of every quantity above. -- [`galaxy_and_plane`](./galaxy_and_plane.md) — what `Galaxy` and `Plane` are. +- [`galaxy_and_plane`](./galaxy_and_plane.md) — what `Galaxy` and `Galaxies` are. - [`../api/analysis_objects`](../api/analysis_objects.md) — how a tracer plugs into `AnalysisImaging` for fitting. - [`../../../skills/al_plot_tracer.md`](../../../skills/al_plot_tracer.md). diff --git a/wiki/core/concepts/weak_lensing.md b/wiki/core/concepts/weak_lensing.md index c5bd360..ffdde72 100644 --- a/wiki/core/concepts/weak_lensing.md +++ b/wiki/core/concepts/weak_lensing.md @@ -5,7 +5,7 @@ sources: paths: - autolens/weak pinned_commit: main -last_updated: 2026-05-22 +last_updated: 2026-07-09 --- # Weak lensing @@ -60,23 +60,63 @@ field. It is not intended to replace a dedicated cosmic-shear pipeline for parameters like `S8`, survey masks, tomography, intrinsic alignments, and survey-wide covariance modeling. +## Shear vs. reduced shear (`is_reduced`) + +What a survey actually measures from galaxy shapes is not the shear +`gamma` but the **reduced shear** `g = gamma / (1 - kappa)` — the +convergence `kappa` rescales the observed ellipticity. Far from the lens +the two converge, but near a cluster core the difference is real. +`WeakDataset` carries an `is_reduced` flag and `FitWeak` computes the +matching model quantity, so the comparison is always like-for-like. The +catalogue loaders (`from_csv`, `from_fits`, `from_arrays`) default to +`is_reduced=True` because real catalogues are ellipticity-based; only the +bare constructor defaults to `False`. + ## Joint strong + weak Joint strong-plus-weak fitting is where this module becomes most useful. The strong-lens data constrain the inner critical region, while weak shear -extends the lever arm to much larger radii. In PyAutoFit terms this is -just another combined-analysis problem: one analysis for the arc data, -one for the weak catalog, one shared mass model. +extends the lever arm to much larger radii. Because the two datasets are +independent measurements of the same mass distribution, the joint log +likelihood is simply the sum: one `Tracer` shared between a `FitImaging` +and a `FitWeak`, or between an `AnalysisImaging` and an `AnalysisWeak` +in a combined-analysis fit. The workspace has a dedicated +strong-plus-weak feature series at +`autolens_workspace:scripts/weak/features/strong_lensing/` +(`simulator.py` → `fit.py` → `modeling.py`). ## API touchpoints -The intended API surface is the `autolens/weak/` package: - -- `PyAutoLens:autolens/weak/dataset.py` for the catalog container -- `PyAutoLens:autolens/weak/model/analysis.py` for the likelihood wrapper - -The companion workspace example is `weak/fit.py`, which is the practical -reference for how the catalog is loaded and handed to the search. +The API surface is the `autolens/weak/` package: + +- **`al.WeakDataset`** (`PyAutoLens:autolens/weak/dataset.py`) — the + catalogue container: irregular source positions with per-source + `(gamma_1, gamma_2)` and noise, optional redshifts, the `is_reduced` + flag, and `from_csv` / `from_fits` / `from_arrays` loaders. +- **`al.FitWeak`** (`PyAutoLens:autolens/weak/fit.py`) — fits a `Tracer` + to the catalogue: `model_shear`, `residual_map`, + `normalized_residual_map`, `chi_squared`, `log_likelihood`, + `figure_of_merit`. Deliberately NumPy-first — weak fits are cheap + enough that no GPU/JAX path is needed. +- **`al.AnalysisWeak`** (`PyAutoLens:autolens/weak/model/`) — wraps + `FitWeak` in the PyAutoFit likelihood interface so any non-linear + search can sample the mass model. + +The workspace series mirrors the imaging one: +`autolens_workspace:scripts/weak/` has `simulator.py`, `fit.py` +(hand-picked model + residuals), `modeling.py` (full Nautilus fit — N=5 +in minutes on a CPU), and `likelihood_function.py`. + +## Real catalogues — the Abell 2744 capstone + +`autolens_workspace:scripts/weak/real_data/a2744.py` fits a dark-matter +halo model to a real JWST-era shape catalogue of Abell 2744 (from the +public pyRRG code; Harvey & Massey 2024, MNRAS 529, 802). It is the +practical template for any new catalogue: load a FITS shape table, +convert RA/Dec to tangent-plane arcseconds, apply the standard +weak-lensing quality cuts, build a `WeakDataset` with `is_reduced=True`, +and sanity-check the result against a model-independent Kaiser-Squires +mass map before trusting any parametric fit. ## Related pages diff --git a/wiki/core/index.md b/wiki/core/index.md index 21aa1f9..cbd343c 100644 --- a/wiki/core/index.md +++ b/wiki/core/index.md @@ -1,7 +1,7 @@ --- title: Core wiki — PyAuto* reference sources: [] -last_updated: 2026-05-22 +last_updated: 2026-07-09 --- # Core wiki — PyAuto\* reference @@ -30,8 +30,8 @@ Physics + framework material the skills lean on. linear vs. operated. - [Mass profiles](./concepts/mass_profiles.md) — Isothermal, PowerLaw, NFW, external shear, multipole. -- [Galaxy and plane](./concepts/galaxy_and_plane.md) — how galaxies, redshifts, and - the tracer compose. +- [Galaxy and Galaxies](./concepts/galaxy_and_plane.md) — how galaxies, redshifts, + and the tracer's redshift planes compose. - [Grids and masks](./concepts/grids_and_masks.md) — slim/native, over-sampling, sub-grids. - [Extra galaxies and noise scaling](./concepts/extra_galaxies_and_noise_scaling.md) — @@ -46,7 +46,7 @@ Physics + framework material the skills lean on. - [Cosmology and units](./concepts/cosmology_and_units.md) — angular ↔ physical conversions. - [Point-source lensing](./concepts/point_source.md) — positions, flux - ratios, image-position solvers. + ratios, deblending, image-position solvers. - [Time-delay cosmography](./concepts/time_delay_cosmography.md) — H0 from time-delay lenses, Fermat potential, mass-sheet degeneracy. - [Substructure and subhaloes](./concepts/substructure_and_subhalos.md) @@ -57,8 +57,9 @@ Physics + framework material the skills lean on. — extra galaxies, scaling relations, cluster-scale composition. - [Multi-wavelength / multi-dataset](./concepts/multi_wavelength.md) — joint fits across bands and instruments. -- [Weak lensing](./concepts/weak_lensing.md) — `WeakDataset` shear - catalogue fits. +- [Weak lensing](./concepts/weak_lensing.md) — `WeakDataset` / `FitWeak` + shear-catalogue fits, reduced shear, real-catalogue workflow, joint + strong+weak. - [Hierarchical / graphical models](./concepts/hierarchical_models.md) — population-level inference, expectation propagation. - [Interferometer theory](./concepts/interferometer_theory.md) — @@ -77,7 +78,8 @@ Task-oriented catalogues — comprehensive lists of what's available, with one-l - [Plotting](./api/plotting.md) — `aplt` entry points, subplot helpers, direct plotting functions. - [Configuration](./api/configuration.md) — `/config/*.yaml` semantics. - [Analysis objects](./api/analysis_objects.md) — `AnalysisImaging`, - `AnalysisInterferometer`, `AnalysisPoint`. + `AnalysisInterferometer`, `AnalysisPoint`, factor-graph multi-dataset + fits, custom `Analysis` subclasses. - [Aggregator and result database](./api/aggregator.md) — bulk loading of completed fits, derived quantities, optional SQLite backend. - [CSV API](./api/csv_api.md) — cluster-scale spreadsheet model @@ -89,6 +91,17 @@ Task-oriented catalogues — comprehensive lists of what's available, with one-l - [Installation](./operations/installation.md) — pip vs. editable clone, version pins. +- [Dataset layout](./operations/dataset.md) — on-disk dataset conventions and + `info.json`. - [Sandbox / restricted environments](./operations/sandbox.md) — `NUMBA_CACHE_DIR`, `MPLCONFIGDIR`, `PYAUTO_TEST_MODE` and friends. -- [HPC](./operations/hpc.md) — running fits on a cluster. +- [HPC](./operations/hpc.md) — running fits on a cluster (concepts: cores, + JAX/GPU, SLURM). +- [HPC infrastructure](./operations/hpc_infrastructure.md) — the `hpc/` template, + batch scripts and the `sync` CLI shipped with this workspace. + +## External resources + +- [External routing index](./external/index.md) — HowToLens, RTD and + `autolens_workspace` catalogues, audience routing, and the per-skill + citation map. diff --git a/wiki/core/operations/installation.md b/wiki/core/operations/installation.md index 91aea04..097e8e9 100644 --- a/wiki/core/operations/installation.md +++ b/wiki/core/operations/installation.md @@ -68,7 +68,7 @@ done Order matters — install bottom-up so each pip install can resolve its previous dependency. -Resolve git URLs via [`../../sources.yaml`](../../sources.yaml) rather than the inline +Resolve git URLs via [`../../../sources.yaml`](../../../sources.yaml) rather than the inline hard-coded URLs above; the YAML is the source of truth and accommodates URL changes. ## Version pins worth knowing diff --git a/wiki/core/stack/autoarray.md b/wiki/core/stack/autoarray.md index f093fb4..1d3acad 100644 --- a/wiki/core/stack/autoarray.md +++ b/wiki/core/stack/autoarray.md @@ -9,7 +9,7 @@ sources: - autoarray/operators/ - README.rst pinned_commit: main -last_updated: 2026-05-22 +last_updated: 2026-07-09 --- # PyAutoArray — arrays, grids, masks, datasets @@ -66,7 +66,7 @@ grid = grid.apply_over_sampling(over_sample_size=over) ``` Sub-grids are denser where it matters and coarser everywhere else. Source: -`PyAutoArray:autoarray/operators/over_sampling.py`. +`PyAutoArray:autoarray/operators/over_sampling/over_sample_util.py`. ## Inversions diff --git a/wiki/core/stack/autogalaxy.md b/wiki/core/stack/autogalaxy.md index 699a17d..8a17f61 100644 --- a/wiki/core/stack/autogalaxy.md +++ b/wiki/core/stack/autogalaxy.md @@ -6,11 +6,11 @@ sources: - autogalaxy/profiles/light/ - autogalaxy/profiles/mass/ - autogalaxy/galaxy/ - - autogalaxy/plane/ + - autogalaxy/galaxy/ - autogalaxy/cosmology/ - README.rst pinned_commit: main -last_updated: 2026-05-22 +last_updated: 2026-07-09 --- # PyAutoGalaxy — light and mass profiles @@ -28,8 +28,8 @@ PyAutoGalaxy is the *galaxy modelling* layer. It defines: external shear, point mass, mass sheets, …) — see [`api/mass_profile_catalog`](../api/mass_profile_catalog.md). - **`Galaxy`** — composes light + mass profiles with a redshift. -- **`Plane`** — one redshift slice of a lens system; multiple planes make a tracer - (in PyAutoLens). +- **`Galaxies`** — one redshift slice of a lens system (lensing theory's "plane"); + multiple make a tracer (in PyAutoLens). - **Cosmology** helpers — distance + redshift conversions. It's used both standalone (for non-lens galaxy fitting) and as PyAutoLens's profile diff --git a/wiki/core/stack/autolens.md b/wiki/core/stack/autolens.md index 1225857..1f6d74d 100644 --- a/wiki/core/stack/autolens.md +++ b/wiki/core/stack/autolens.md @@ -10,7 +10,7 @@ sources: - autolens/quantity/ - README.rst pinned_commit: main -last_updated: 2026-05-22 +last_updated: 2026-07-09 --- # PyAutoLens — strong lensing umbrella @@ -31,7 +31,7 @@ packages. ## Tracer — multi-plane ray tracing The headline lensing object. A `Tracer` takes an ordered list of `Galaxy` objects -(or `Plane` objects for multi-plane systems) and knows how to ray-trace image-plane +(grouped into per-redshift `Galaxies` planes for multi-plane systems) and knows how to ray-trace image-plane grids through them. ```python From 18df02529302738314769b9a1f4e399cfb405df5 Mon Sep 17 00:00:00 2001 From: Jammy2211 Date: Thu, 9 Jul 2026 17:23:29 +0100 Subject: [PATCH 2/2] =?UTF-8?q?feat:=20--check-citations=20audit=20mode=20?= =?UTF-8?q?=E2=80=94=20resolve=20Project:path=20cites=20against=20checkout?= =?UTF-8?q?s?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The release audit found ~30 stale `Project:path` citations on a tree the symbol audit scored 0-broken: symbols and citation paths are independent failure axes. New mode scans skills/, wiki/core/, AGENTS.md and llms.txt (inline cites, projects from sources.yaml, plus wiki frontmatter sources[].paths[]) and resolves each path against a checkout (installed -> sources/ -> sibling); `...` abbreviations only need their concrete prefix. Missing path = ERROR (exit 1); unresolvable project tree = warning. First run caught 5 more live ones (README.rst -> README.md in the stack pages) — fixed here. Documented as al_audit_skill_apis §6; added to al_refresh_api_docs "complete" criteria. Issue: https://github.com/PyAutoLabs/autolens_assistant/issues/40 Co-Authored-By: Claude Fable 5 --- autoassistant/audit_skill_apis.py | 120 ++++++++++++++++++++++++++++++ skills/al_audit_skill_apis.md | 21 ++++++ skills/al_refresh_api_docs.md | 5 +- wiki/core/stack/autoarray.md | 2 +- wiki/core/stack/autoconf.md | 4 +- wiki/core/stack/autofit.md | 4 +- wiki/core/stack/autogalaxy.md | 2 +- wiki/core/stack/autolens.md | 2 +- 8 files changed, 152 insertions(+), 8 deletions(-) diff --git a/autoassistant/audit_skill_apis.py b/autoassistant/audit_skill_apis.py index f497b32..ed3e547 100644 --- a/autoassistant/audit_skill_apis.py +++ b/autoassistant/audit_skill_apis.py @@ -1124,6 +1124,117 @@ def write_provenance(root: Path, only: Optional[list[Path]] = None) -> int: return 0 +# --------------------------------------------------------------------------- +# Citation-path check (--check-citations) +# --------------------------------------------------------------------------- +# The symbol audit and the citation paths are independent failure axes: a page can +# cite only live `al.*` symbols while its `Project:relative/path` source citations +# point at a pre-refactor file layout (module → package moves, deleted files). This +# check resolves every citation — inline backticked `:` in the body +# and each `sources[].paths[]` entry in wiki frontmatter — against a real checkout. + + +def _sources_projects(root: Path) -> list[str]: + """Project names from sources.yaml — the canonical set a citation may reference.""" + if yaml is None: + return list(PROJECT_IMPORT) + try: + data = yaml.safe_load((root / "sources.yaml").read_text(encoding="utf-8")) or {} + names = [p["name"] for p in data.get("projects", []) if isinstance(p, dict) and p.get("name")] + return names or list(PROJECT_IMPORT) + except Exception: # noqa: BLE001 + return list(PROJECT_IMPORT) + + +def _project_tree(project: str, root: Path) -> Optional[Path]: + """A directory holding `project`'s source tree: the installed checkout + (_project_repo), else `sources//`, else a sibling clone of this repo. + None when nothing is present — the caller downgrades to a warning.""" + repo = _project_repo(project) + if repo is not None: + return repo + for candidate in (root / "sources" / project, root.parent / project): + if candidate.is_dir(): + return candidate + return None + + +def check_citations(root: Path) -> int: + """Resolve every `Project:path` citation in skills/, wiki/core/, AGENTS.md and + llms.txt against a checkout of the cited project. A path containing `...` is a + deliberate abbreviation — only its concrete prefix is required to exist. Exit 1 + on any missing path; unresolvable project trees are warnings (packaged install + with no clone), missing paths are errors.""" + projects = _sources_projects(root) + cite_re = re.compile( + r"`(" + "|".join(re.escape(p) for p in projects) + r"):([A-Za-z0-9_./\-]+?)`" + ) + files = sorted((root / "skills").glob("*.md")) + files += sorted((root / "wiki" / "core").rglob("*.md")) + files += [p for p in (root / "AGENTS.md", root / "llms.txt") if p.exists()] + + errors: list[str] = [] + warns: list[str] = [] + seen_missing_tree: set[str] = set() + n_citations = 0 + + def _check_one(rel_file: Path, project: str, cited: str) -> None: + nonlocal n_citations + n_citations += 1 + if project == "autolens_assistant": + tree: Optional[Path] = root # self-citations resolve against this repo + else: + tree = _project_tree(project, root) + if tree is None: + if project not in seen_missing_tree: + seen_missing_tree.add(project) + warns.append( + f"{project}: no checkout resolvable (installed/sources/sibling) — " + f"its citations skipped." + ) + return + concrete = cited.split("...")[0].rstrip("/") + if not concrete: + return + if not (tree / concrete).exists(): + errors.append(f"{rel_file}: `{project}:{cited}` — {concrete} not in {tree}") + + for f in files: + text = f.read_text(encoding="utf-8") + if IDIOM_SKIP_MARKER in text: + continue + rel = f.relative_to(root) + fm_text, body = split_frontmatter(text) + for m in cite_re.finditer(body if fm_text is not None else text): + _check_one(rel, m.group(1), m.group(2)) + # Frontmatter sources[].paths[] make the same claim in structured form. + if fm_text is not None and yaml is not None: + try: + meta = yaml.safe_load(fm_text) or {} + except yaml.YAMLError: + meta = {} + if isinstance(meta, dict): + for source in meta.get("sources") or []: + if not isinstance(source, dict): + continue + project = source.get("project") + if project not in projects: + continue + for path in source.get("paths") or []: + _check_one(rel, project, str(path)) + + for w in warns: + print(f"[citations] warn {w}", file=sys.stderr) + for e in errors: + print(f"[citations] ERROR {e}", file=sys.stderr) + print( + f"[citations] scanned {len(files)} files, {n_citations} citation(s) — " + f"{len(errors)} missing path(s), {len(warns)} warning(s).", + file=sys.stderr, + ) + return 1 if errors else 0 + + # --------------------------------------------------------------------------- # Code gate (--code / --file) # --------------------------------------------------------------------------- @@ -1289,6 +1400,13 @@ def main() -> int: help="With --write-provenance, restrict stamping to this page (repeatable). The " "honest default for a targeted refresh — only stamp what you re-validated.", ) + parser.add_argument( + "--check-citations", + action="store_true", + help="Resolve every `Project:path` source citation (inline + wiki frontmatter " + "sources paths) against a checkout of the cited project and exit (0 ok, 1 " + "missing). Catches pre-refactor path staleness the symbol audit is blind to.", + ) parser.add_argument( "--strict", action="store_true", @@ -1330,6 +1448,8 @@ def main() -> int: return write_provenance(root, only=only) if args.check_provenance: return check_provenance(root, strict=args.strict) + if args.check_citations: + return check_citations(root) # Baseline actions short-circuit the (expensive) Markdown/script scan. if args.check_version: diff --git a/skills/al_audit_skill_apis.md b/skills/al_audit_skill_apis.md index 7b2d2f6..1dc56dc 100644 --- a/skills/al_audit_skill_apis.md +++ b/skills/al_audit_skill_apis.md @@ -155,6 +155,27 @@ honest re-pin, the partner of `al_update_wiki` step 4): python autoassistant/audit_skill_apis.py --write-provenance --page wiki/core/api/.md ``` +### 6. Citation paths — the layout axis the symbol audit is blind to + +A page can cite only live `al.*` symbols while its `` `Project:relative/path` `` +source citations point at a pre-refactor file layout (module → package moves like +`nest/nautilus.py` → `nest/nautilus/`, deleted files, renamed READMEs). The +2026-07-09 release audit found ~30 such stale paths on an otherwise +0-broken-symbols tree — the two are independent failure axes. + +```bash +python autoassistant/audit_skill_apis.py --check-citations # 0 ok, 1 on missing path +``` + +It scans `skills/`, `wiki/core/`, `AGENTS.md` and `llms.txt` for inline +`` `:` `` citations (projects from `sources.yaml`) plus each wiki +page's frontmatter `sources[].paths[]`, and resolves every path against a checkout +of the cited project (installed → `sources//` → sibling clone). A path +containing `...` is a deliberate abbreviation — only its concrete prefix must +exist. Missing paths are ERRORs; a project with no resolvable checkout downgrades +to a warning. Run it alongside the symbol audit in every refresh +(`al_refresh_api_docs`) and before a release. + ERRORs fail the check; warnings (unpinned `main`, unstamped legacy pages) do not unless `--strict`, so the release/PR check goes red on genuine forgery/staleness without nuking the 50+ legacy `main`-pinned pages that predate the discipline. diff --git a/skills/al_refresh_api_docs.md b/skills/al_refresh_api_docs.md index da7f6f7..10faf56 100644 --- a/skills/al_refresh_api_docs.md +++ b/skills/al_refresh_api_docs.md @@ -125,10 +125,13 @@ script that skill would generate. ## Combine — what "complete" looks like -A full refresh pass is complete when all four are true: +A full refresh pass is complete when all five are true: - the PyAuto* stack imports in the target environment - `autoassistant/audit_skill_apis.py --scope ` reports zero misses +- `autoassistant/audit_skill_apis.py --check-citations` reports zero missing paths + (symbols and citation paths are independent failure axes — see + `al_audit_skill_apis` §6) - every touched wiki page has either an updated `pinned_commit` or an explicit decision that the source diff was cosmetic - any materially changed skill recipe has been smoke-tested with `PYAUTO_TEST_MODE=1` diff --git a/wiki/core/stack/autoarray.md b/wiki/core/stack/autoarray.md index 1d3acad..f24816c 100644 --- a/wiki/core/stack/autoarray.md +++ b/wiki/core/stack/autoarray.md @@ -7,7 +7,7 @@ sources: - autoarray/dataset/ - autoarray/mask/ - autoarray/operators/ - - README.rst + - README.md pinned_commit: main last_updated: 2026-07-09 --- diff --git a/wiki/core/stack/autoconf.md b/wiki/core/stack/autoconf.md index 37e719e..07052d4 100644 --- a/wiki/core/stack/autoconf.md +++ b/wiki/core/stack/autoconf.md @@ -6,9 +6,9 @@ sources: - autoconf/conf.py - autoconf/dictable.py - autoconf/json_prior/ - - README.rst + - README.md pinned_commit: main -last_updated: 2026-05-22 +last_updated: 2026-07-09 --- # PyAutoConf — the configuration layer diff --git a/wiki/core/stack/autofit.md b/wiki/core/stack/autofit.md index fc0a8bc..e697c53 100644 --- a/wiki/core/stack/autofit.md +++ b/wiki/core/stack/autofit.md @@ -7,9 +7,9 @@ sources: - autofit/non_linear/ - autofit/aggregator/ - autofit/database/ - - README.rst + - README.md pinned_commit: main -last_updated: 2026-05-22 +last_updated: 2026-07-09 --- # PyAutoFit — model composition + non-linear search diff --git a/wiki/core/stack/autogalaxy.md b/wiki/core/stack/autogalaxy.md index 8a17f61..db52b60 100644 --- a/wiki/core/stack/autogalaxy.md +++ b/wiki/core/stack/autogalaxy.md @@ -8,7 +8,7 @@ sources: - autogalaxy/galaxy/ - autogalaxy/galaxy/ - autogalaxy/cosmology/ - - README.rst + - README.md pinned_commit: main last_updated: 2026-07-09 --- diff --git a/wiki/core/stack/autolens.md b/wiki/core/stack/autolens.md index 1f6d74d..87ea856 100644 --- a/wiki/core/stack/autolens.md +++ b/wiki/core/stack/autolens.md @@ -8,7 +8,7 @@ sources: - autolens/interferometer/model/ - autolens/point/ - autolens/quantity/ - - README.rst + - README.md pinned_commit: main last_updated: 2026-07-09 ---