docs: release polish — API docs, cluster maturity, 3 learning paths

[Jammy2211]

Overview

Release-readiness documentation polish across the PyAutoLens stack. Three
independent pieces: (i) audit the docs/api/*.rst reference pages of PyAutoFit,
PyAutoGalaxy and PyAutoLens so they are clean and current; (ii) drop the "in
development / beta" framing from cluster examples now that cluster (and weak)
lensing are mature, user-facing features; (iii) document the three ways to learn
PyAutoLens (manual workspace navigation, an llms.txt-driven chat assistant, and
fully agentic AI) in the overview guide and both READMEs.

Filed from PyAutoMind/docs/autolens/release_docs_polish_and_learn_paths.md.
Run autonomously under --auto at effective level supervised (docs cap at
too-large) — plan written here, ship sign-off parks per the autonomy contract.

Plan

• Phase A — API docs audit (audit_docs): check docs/api/*.rst in
  PyAutoGalaxy and PyAutoLens for broken module paths, missing classes and stale
  autosummary entries; fix in place. PyAutoFit's audit is parked — its
  worktree is claimed by ep-graphical-docs (a PyAutoFit docs task) — to avoid a
  collision; it resumes once that ships.
• Phase B — cluster maturity de-flagging (autolens_workspace): remove the
  "IN DEVELOPMENT" / "Beta Feature" language from scripts/cluster/README.md and
  scripts/cluster/start_here.py, rewriting it as a mature, usable feature;
  regenerate the paired notebook. Weak lensing carries no dev flags — verified,
  no change needed there.
• Phase C — three ways to learn: after the sentence "The autolens_workspace
  contains a suite of example Jupyter Notebooks, organised by lens scale and
  dataset type", add a subsection presenting three learning paths —
  (1) manual navigation (read the workspace guides; flags the two questions
  that follow), (2) AI chat assistant via the llms.txt interface with a
  concrete ChatGPT-style example, (3) fully agentic AI via Claude / Codex —
  pointing to autolens_assistant for both. Applied to
  PyAutoLens/docs/overview/overview_2_new_user_guide.md,
  autolens_workspace/start_here.py (§418, + notebook), and the README.md of
  both PyAutoLens and autolens_workspace. Also soften the "experimental / not
  recommended" assistant note in PyAutoLens/docs/index.md to match the new
  third-way framing.

Resolves to 3 PRs (PyAutoLens, autolens_workspace, PyAutoGalaxy) plus a
parked PyAutoFit API-docs sub-phase.

Detailed implementation plan

Affected Repositories

• PyAutoLens (primary) — Phase A + Phase C
• autolens_workspace — Phase B + Phase C
• PyAutoGalaxy — Phase A
• PyAutoFit — Phase A (parked: worktree conflict with ep-graphical-docs)

Branch Survey

Repository	Current Branch	Dirty?	Worktree claim
./PyAutoLens	main	overview_2 reflow (WIP, trivial)	free
./autolens_workspace	main	stray dataset.fits (untracked)	free
./PyAutoGalaxy	main	clean	free
./PyAutoFit	main	clean	claimed by ep-diagnostics, ep-graphical-docs

Suggested branch: feature/release-docs-polish-learn-paths
Worktree root: ~/Code/PyAutoLabs-wt/release-docs-polish-learn-paths/
Work Classification: Both (library + workspace)

Implementation Steps

Phase C — three learning paths (highest-specified, do first)

1. PyAutoLens/docs/overview/overview_2_new_user_guide.md — insert the
   "Three ways to learn" subsection immediately after the "suite of example
   Jupyter Notebooks…" sentence; option 1 = manual navigation, flagging the two
   questions below.
2. autolens_workspace/start_here.py (line ~418) — same subsection, in-comment
   prose matching the script's markdown-in-docstring style; regenerate
   start_here.ipynb.
3. PyAutoLens/README.md and autolens_workspace/README.md — add a concise
   "Three ways to learn PyAutoLens" section (link autolens_assistant:
   https://github.com/PyAutoLabs/autolens_assistant).
4. PyAutoLens/docs/index.md:12 — reword the assistant note from
   "experimental / not the recommended starting point" to present agentic AI as a
   legitimate third path.
   • llms.txt example source: autolens_assistant/README.md:51–54; llms.txt
     files live at PyAutoLens/llms.txt, autolens_workspace/llms-full.txt.

Phase B — cluster de-flagging (autolens_workspace)
5. scripts/cluster/README.md — remove the "# IN DEVELOPMENT" header + prose;
reframe as a supported feature.
6. scripts/cluster/start_here.py:29,61–73 — remove the "Beta Feature" bullet
and "Beta Feature" block; keep any genuine caveats as ordinary notes.
Regenerate notebooks/cluster/start_here.ipynb.

Phase A — API docs audit (audit_docs skill)
7. PyAutoGalaxy docs/api/*.rst (8 files) — audit + fix.
8. PyAutoLens docs/api/*.rst (10 files) — audit + fix.
9. PyAutoFit docs/api/*.rst (8 files) — parked; comment on this issue when
ep-graphical-docs ships, then resume.

Key Files

• PyAutoLens/docs/overview/overview_2_new_user_guide.md — Phase C anchor sentence
• autolens_workspace/start_here.py — §418 anchor sentence (+ notebook)
• PyAutoLens/README.md, autolens_workspace/README.md — Phase C sections
• PyAutoLens/docs/index.md — assistant framing
• autolens_workspace/scripts/cluster/{README.md,start_here.py} — Phase B
• {PyAutoGalaxy,PyAutoLens,PyAutoFit}/docs/api/*.rst — Phase A

Testing / validation

• Docs-only: no library unit tests affected. Validate rst via the audit_docs
  checks; validate regenerated notebooks parse. Ship each repo's changes as its
  own pending-release PR behind the four-leg autonomous-ship gate; supervised →
  park at ship sign-off with a question on this issue.

Original Prompt

Click to expand starting prompt

I am readying for a PyAuto release and want to do the following on PyAutoLens docs:

(i) make sure API docs are clean and up to date (do this also for PyAutoGalaxy and PyAutoFit);

(ii) clusters and weak lensing are now working and mature and thus can be explained to users as things they can go and use (e.g. no "in development" flags); make sure this is also reflected on autolens_workspace;

(iii) in overview_2_new_user_guide.md, the equivalent location in autolens_workspace, and the README.md files of PyAutoLens and autolens_workspace, explain that there are 3 ways to learn to use PyAutoLens:
(i) reading workspace guides;
(ii) asking questions to an AI assistant like ChatGPT, which uses the llms.txt interface, giving an example of how a user does that (there is an example somewhere in the source code; the user points to the URL for the autolens_assistant);
(iii) fully agentic AI use via Claude, Codex, pointing to autolens_assistant for more information.

Put these 3 options as a subsection after this sentence: "The autolens_workspace contains a suite of example Jupyter Notebooks, organised by lens scale and dataset type", with the first option being "manual navigation" flagging the questions below that are about to follow.

[Jammy2211]

--auto checkpoint (supervised) — ready for sign-off + one scope decision

Effective level supervised (docs cap at too-large). Per the autonomy contract, ship sign-off parks for human review and a scope fork parks with a question. Nothing is committed or pushed; all edits sit in the worktree ~/Code/PyAutoLabs-wt/release-docs-polish-learn-paths on feature/release-docs-polish-learn-paths. Resume with claude --resume b7b89cda-1dac-48c9-8880-7d467cd91f58.

✅ Phase C — Three Ways to Learn (DONE, awaiting ship sign-off)

Added a "Three Ways to Learn PyAutoLens" subsection (manual navigation → chat assistant → fully agentic AI) immediately after the "suite of example Jupyter Notebooks…" sentence, and reframed the assistant from "experimental / not recommended" to a first-class option.

• PyAutoLens/docs/overview/overview_2_new_user_guide.md — subsection inserted; option 1 (manual navigation) flags the two questions that follow.
• PyAutoLens/docs/index.md — reworded the assistant note.
• PyAutoLens/README.md — reworded the assistant blockquote + new "Three Ways to Learn" section.
• autolens_workspace/start_here.py (§ Where To Next) + regenerated start_here.ipynb.
• autolens_workspace/README.md — new "Three Ways to Learn" section.

Per mid-run feedback, options 2/3 were trimmed: no llms.txt mechanics and no "if the chat can't browse GitHub" caveat — a user is just told to point ChatGPT/Claude at autolens_assistant and ask.

✅ Phase B — cluster maturity de-flagging (DONE, awaiting ship sign-off)

Cluster lensing is now presented as a supported feature (weak lensing already carried no dev flags — verified, unchanged).

• autolens_workspace/scripts/cluster/README.md — removed the "# IN DEVELOPMENT" block.
• autolens_workspace/scripts/cluster/start_here.py — "Beta Feature" section → "Capabilities" (kept the genuine strengths + a practical visualization tip; dropped "in beta"/"in active development" language). Regenerated notebooks/cluster/start_here.ipynb (+ its README copy).

Scripts compile; scripts/check_sizes.sh clean. Two workspace repos here (autolens_workspace) and one library-docs repo (PyAutoLens); no library code changed, so no unit-test surface — docs/tutorial only.

⛔ Phase A — API-docs audit: real, larger-than-expected finding (needs a decision)

Audited docs/api/*.rst for PyAutoGalaxy + PyAutoLens against the installed stack (2026.7.6.649). Modules all import fine, but many autosummary entries reference classes that no longer exist (verified truly absent via a full pkgutil walk of autoarray/autogalaxy/autolens/autofit, not just top-level):

A1 — the whole OO plotting subsystem is gone (dominant issue). plot.rst in both repos documents MatPlot1D/2D, Visuals1D/2D, Units/Figure/Axis/Cmap/Colorbar/…/Title/Legend, and every *Scatter / *Overlay / overscan-plot wrapper (~34 entries per file). These were removed when plotting moved to the functional API (aplt.plot_array(...), aplt.subplot_*(...), output_path= kwargs — no MatPlot2D/Cmap/Output objects). Only Output survives (in autoarray.plot). So plot.rst describes a subsystem that no longer exists.

A2 — individual removed/moved refs (mechanical):

• Removed entirely: autolens.OverSampling, autolens.PointDict, autolens.SetupAdapt, autofit.PySwarmsLocal, autofit.PySwarmsGlobal (modeling.rst, data.rst, point.rst).
• Moved: Chameleon / ChameleonSph — docs point at light_linear_and_mass_profiles; they live in light_and_mass_profiles (re-point currentmodule).
• Duplicate Convolver entry in data.rst (both repos) → dedupe.

❓ Decision needed — how should Phase A handle plot.rst (A1)?

• (A) Prune plot.rst to only the surviving symbols now — fast and makes the docs correct, but leaves plotting under-documented.
• (B) Rewrite plot.rst to document the new functional plot API — the proper fix, but a real docs task in its own right.
• (C) [recommended] Do (A) now (prune the dead OO entries) and apply all A2 mechanical fixes in this PR, then file a separate follow-up prompt for (B) the functional-API rewrite.

On your answer I'll execute Phase A (A2 + the chosen A1 path) in the PyAutoGalaxy/PyAutoLens worktrees. Until then PyAutoGalaxy is untouched.

⏸ PyAutoFit API-docs audit — parked (not a decision)

PyAutoFit is claimed by the ep-graphical-docs worktree (itself a PyAutoFit docs task, PR #1334). To avoid a collision its API-docs audit is deferred; I'll pick it up once #1334 merges. (Its plot.rst almost certainly has the same removed-OO-plotting issue.)

🧹 Out-of-scope hygiene finding (FYI — not touched)

Regenerating notebooks surfaced that the workspace's generated artifacts are globally stale: generate.py rebuilt/added ~30 unrelated notebooks (missing weak/, cluster/lenstool/, guides/ notebooks; drift in imaging/, group/) plus llms-full.txt / workspace_index.json. I discarded all of that and committed only the two notebooks matching my two edited scripts. The drift is a real but separate maintenance task — recommend a dedicated "regenerate workspace notebooks + catalogue" prompt.

---

On sign-off: reply to greenlight Phase B/C ship (→ ship_workspace + ship_library, PRs opened against pending-release) and pick A/B/C for Phase A. I'll then execute Phase A and ship. Merge/close stay with you.

[Jammy2211]

Phase A executed — option C (prune + mechanical fixes now; functional-API rewrite deferred)

Re-audit is clean for PyAutoGalaxy + PyAutoLens (zero broken modules / missing names / dups). All edits are byte-safe (CRLF preserved on the CRLF files — no line-ending churn).

Pruned (dead OO plotting subsystem): removed the "Plot Customization" + "Matplotlib Wrappers" sections from plot.rst in both repos (−80 lines each — MatPlot1D/2D, Visuals1D/2D, all wrapper/*Scatter classes). Surviving Plotters + Non-linear Search sections kept.

Mechanical fixes:

File	Change
PyAutoGalaxy/docs/api/data.rst	removed SettingsImaging, SettingsInterferometer, duplicate Convolver
PyAutoGalaxy/docs/api/modeling.rst	removed PySwarmsLocal/Global, AdaptImageMaker
PyAutoLens/docs/api/data.rst	removed duplicate Convolver; OverSampling → OverSampler (rename)
PyAutoLens/docs/api/modeling.rst	removed PySwarmsLocal/Global; SetupAdapt → AdaptImages (rename, keeps the Adapt section)
PyAutoLens/docs/api/point.rst	removed PointDict
PyAutoLens/docs/api/mass.rst	removed dead linear Chameleon/ChameleonSph (already documented correctly under light_and_mass_profiles)

Follow-ups filed as PyAutoMind prompts:

• docs/autolens/plot_rst_functional_api_rewrite.md — the positive rewrite of plot.rst to the functional plot API (incl. PyAutoFit once #1334 merges).
• maintenance/autolens_workspace/regenerate_notebooks_catalogue.md — the ~30-file global notebook/catalogue drift.

PyAutoFit API-docs audit remains parked behind ep-graphical-docs (#1334); it has the same plot.rst issue and is folded into the rewrite prompt.

Ready to ship — final diff (all implementation done)

• PyAutoLens (8 files): Phase A rst (data/mass/modeling/plot/point) + Phase C (README, docs/index, overview_2).
• PyAutoGalaxy (3 files): Phase A rst (data/modeling/plot).
• autolens_workspace (7 files): Phase B cluster de-flag + Phase C (start_here + README + 2 notebooks).

Awaiting ship sign-off (supervised --auto keeps ship + merge human): on your go I'll run ship_library (PyAutoLens, PyAutoGalaxy) + ship_workspace (autolens_workspace) → pending-release PRs behind the library-first gate.

[Jammy2211]

✅ Shipped — 3 PRs open (autonomous ship gate passed, ended at PR-open)

Human ship sign-off given; four-leg gate passed — tests n/a (0 Python source changed), smoke n/a (workspace .py diffs are docstring-only), review CLEAN (re-audit confirms all autosummary refs resolve; CRLF preserved), Heart YELLOW with the 5-reason chronic set human-acknowledged at ship.

PR	Repo	Contents
#593	PyAutoLens	Phase C (3-ways guide, README, docs/index) + Phase A (rst prune)
#492	PyAutoGalaxy	Phase A (rst prune)
#254	autolens_workspace	Phase B (cluster de-flag) + Phase C (start_here, README, notebooks)

All labelled pending-release. Library-first merge gate: merge #593 + #492 before #254. Merge + issue-close stay with you.

Still open (deliberately):

• PyAutoFit API-docs audit — parked behind ep-graphical-docs (#1334); folded into the plot.rst rewrite follow-up.
• Follow-up prompts filed in PyAutoMind: docs/autolens/plot_rst_functional_api_rewrite.md and maintenance/autolens_workspace/regenerate_notebooks_catalogue.md.

[Jammy2211]

Merged all three PRs (human-authorized): PyAutoLens#593, PyAutoGalaxy#492, autolens_workspace#254 (squash, remote branches deleted). Phases A (autogalaxy+autolens), B, C complete. PyAutoFit API-docs audit remains as the filed follow-up (behind ep-graphical-docs #1334). Closing.