docs: markdown example renderings with images (generator + autolens pilot)

[Jammy2211]

Overview

Add a markdown-rendering pipeline to PyAutoBuild: curated flagship workspace examples are executed for real and rendered to .md files with their output images, committed to the workspace repo, so new users browsing GitHub can read the examples cleanly formatted with plots visible. GitHub already renders the output-stripped .ipynb mirror, so a no-image .md mirror adds nothing — the whole value is the executed images, which is why the set stays small and curated. This task builds the generator and pilots it on autolens_workspace; rollout to the remaining dataset types, the other workspaces and HowTo is phase 2.

Plan

• Build a generator in PyAutoBuild that converts each curated script to a notebook (reusing the existing py→ipynb machinery), executes it for real, and renders .md + PNG output images into a markdown/ folder mirroring scripts/, plus an index page.
• Curated list is workspace-owned config (config/build/markdown_examples.yaml); anything under a features/ folder is refused by rule.
• Execution is never TEST_MODE (truncated sampling produces wrong images). Model-fits rely on PyAutoFit's completed-run resume: the first build pays one full real sampling run per modeling script, and every regeneration loads the completed result near-instantly with correct plots. A per-script timeout guard fails loudly rather than truncating.
• Regeneration is manual / at-release, only when a flagged script changes — never per-commit.
• Pilot on autolens_workspace: root start_here.py, the imaging five (start_here, simulator, likelihood_function, fit, modeling), and guides/{galaxies,lens_calc,tracer}.py. Commit the generated .md + images and link the index prominently from the README.
• Ship two PRs behind the four-leg gate: PyAutoBuild (generator + tests) and autolens_workspace (config, generated artifacts, README link). Ends at PR-open; merge stays human.

Detailed implementation plan

Affected Repositories

• PyAutoBuild (primary — the generator)
• autolens_workspace (pilot — curated config, generated markdown/, README link)

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoBuild	main	clean
./autolens_workspace	main	1 stray untracked dataset.fits (left alone)

Suggested branch: feature/markdown-example-renderings
Worktree root: ~/Code/PyAutoLabs-wt/markdown-example-renderings/

Implementation Steps

1. PyAutoBuild/autobuild/generate_markdown.py — new CLI, run from the workspace root like generate.py (python ../PyAutoBuild/autobuild/generate_markdown.py autolens [--only <substring>]):
   • Read the curated list from <workspace>/config/build/markdown_examples.yaml: a flat list of entries, each {script: <path relative to workspace root>, max_minutes: <int, default 60>}.
   • Hard guards: refuse any script path containing features/; refuse to run if PYAUTO_TEST_MODE is set (wrong images); error loudly on a missing script.
   • Per script: build_util.py_to_notebook(...) (reuse — do NOT re-implement the docstring→markdown-cell conversion; do NOT call inject_colab_setup), execute the notebook with cwd = workspace root and cell timeout = max_minutes*60 (nbclient or jupyter nbconvert --to notebook --execute), then nbconvert --to markdown into markdown/<script relpath without scripts/ prefix>/ — nbconvert emits <name>.md + <name>_files/*.png alongside.
   • Inject a header line into each .md: auto-generated notice + links back to the source .py and .ipynb.
   • Generate markdown/README.md index from the curated list (title + one-line hook per example, taken from the script's opening docstring title).
   • git add -f all generated outputs (mirrors copy_to_notebooks); verify none are swallowed by .gitignore (git check-ignore) — images must be tracked.
2. PyAutoBuild tests (tests/) — unit tests for config parsing, the features/ and TEST_MODE guards, and script→markdown path mapping. No execution in tests.
3. autolens_workspace/config/build/markdown_examples.yaml — the pilot list: start_here.py (workspace root), scripts/imaging/{start_here,simulator,likelihood_function,fit,modeling}.py, scripts/guides/{galaxies,lens_calc,tracer}.py. Generous max_minutes for imaging/modeling.py (real sampling, first run only).
4. Run the pilot build in the task worktree with the PyAuto venv: only imaging/modeling.py (and possibly the two start_here scripts if they sample) pays a real search; all others are fast real runs. The completed-run cache lands in the worktree's gitignored output/ — re-derivable, so worktree cleanup losing it is acceptable (images are committed).
5. autolens_workspace/README.md — prominent link near the top: browse the examples with output images → markdown/README.md.
6. Verify rendering on GitHub — check the PR's rich diff / branch view shows the images inline (relative <name>_files/ paths must resolve).
7. Ship — ship_library for PyAutoBuild, ship_workspace for autolens_workspace (library-first gate), four-leg autonomous-ship gate, PR-open.

Key Files

• PyAutoBuild/autobuild/generate.py — existing py→ipynb pipeline; pattern for config resolution, iter_script_paths, git add -f
• PyAutoBuild/autobuild/build_util.py — py_to_notebook (reused as-is)
• autolens_workspace/config/build/ — home of the new markdown_examples.yaml (same pattern as copy_files.yaml)
• autolens_workspace/scripts/imaging/*.py, scripts/guides/*.py, start_here.py — the pilot curated set

Phase 2 (separate prompt, filed when this ships)

Remaining autolens dataset types (interferometer, point_source, multi, group, weak; cluster is known-heavy — needs a runtime decision), autogalaxy_workspace + autofit_workspace curated sets, and the HowTo trio (curated list there still to be decided). The generator is built repo-agnostic so phase 2 is config + execution only.

Autonomy

--auto, effective safe (docs ≤ medium after phasing; header safe). Plan recorded here per the contract; run proceeds to PR-open behind the four-leg gate. No Heart YELLOW acknowledgement was given at launch — any YELLOW at ship parks the run.

Original Prompt

Click to expand starting prompt

Markdown renderings (with images) of curated workspace + HowTo examples

Type: docs
Target: PyAutoBuild
Difficulty: small
Autonomy: safe
Priority: normal
Status: formalised

Generate markdown (.md) renderings of a small curated subset of flagship examples across the workspaces (autofit/autogalaxy/autolens_workspace) AND the HowTo tutorials (HowToFit/HowToGalaxy/HowToLens), executed with real output images, so new users browsing GitHub can read them cleanly formatted with plots visible. Scope decided at intake: curated subset ONLY (skip a blanket no-image .md mirror — GitHub already renders the output-stripped .ipynb files natively, so no-image .md adds little); images committed in each workspace/HowTo repo next to its .md (PNGs at fixed dpi, .gitignore must not swallow them), not hosted externally. Generator slots into the PyAutoBuild pipeline alongside the existing py->ipynb generation: execute flagged scripts, nbconvert to markdown, write .md + image dir, plus an index page linked prominently from workspace and HowTo READMEs ('Browse the examples with output'). Regeneration is manual/at-release (real runs, not TEST_MODE), only when a flagged script changes. Original motivation verbatim: 'we could also make markdown (.md) files from the .py files... on github when a user browses the github webpages they can literally read every example there in a way that is more cleanly formatted than .py files... could do for a small subset of examples (e.g. to entice new users).'

Curated list (user-decided 2026-07-10): each workspace's start_here.py plus, per dataset type, simulator.py, likelihood_function.py, fit.py and modeling.py; NOTHING from any features/ folder; autolens_workspace guides/galaxies, guides/lens_calc and guides/tracer.py; and the base workspace start_here.py. Execution-time constraint: the image build must NOT spend ages sampling, but PYAUTO_TEST_MODE truncates the search and would produce WRONG images — think carefully here (e.g. run modeling searches once to completion and cache/load results for regeneration, or use fast-but-real search settings validated to give correct plots; TEST_MODE output images are unacceptable).

[Jammy2211]

Progress update (autonomous run, effective safe):

• Generator built — autobuild/generate_markdown.py: reuses py_to_notebook, executes each curated script for real, cleans captured stream output (ANSI stripped, \r progress resolved, long search logs truncated head+tail), renders via nbconvert --to markdown into markdown/ mirroring scripts/, prepends an auto-generated banner with links back to the .py/.ipynb, and generates the markdown/README.md index. Guards: refuses PYAUTO_TEST_MODE, refuses features/ paths, requires a clean tracked tree and restores tracked files scripts modify — needed because imaging/simulator.py rewrites its tracked dataset/imaging/simple/ with a fresh noise realization every run (noise_seed=-1).
• 19 unit tests green (tests/test_generate_markdown.py).
• Pilot validated end-to-end — guides/tracer.py rendered in 209s: 12 PNGs (872K), all image refs resolve, staged and not gitignored (the workspace's **/images/ ignore rule doesn't touch nbconvert's *_files/ naming).
• Full curated build now running (9 scripts): root start_here.py, imaging start_here/simulator/fit/likelihood_function/modeling, guides tracer/galaxies/lens_calc. The two Nautilus scripts pay their one-time real sampling runs (n_live=100) — expected to take hours; subsequent regenerations resume from the completed output/ cache near-instantly.
• README link added ("Browse Examples With Images") + docs/internals.md entry.

Next: verify the built pages, then ship both PRs behind the four-leg gate (PyAutoBuild first, autolens_workspace behind the library-first merge gate).

[Jammy2211]

Shipped to PR-open (autonomous run complete — merge stays human):

• Library PR: feat: generate_markdown.py — executed markdown renderings of curated examples #137 — generate_markdown.py + 22 tests (suite 100 green) + internals doc. Merge this one first.
• Workspace PR: docs: executed markdown renderings of curated examples (browse on GitHub) autolens_workspace#263 — 9 curated examples fully executed with real output images (83 PNGs, ~19M), index + README links, plus two docstring title fixes the index surfaced (Iamging typo, stale Fits on tracer). Preview as users will see it: markdown/README.md on the branch.

Run notes worth knowing:

• The sampling-cost question resolved cleanly: both Nautilus fits ran to completion for real (2h05m ring, 1h55m simple), and regeneration via the completed-run resume cache was measured at 122s for modeling.py — so wrong TEST_MODE images were never needed and regeneration is cheap forever.
• Review pass found and fixed two real issues before ship: local machine paths leaking into published pages (now redacted, catch-all ~), and stale <name>_files/ PNGs surviving regeneration (now cleaned).
• The tracked-file protection restored dataset rewrites from both simulator.py and (unexpectedly) imaging/start_here.py — without it the PR would carry binary dataset churn.
• Heart YELLOW (7-reason set) was human-acknowledged in-session; gate details in each PR's validation checklist.

Phase 2 (remaining dataset types, autogalaxy/autofit workspaces, HowTo trio) is filed as docs/pyautobuild/markdown_example_renderings_rollout.md — gated on these PRs merging. Cluster stays excluded pending a runtime decision; the HowTo curated list needs a human call.

[Jammy2211]

Merged and complete — #137 (700042d) and autolens_workspace#263 (49277d04) both merged 2026-07-10, human-authorized in-session; branches deleted, worktree cleaned (the completed-search resume cache was copied to the canonical checkout's output/ first, so regeneration stays ~2min). Live pages: https://github.com/PyAutoLabs/autolens_workspace/blob/main/markdown/README.md — linked from the README. Phase 2 (remaining dataset types + autogalaxy/autofit + HowTo) is filed as docs/pyautobuild/markdown_example_renderings_rollout.md, to be run on Opus per the user. Closing.