docs: restore truncated HowTo tutorials + add completeness CI check

[Jammy2211]

Overview

Several HowTo tutorial scripts were cut off mid-generation when the HowTo
repos were bootstrapped from the workspace tutorials — the model hit an output
cutoff while re-emitting a long file, and the truncated scripts have been frozen
at that length ever since (the original workspace source no longer exists). In
HowToLens/scripts/chapter_1_introduction, tutorial_1_grids_and_galaxies.py
and tutorial_2_ray_tracing.py both end on a docstring that promises a plot
which never appears, with no trailing code and no Wrap Up. This restores the
lost content and adds a CI check so the regression cannot silently recur.

Plan

• Audit every tutorial across HowToLens (all chapters) for the truncation
  signature: ends on a docstring that promises a plot/code that never
  follows, with no terminal Wrap Up. (Confirmed so far: ch1 tutorial_1,
  tutorial_2. Deliberate not written yet stubs — ch1 tutorial_4_point_sources,
  tutorial_5_lensing_formalism — are NOT truncation and must be excluded.)
• Fully restore each truncated script: author the missing code + prose (Opus
  tutorial-prose work), mirroring the complete HowToGalaxy/HowToFit sibling
  where one exists, lens-flavoured, ending in a proper Wrap Up.
• Add a tutorial-completeness linter + CI workflow to HowToLens that fails when
  a tutorial ends mid-docstring / lacks a terminal Wrap Up section.
• Ship HowToLens as the phase-1 PR.
• Repeat the same audit → restore → linter/CI in HowToGalaxy (phase 2) and
  HowToFit (phase 3), each as its own docs-only PR.

Detailed implementation plan

Affected Repositories

• HowToLens (primary, phase 1)
• HowToGalaxy (phase 2)
• HowToFit (phase 3)

Branch Survey

Repository	Current Branch	Dirty?
./HowToLens	main	dirty (20 files, all under dataset/ — generated artifacts, no source edits)
./HowToGalaxy	main	dirty (13 files, all dataset/)
./HowToFit	main	dirty (315 files, all dataset/)

Suggested branch: feature/restore-truncated-howto-tutorials

Root cause

Line counts for the affected scripts have been frozen since the bootstrap
commit (62a5f3a in HowToLens, "bootstrap HowToLens from workspace tutorials")
— a classic LLM output-token cutoff during that generation, baked into commit
#1. Not a runtime bug. tutorial_1 is 429 lines vs the complete 654-line
HowToGalaxy sibling.

Truncation signature (what the audit + linter must detect)

A script is truncated when its final docstring block promises output that
never appears — e.g. ends on "...plotted in log10 space reveals more details."
or "...in logarithmic space:" with no plt/aplt call after it — and there is
no terminal Wrap Up section. A missing literal __Wrap Up__ header is NOT
sufficient on its own: tutorial_4_dealing_with_failure and
tutorial_4_bayesian_regularization end on genuine prose conclusions and are
complete. The linter should therefore require every non-stub tutorial to end
with a recognized terminal section (Wrap Up / summary), and normalize the
prose-conclusion tutorials under that marker so they pass.

Implementation Steps (phase 1 — HowToLens)

1. Audit all HowToLens/scripts/**/tutorial_*.py for the signature above;
   produce the confirmed truncated set (seed: ch1 tutorial_1, tutorial_2).
2. For each, restore missing code + prose, mirroring the HowToGalaxy sibling
   (e.g. tutorial_1_grids_and_galaxies → log10 plot + Over Sampling section +
   Wrap Up), lens-flavoured. Validate the added code executes (mirror
   already-proven sibling plot patterns; do NOT run in TEST_MODE).
3. Add HowToLens/scripts/check_tutorials_complete.py + a .github/workflows
   CI step invoking it; wire the stub-exclusion list.
4. Ship phase-1 PR (docs-only, no library changes).

Phases 2-3

Repeat steps 1-4 for HowToGalaxy then HowToFit (per user: HowToLens first).

Key Files

• HowToLens/scripts/chapter_1_introduction/tutorial_1_grids_and_galaxies.py — truncated at 429 lines
• HowToLens/scripts/chapter_1_introduction/tutorial_2_ray_tracing.py — truncated at 214 lines
• HowToGalaxy/scripts/chapter_1_introduction/tutorial_1_grids_and_galaxies.py — complete 654-line reference sibling
• HowToLens/scripts/check_tutorials_complete.py — new completeness linter
• HowToLens/.github/workflows/*.yml — CI wiring

Original Prompt

Click to expand starting prompt

The first two scripts in HowToLens/chapter_1 have lost python code and cut off,
which is an issue we diagnosed and solved before. Investigate this, work out why
they were cut off, restore them, then look for all instances of this happening
across all scripts and repos, fix all, and prevent this from happening long term.

(Filed prompt: docs/workspaces/restore_truncated_howto_tutorial_scripts_and_add.md
— Type docs, Target workspaces, Difficulty medium, Autonomy supervised, Priority high.)

[Jammy2211]

Phase 1 (HowToLens) shipped → #27 (pending-release, awaiting your merge).

Restored the two truncated chapter-1 tutorials:

• tutorial_1_grids_and_galaxies (429 → 672 lines): log10 image + __Galaxies__ + __Units__ + __Wrap Up__ + advanced topics, adapted from the complete HowToGalaxy sibling to the autolens API and lens framing.
• tutorial_2_ray_tracing (214 → 391 lines): log-space plots + __Ray Tracing Grids__ (lens equation) + __Ray Tracing Images__ + __Galaxies__ + __Tracer__ + __Mappings__ + __Wrap Up__.

Both validated end-to-end; curated smoke 6/6.

Root cause confirmed: LLM output-token cutoff while re-emitting a long file during the workspace bootstrap (frozen at the truncated length since commit #1; original workspace source gone). Only these two scripts were truncated in HowToLens — the deliberate not written yet stubs are correctly left alone.

Prevention: .github/scripts/check_tutorials_complete.py + tutorials_complete.yml CI — every non-stub tutorial must reach a terminal __Wrap Up__/__Summary__ section (a truncated script never does). Verified it catches the pre-restore tutorial_1/2 and passes the restored tree. Five complete-but-unmarked tutorials were normalized with the marker.

Next: phases 2–3 apply the same audit → restore → CI to HowToGalaxy then HowToFit, each as its own PR. This issue stays open until all three land.

[Jammy2211]

Phases 2 & 3 shipped — 3-repo rollout complete (all PRs pending-release, awaiting your merge).

Phase	Repo	PR	Work	Smoke
1	HowToLens	#27	Restored truncated tutorial_1 + tutorial_2; linter + CI; 5 normalizations	6/6
2	HowToGalaxy	#19	Filled empty tutorial_4_methods (stub); 3 normalizations; linter + CI	4/4
3	HowToFit	#16	4 normalizations (incl 2 commented-code endings + a bare "Finish."); linter + CI	10/10

Key finding: the only genuinely truncated tutorials across all three repos were HowToLens ch1 tutorial_1 and tutorial_2 — exactly the two you reported. HowToGalaxy and HowToFit had no lost content; their linter failures were complete-but-unmarked tutorials (prose conclusions / end-on-code / optional commented appendices) plus HowToGalaxy's empty tutorial_4_methods (0 bytes since bootstrap → filled with a not written yet stub per your call).

Long-term prevention is now in all three repos: .github/scripts/check_tutorials_complete.py + a Tutorials Complete CI workflow that fails any PR where a non-stub tutorial doesn't reach a terminal __Wrap Up__/__Summary__ section — a truncated script never gets there, so this catches the regression on every push.

Once all three PRs merge, this issue can close.

[Jammy2211]

All three phases merged: HowToLens#27, HowToGalaxy#19, HowToFit#16. Truncated ch1 tutorial_1/tutorial_2 restored, and the tutorials-complete CI guard is now live in all three HowTo repos. Closing.