feat: Colab maturity — generated setup cells, setup_colab registry, URL guards

[Jammy2211]

Overview

A census of the Google Colab infrastructure (2026-07-09) found that while the release-time machinery is solid (tag bumper wired into release.yml, central Heart URL sweep), only 9 of 489 notebooks are actually Colab-runnable — docs and READMEs link 100+ notebooks (all HowTo chapters, the PyAutoFit flagship badge) that die on ModuleNotFoundError. This task makes Colab support mature and maintainable by construction: a generalized setup_colab registry in PyAutoConf, generation-time injection of a standard setup cell into every notebook in PyAutoBuild, and hardened forbidden-URL guards in PyAutoHeart. After this merges, the next release regenerates all notebooks Colab-ready automatically, with zero per-script maintenance.

Phase 2 (link rot: HowTo chapter README URLs, library-docs dead links, allowlist purges) is parked at PyAutoMind/feature/workspaces/colab_link_rot.md and will be issued when this phase nears shipping.

Plan

• Generalize autoconf.setup_colab into one parameterized per-project registry covering all 6 notebook repos (autofit/autogalaxy/autolens workspaces + HowToFit/HowToGalaxy/HowToLens), fixing the GPU-detection bug, the import-time env mutation, and duplicated package lists; clone the release-matched tag shallowly; add the module's first unit tests.
• Inject a standard Colab setup cell (markdown + code) into every notebook PyAutoBuild generates, parameterized by project, skipping notebooks that already carry a hand-written setup; add tests and document the end-to-end Colab architecture in docs/internals.md.
• Extend PyAutoHeart's offline URL guard with the two rot classes the census found: unpinned blob/main Colab URLs (invisible to the release bumper) and Colab links to notebooks/chapter_* paths in workspace repos (chapters live in the HowTo repos). This intentionally turns the Monday sweep red until phase 2 fixes the rot.

Detailed implementation plan

Affected Repositories

• PyAutoBuild (primary)
• PyAutoConf
• PyAutoHeart

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoConf	main	clean
./PyAutoBuild	main	clean
./PyAutoHeart	main	clean

Suggested branch: feature/colab-maturity
Work Classification: Library
Worktree root: ~/Code/PyAutoLabs-wt/colab-maturity/ (created by /start_library)

Implementation Steps

1. PyAutoConf — autoconf/setup_colab.py rewrite

   • _PROJECTS registry keyed autofit | autogalaxy | autolens | howtofit | howtogalaxy | howtolens → package list, workspace repo URL, workspace dir, display name, GPU error message. One shared extras list (pyvis, dill, jaxnnls, nautilus-sampler, timeout_decorator, anesthetic) + per-project core stack — kills the duplicated lists in for_autolens/for_autogalaxy.
   • Public API: setup(project: str, raise_error_if_not_gpu: bool = False). Keep for_autolens / for_autogalaxy as thin back-compat wrappers (released notebooks at old tags call them); add for_autofit and HowTo wrappers.
   • Fix check_jax_using_gpu: accumulate across all devices (today the last device wins) and handle the empty-device case (today UnboundLocalError).
   • Delete the module-level os.environ['XLA_FLAGS'] mutation (line 4) — it already happens inside _colab_setup; today every from autoconf import setup_colab clobbers user env even outside Colab.
   • Clone --depth 1 --branch <installed-version> (version via importlib.metadata.version of the project's top package), falling back to the default branch — removes the version skew between the pip-installed release, the tagged notebook, and today's main-HEAD clone; --depth 1 makes the clone fast on the large workspaces.
   • New test_autoconf/test_setup_colab.py: registry completeness/shape, wrapper delegation, clone tag-fallback (mocked subprocess), GPU-detection logic on stub device objects (numpy-only — no JAX in library unit tests), no import-time env mutation.

2. PyAutoBuild — generation-time injection

   • autobuild/build_util.py: new inject_colab_setup(notebook_path, project) — prepend a markdown cell (what the cell does, GPU-runtime tip) and a code cell (try: import google.colab → bootstrap pip install autoconf --no-deps → from autoconf import setup_colab; setup_colab.setup("<project>")). Skip injection when the notebook source already contains setup_colab (the 9 hand-written start_here setups keep working, no double setup).
   • autobuild/generate.py: call the injector after both py_to_notebook sites (root start_here*.py + the scripts loop), mapping the project CLI arg to the registry key. Copy-files (verbatim .py) are not notebooks and are not injected.
   • Tests (tests/): cell injected with valid JSON, idempotent, skip-if-present, project key threaded correctly.
   • docs/internals.md: new "Colab architecture" section — bootstrap module (PyAutoConf) → generation-time injection (Build) → release-time URL bumper (bump_colab_urls.sh) → weekly central sweep (Heart).

3. PyAutoHeart — guard hardening

   • heart/checks/url_check.sh: two new forbidden entries: colab.research.google.com/github/PyAutoLabs/(autofit_workspace|autogalaxy_workspace|autolens_workspace|HowToFit|HowToGalaxy|HowToLens)/blob/main/ ("pin to a date tag — the release bumper skips main") and colab.research.google.com/github/PyAutoLabs/(autofit|autogalaxy|autolens)_workspace/blob/<ref>/notebooks/chapter_ ("chapter tutorials live in the HowTo repos").
   • Extend tests/test_url_check.py with both patterns (positive + negative cases).

Key Files

• PyAutoConf/autoconf/setup_colab.py — runtime bootstrap, rewritten around the registry
• PyAutoConf/test_autoconf/test_setup_colab.py — new test module
• PyAutoBuild/autobuild/build_util.py — py_to_notebook + new inject_colab_setup
• PyAutoBuild/autobuild/generate.py — injection call sites
• PyAutoBuild/docs/internals.md — Colab architecture docs
• PyAutoHeart/heart/checks/url_check.sh — forbidden-pattern guard
• PyAutoHeart/tests/test_url_check.py — guard tests

Trade-offs

• Setup cells go into all notebooks, including advanced pipelines impractical on Colab — harmless (no-op outside Colab, and bootstrap is still correct if opened there).
• The --no-deps install strategy is deliberately kept (Colab ships the scientific stack; full dependency resolution is slow and can break Colab's numpy).
• Injected cells use raise_error_if_not_gpu=False for frictionless first contact; the markdown cell tells users how to enable a GPU runtime.
• The new Heart guard patterns will fail the Monday sweep until phase 2 lands — accepted forcing function.

Original Prompt

Click to expand starting prompt

See PyAutoMind/issued/colab_maturity.md — includes the full 2026-07-09 census (coverage counts, URL-form inventory, allowlist rot, setup_colab defect list).

We have most of the infrastructure in place to make workspace exampels and tutorials run in Google colab, such that users can mess around with everything without installation. This includes docs pointing to them, links on our README.md, etc. But, the infrastructure could be developed in a more mature and robust way, including better on going maintenance during release and whatnot. Given we only have Fable access until Sunday, can you do a census of the Google Colab setup for all projects and do the work to make it more polished, mature and maintainable?

[Jammy2211]

Library PRs Created — phase 1 complete at PR-open

All three phase-1 PRs are open with pending-release, full test suites green:

Repo	PR	Tests
PyAutoConf	PyAutoLabs/PyAutoConf#120 — setup_colab registry (all 6 notebook repos), GPU/env fixes, tag-pinned shallow clones	132 passed (15 new)
PyAutoBuild	#125 — Colab setup cell injected into every generated notebook	78 passed (7 new)
PyAutoHeart	PyAutoLabs/PyAutoHeart#44 — url_check forbids unpinned blob/main + workspace chapter_* Colab URLs	232 passed (5 new)

Verification beyond unit tests: end-to-end py_to_notebook + injection on a real HowToLens tutorial (cells placed after the title, correct project key), skip-path verified on imaging/start_here.py (hand-written setup preserved), and the import chain smoke-checked with the worktree autoconf (import autolens clean; XLA_FLAGS ownership confirmed to live in jax_wrapper, which autoconf/__init__ always imports — the removed module-level mutation was redundant, so local behavior is byte-identical).

Merge coupling: merge PyAutoConf#120 and PyAutoBuild#125 together — the injected cell calls setup_colab.setup(), which reaches users via the next release (notebooks only regenerate at release time, so ordering is safe by construction). PyAutoHeart#44 is independent, but once merged the Monday sweep goes red on existing link rot until phase 2 lands — intended forcing function.

Next: phase 2 (link rot: HowToLens blob/main READMEs, HowToGalaxy wrong-repo links, dead filenames, allowlist purges, euclid README) is parked at PyAutoMind/feature/workspaces/colab_link_rot.md, to be issued when these merge. No workspace migration is needed for phase 1 (API changes are added-only; existing for_autolens/for_autogalaxy calls unchanged).

[Jammy2211]

Phase 1 merged 🎉

All three PRs are merged: PyAutoLabs/PyAutoConf#120 + #125 (together, per the injection↔registry coupling) and PyAutoLabs/PyAutoHeart#44. From the next release, every generated notebook is Colab-runnable by construction.

Phase 2 (link-rot fixes + allowlist purges) starts now under --auto from PyAutoMind/feature/workspaces/colab_link_rot.md — keeping this issue open as the phase tracker until the Monday sweep can go green.

[Jammy2211]

Phase 2 merged + sweep validation

All four link-rot PRs merged (HowToLens#22, HowToGalaxy#16, HowToFit#13, euclid#24 — merged-unchanged, calibration logged) and a manually-dispatched central URL sweep confirms HowToFit, HowToGalaxy, HowToLens, autofit/autogalaxy/autolens workspaces are clean (offline + live).

What still keeps the sweep red (full detail on PyAutoLabs/PyAutoHeart#10):

• PyAutoLens + PyAutoGalaxy — exactly the deferred docs leg (chapter_optional removed tutorials, tutorial_6_model_fit, tutorial_4_complex_source, tutorial_11_adapt_regularization.py.ipynb + stale allowlist entries). Registered as colab-link-rot-libdocs in PyAutoMind planned.md, blocked on release-docs-polish-learn-paths which holds both repos.
• Unrelated hygiene (not Colab): euclid README has a https://https:// double-scheme typo (the offline guard's hhttps:// pattern misses this form — worth adding); PyAutoFit pyproject.toml VCS pins (git@<sha>) are naive-HEAD false positives worth allowlisting; conda.io/miniconda.html returned HTTP 429 (transient rate-limit); autolens_workspace has two dead raw.githubusercontent data links (pyRRG, SMACS0723).

This issue stays open until the libdocs leg lands and the sweep goes green.

[Jammy2211]

Colab maturity: complete 🎉

The final libdocs leg merged (PyAutoLabs/PyAutoLens#594, PyAutoLabs/PyAutoGalaxy#493 — human-directed claim override, zero file overlap with the in-flight docs task verified) and a fresh central sweep confirms zero Colab URLs flagged anywhere in the ecosystem.

Full scope delivered across both phases:

• setup_colab registry for all 6 notebook repos (PyAutoConf#120)
• Colab setup cell injected into every generated notebook — coverage 9/489 → all, by construction at the next release (PyAutoBuild#125)
• Hardened forbidden-URL guards (PyAutoHeart#44)
• All link rot repaired: 71 + 28 + 6 dead/unpinned/wrong-repo links across HowToLens, HowToGalaxy, HowToFit, euclid, PyAutoLens docs, PyAutoGalaxy docs; 39 grandfathered allowlist entries purged

The sweep's remaining flags are unrelated pre-existing hygiene, tracked on PyAutoLabs/PyAutoHeart#10: conda.io HTTP 429 (transient), PyAutoFit pyproject.toml VCS-pin false positives, two dead raw-data links in autolens_workspace (pyRRG, SMACS0723), and a https://https:// typo in the euclid README. Closing this tracker.