maint: design lock-in + results/dashboard groundwork (polish phase 1)

[Jammy2211]

Overview

autolens_profiling is mature and well-separated, and is about to become the home of the PreOptimizationTimes baseline — the last full profiling before the optimization push. Before those runs (phases 2–4), this task does one final holistic design review to lock the core design in for future extension (more datasets, instruments, packages), and puts the results-presentation story in place so campaign results land as browsable .md/.json on GitHub.

Phase 1 of 4 of PyAutoMind task maintenance/autolens_profiling/polish.md. Autonomy: running under --auto at effective level supervised (checkpoint-and-continue per PyAutoBrain/AUTONOMY.md) — plan below is the approved plan of record.

Plan

• Deep design review of the package layout (instruments, likelihood, likelihood_runtime, likelihood_breakdown, vram, latent, searches, simulators, quick_update, hpc) and the shared _profile_cli.py helper; propose and apply small lock-in refactors only where they make extension cheaper — no behaviour-changing rewrites.
• Define the PreOptimizationTimes results convention: versioned .json (existing) plus a rendered .md view per package, so every result is readable on GitHub without opening JSON.
• Audit per-package docs: add missing READMEs (likelihood/, quick_update/, hpc/, scripts/), fix stale content (roadmap still points at retired PyAutoPrompt registry; results/README.md misses the runtime/breakdown/notes/quick_update sections).
• Design the top-level README dashboard and update scripts/build_readme.py (queued path-update follow-up since the likelihood split; CI already checks its idempotence) so phases 2–4 slot results straight in.
• Decide the home of campaign artifacts: runtime results currently aggregate from autolens_workspace_developer/jax_profiling/ — settle whether PreOptimizationTimes lives wholly in this repo.

Detailed implementation plan

Affected Repositories

• autolens_profiling (primary, only)

Branch Survey

Repository	Current Branch	Dirty?
./autolens_profiling	main	clean

Suggested branch: feature/profiling-polish-design
Work Classification: Workspace
Worktree root: ~/Code/PyAutoLabs-wt/profiling-polish-design/

Implementation Steps

1. Survey every package's script inventory + _profile_cli.py/sweep.py interfaces; write a short design note (kept in-repo, e.g. results/notes/) recording what is locked in and any agreed renames/moves.
2. Apply approved lock-in refactors (small, ruff-clean, smoke-import safe under AUTOLENS_PROFILING_SMOKE=1).
3. Add .md result rendering to the results pipeline (JSON → per-package markdown table), tagged PreOptimizationTimes.
4. Add/repair per-package READMEs and results/README.md sections; fix stale roadmap/registry references in the top-level README.
5. Update scripts/build_readme.py to the post-split paths and design the dashboard section (populated in phase 4); keep --check green in lint.yml.

Key Files

• _profile_cli.py — shared sweep flags / output-path resolver; the extension surface to lock in
• likelihood_runtime/sweep.py, likelihood_runtime/aggregate.py — campaign driver + cross-repo aggregation to settle
• scripts/build_readme.py — retired dashboard generator, queued for path update; CI runs --check
• results/README.md, per-package README.mds — conventions + missing docs

Original Prompt

Click to expand starting prompt

polish phase 1 — holistic design review and lock-in

Type: maintenance
Target: autolens_profiling
Difficulty: medium
Autonomy: supervised
Priority: normal
Status: formalised

Phase 1 of 4 of polish.md (see parent for full intent). No profiling runs in
this phase — design and results-presentation groundwork only.

autolens_profiling is now a mature project with good separation into packages
(instruments, latent, likelihood_runtime, likelihood_breakdown, vram, …). Before
the PreOptimizationTimes baseline campaign (phases 2–4), do deep research on the
project and have one last think about ways to improve it: redesigns that make it
more concise and clearer. We will be extending it with more datasets, packages
and instruments, so this is the opportunity to lock the core design in well.

Also scan the repo and make sure that when profiling results come in they are
saved as .md or .json files and clearly displayed in per-package GitHub
.md files for browsing. Design (but do not yet populate) a high-level results
dashboard on the GitHub README. The baseline result set will be named
PreOptimizationTimes — the comparison base for the optimization work that
follows.

Out of scope: any profiling runs (phases 2–4); searches; point_source; laptop
GPU (user runs those in a follow-up); the future PyAutoBrain profiling agent
idea recorded in the parent prompt.

[Jammy2211]

Milestone 1 — design review done, groundwork implemented (branch feature/profiling-polish-design, 2 commits)

The deep design review verdict: the core is in good shape — the runtime/breakdown/vram package split, the <class>/<model> cell grid, _profile_cli.py as the single CLI surface, and instrument framing are all locked in unchanged. Findings and decisions are recorded in-repo at results/notes/design_lock_in.md.

What the review surfaced and fixed:

• Results now live wholly in this repo — sweep.py/aggregate.py defaulted their output to autolens_workspace_developer/jax_profiling/results/jit (a migration leftover); the default is now results/runtime/. Also dropped the hard-coded /home/jammy/venv/... interpreter default (sys.executable now), added imaging/mge to the sweep grid, and gave sweep.py a --sparse passthrough so the sparse-vs-mapping comparison runs through the same driver.
• The README dashboard was dormant — scripts/build_readme.py still targeted the likelihood/ package retired in the runtime/breakdown split and scanned 0 artifacts. Retargeted to the real results/ sections: the top-level README now renders a live headline (the existing v2026.5.29.4 breakdown table populates it already), per-package READMEs got auto-table sections, and a named-baseline column hook reads results/baselines/<Name>/ once populated.
• PreOptimizationTimes convention defined — a baseline is a frozen, append-only snapshot under results/baselines/PreOptimizationTimes/ mirroring the runtime layout plus a rendered one-page .md; the dashboard grows a baseline column automatically.
• Docs polish — stale likelihood/, z_projects/, and PyAutoPrompt references fixed across AGENTS.md/READMEs; quick_update/, hpc/, scripts/ gained READMEs; results/README.md rewritten around the two artifact shapes.
• Repo-wide ruff cleanup (separate commit) — CI installs unpinned latest ruff and 0.15.x flags 204 pre-existing violations, which would have failed the next PR regardless; auto-fixes + format applied, two real dead-code finds removed, B905/B007 added to the ignore policy with rationale.

Verified: ruff check + format --check clean, build_readme.py --check idempotent, compileall clean, 8/8 unit tests, smoke imports pass on one script per area.

Next: ship gate (/ship_workspace) → PR. Phases 2–4 (vram-first validation, the PreOptimizationTimes runtime campaign, breakdown + dashboard population) follow as separate tasks.

[Jammy2211]

Checkpoint — ready to ship, awaiting your go (supervised autonomous run: parking per the autonomy contract; everything below is answerable cold)

Where things stand. Phase 1 is implemented and verified on feature/profiling-polish-design (local worktree, 3 commits, not yet pushed):

1. maint: design lock-in + results/dashboard groundwork — the design-review deliverables (see milestone comment above and results/notes/design_lock_in.md).
2. style: ruff 0.15 cleanup — repo-wide mechanical lint/format pass (CI installs unpinned ruff; the next PR faces 0.15.x regardless of this task).
3. fix: retarget per-cell and latent output defaults in-repo — a real find from the review leg: HPC submit scripts pass no --output-dir, so A100 results would have landed in the retired results/likelihood/ layout; per-cell defaults now match the sweep layout, and aggregate.py filters config stems so summaries can't corrupt comparison.json.

Ship-gate legs. Tests: 8/8 unit tests pass; downstream library suites n/a (standalone leaf repo, no PyAuto* source touched). Smoke: six-workspace curated smoke n/a (no downstream script surface); the repo's own AUTOLENS_PROFILING_SMOKE=1 import gate passes on one script per area. Review: CLEAN after fixing the finding in commit 3. Heart: YELLOW — 9 reasons (workspace-test failures from 2026-06-09, 55 stale parked scripts, autolens_assistant version pin behind, install/release validation not run, 4 old open PRs). None touch this repo.

The question: ship now — push the branch and open the pending-release PR — acknowledging the YELLOW reason set as unrelated to this change?

• (a) Ship (recommended — every YELLOW reason predates this task and lives outside this repo; the PR gate on this repo is lint.yml, which is green locally) → I push, open the PR with the ## Scripts Changed + validation checklist, record the heart-ack list, set the task to awaiting-merge, and append the calibration row. Merge stays with you.
• (b) Hold until the organism is GREEN → task stays parked as-is; nothing is pushed.
• (c) Request changes → comment what to change; I'll iterate on the branch first.

Task state: active.md → awaiting-input, pointer to this comment. Answer here or relaunch in any session — the state carries.

[Jammy2211]

Shipped to PR — you answered (a): PR #53 is open with the pending-release label, Heart-YELLOW acknowledgement recorded in the task state (active.md → awaiting-merge, calibration row appended to the autonomy log).

Per the autonomy contract the run stops at PR-open — merge stays with you. On merge, phase 2 (maintenance/autolens_profiling/polish_phase_2_vram.md — the vram-first validation sweep that also computes the vmap batch sizes phase 3 consumes) becomes ready to start.

[Jammy2211]

Shipped — 2026-07-08

PRs

• maint: design lock-in + results/dashboard groundwork (polish phase 1) #53 (merged)

Summary

Phase 1 (design lock-in + results/dashboard groundwork) is merged to main. The core design is locked and recorded in results/notes/design_lock_in.md; all output defaults now converge in-repo (results/runtime/, results/latent/); the README dashboard is live again and renders the existing breakdown artifacts; the PreOptimizationTimes baseline convention is defined and the dashboard grows a baseline column automatically once results/baselines/PreOptimizationTimes/ is populated.

Session Notes

• Key decision: campaign artifacts live wholly in this repo — the workspace_developer tree is frozen history. A cell dir may hold both sweep configs and versioned summaries; aggregate.py filters config-shaped stems.
• Gotcha for phase 2/3: the HPC submit scripts pass no --output-dir, so they depend on the per-cell defaults fixed here — HPC checkouts must pull this merge before any A100 campaign run, or results land in the retired layout.
• The ruff cleanup means CI's unpinned pip install ruff is green at 0.15.x; if a future ruff release breaks the gate again, consider pinning in lint.yml instead of another mass-format.
• Next: phase 2 maintenance/autolens_profiling/polish_phase_2_vram.md (vram-first validation; computes the vmap batch sizes phase 3 consumes), then phases 3–4. Laptop-GPU rows and the PyAutoBrain profiling agent remain user follow-ups.