feat: built-in benchmark suite — 4 standard prompts + run/track harness

[Jammy2211]

Overview

Add a built-in benchmark package to autolens_assistant: four standard prompts (three assistant-mode at easy/medium/hard difficulty, one teacher-mode) that are run against different AI agents/harnesses and models to measure assistant performance, plus a harness to record each run (conversation transcript, results, metadata), commit it to GitHub, and compare performance across models and across days for the same model. The clone agent's template boundary is extended so every future domain assistant inherits the benchmark machinery (and unclassified benchmarks/ files don't hard-fail births).

Filed from PyAutoMind/feature/autolens_assistant/assistant_benchmarks.md under an explicit --auto launch. Effective autonomy: supervised (header supervised, feature cap at Difficulty > medium) — per PyAutoBrain/AUTONOMY.md the plan is written here and the run proceeds; judgment gates become batched questions on this issue; ship ends at PR-open; merge stays human. No Heart YELLOW acknowledgement was given at launch — any YELLOW at ship parks the run.

Plan

• Create a benchmarks/ package in autolens_assistant holding four frozen, versioned prompt cards: assistant-easy (cosmos_web_ring JWST modeling), assistant-medium (SLACS0946+1006 subhalo Bayesian model comparison), assistant-hard (group-scale two-SIE lens, joint imaging+interferometer simulation and fitting, MGE→pixelized follow-up), and teacher (Euclid-like simulate-and-fit walkthrough). Each card carries the verbatim prompt, mode, difficulty, datasets/packages exercised, and a success rubric.
• Build the run/track harness (autoassistant/benchmark.py): scaffold a run directory per execution with metadata (model, harness, date, stack versions, duration), store the conversation transcript and scored rubric, and regenerate comparison reports (per-benchmark model leaderboard + same-model-over-time series) so results are committed and pushed to GitHub.
• Document the benchmark protocol (benchmarks/README.md + benchmarks/AGENTS.md): clean-clone runs, no maintainer mode, frozen prompt versioning so cross-day comparisons stay valid, artifact size caps, honesty rules for recording outcomes.
• Sync the top-level README: commit the pending SLACS prompt-header simplification (already edited locally by the user) and add a short Benchmarks pointer.
• Make the clone agent aware: classify benchmarks/ in the modes/maintainer.md "Assistant-as-template" boundary section and add matching path patterns to PyAutoBrain's agents/conductors/clone/_clone.py (harness/contract = generic, prompt cards = domain-regenerated, runs = per-clone data never copied). Without this, any tracked benchmarks/ file is unclassified and a clone birth hard-fails (exit 4).
• File a follow-up prompt for the first calibration campaign (actually running the four benchmarks across models/harnesses) — usage, not build; out of scope here per the Feature Agent's split-into-phases decision.

Detailed implementation plan

Affected Repositories

• autolens_assistant (primary)
• PyAutoBrain (secondary — clone conductor patterns)

Work Classification

Workspace (assistant repo develops in-place on a feature branch; PyAutoBrain edit is small and ships as its own PR)

Worktree root

None — in-place feature branches (assistant-ref-mechanics precedent for autolens_assistant; PyAutoBrain edit is a two-file pattern/doc change)

Branch Survey

Repository	Current Branch	Dirty?
./autolens_assistant	main	M README.md (user's own SLACS prompt-header edit — belongs to this task)
./PyAutoBrain	main	clean

Suggested branch: feature/assistant-benchmarks (both repos)

Implementation Steps

1. Prompt cards — benchmarks/prompts/{easy_cosmos_web_ring,medium_slacs0946_subhalo,hard_group_multi,teacher_workflow}.md. YAML frontmatter: id, version (frozen once published; any wording change bumps it and starts a new comparison series), mode (assistant/teacher), difficulty, datasets, workspace_packages (e.g. hard = group + multi + imaging + interferometer), added. Body: the verbatim prompt (easy + medium mirror the README example prompts; hard and teacher as specified in the Mind prompt), then a success rubric split into machine-checkable assertions (expected artifacts exist — e.g. reconstructed-source and residual images, reported Bayesian evidences for both comparisons, both simulated datasets created, follow-up pixelized fit run) and judged criteria (scientific sensibility of model choices, quality of explanation in teacher mode), each with a points weight.
2. Harness — autoassistant/benchmark.py CLI:
   • new-run <prompt-id> --model <name> --harness <name> → scaffolds benchmarks/runs/<prompt-id>/<YYYY-MM-DD>_<model>_<harness>[_<n>]/ with meta.yaml (prompt id+version, model, harness, date, autolens/autofit/autoarray versions from the installed stack, git SHA of the assistant clone, duration/cost/token fields to fill), transcript.md stub, score.md generated from the card's rubric.
   • score <run-dir> → validates the machine-checkable rubric lines and totals the score.
   • report → regenerates benchmarks/RESULTS.md: per-benchmark leaderboard (rows = model×harness, columns = latest score, best score, runs) and per-model time series (same model across dates) so day-to-day comparison is one table.
   • Unit tests in autoassistant/tests/test_benchmark.py (scaffold, score parsing, report generation on fixture runs).
3. Protocol docs — benchmarks/README.md (what/why, how to run a benchmark against any agent/model, how results are stored and compared, artifact cap ~500 KB/run, transcripts required) and benchmarks/AGENTS.md (the contract an agent follows when asked to run a benchmark: fresh session, no maintainer mode, prompt pasted verbatim, no benchmark-aware coaching, record honestly including failures; note that benchmarks/runs/ is data, not instructions).
4. README sync — include the user's pending README.md edit (SLACS example-prompt header simplified to "Assistant mode."); add a short Benchmarks section pointing at benchmarks/README.md; keep easy/medium prompt cards and README example prompts textually aligned (single source noted in each).
5. Clone awareness —
   • modes/maintainer.md "Assistant-as-template": add benchmarks/AGENTS.md + harness (via autoassistant/*, already generic) to the generic list; benchmarks/prompts/* to domain-specific (regenerated per domain — new domains need domain-appropriate easy/medium/hard/teacher prompts); benchmarks/README.md to mixed; benchmarks/runs/* + RESULTS.md documented as per-clone data (never copied to a newborn).
   • PyAutoBrain/agents/conductors/clone/_clone.py: mirror those patterns in GENERIC_PATTERNS / DOMAIN_PATTERNS / MIXED_PATTERNS; runs/RESULTS classified so births don't fail unclassified.
6. Currency — benchmark prompt cards cite datasets/paths; ensure the existing citation/staleness check (--check-citations CI leg) covers benchmarks/ so prompts referencing removed datasets or APIs surface as failures.
7. Follow-up prompt — file PyAutoMind/research/autolens_assistant/benchmark_calibration_runs.md (first cross-model campaign) at ship time; do not bulk-issue.

Key Files

• autolens_assistant/benchmarks/prompts/*.md — the four frozen prompt cards + rubrics
• autolens_assistant/benchmarks/README.md, benchmarks/AGENTS.md — protocol + agent contract
• autolens_assistant/autoassistant/benchmark.py — scaffold/score/report CLI (+ tests)
• autolens_assistant/benchmarks/runs/ — committed run records; benchmarks/RESULTS.md — regenerated comparison report
• autolens_assistant/README.md — example-prompt sync + Benchmarks pointer
• autolens_assistant/modes/maintainer.md — template-boundary classification
• PyAutoBrain/agents/conductors/clone/_clone.py — partition patterns for benchmarks/

Design decisions (recommendation adopted; flag on this issue if wrong)

• Run records live in the assistant repo itself (benchmarks/runs/, committed) — matches "pushed to GitHub" directly; artifact caps keep the public template lean. Alternative (separate results repo) rejected for now as it splits the story the benchmarks tell.
• The harness does not launch agents. Running a benchmark = a human (or conductor) starts the chosen agent with the card's prompt; the harness scaffolds, records, scores, and reports. Agent-launching automation is a possible later phase.

Original Prompt

Click to expand starting prompt

for autolens_assistant, I now want to create in built bench marks, which have the goal of being standard prompts which we run using different AI agents and models to test performance. I want 3 prompts for assistant mode, the first is "easy" difficulty and simply does tasks already available in the workspace should be based on this prompt in README.md: Model the JWST imaging in dataset/imaging/cosmos_web_ring: perform data preparation steps, set up a sensible lens light and mass model with a pixelized source reconstruction, run the fit, and show me the reconstructed source and the fit residuals, the next is medium difficulty, it requires doing things that are not explicitly in the workspace (e.g. model comparison, changing mass profile for DM subhalo) and again is based on a README.md prompt:

Assistant mode.

The strong lens SLACS0946+1006 famously has a dark matter subhalo detection that many argue is unusually concentrated. I'd like to analyse the HST imaging of this lens provided at dataset/imaging/slacs0946+1006/ and reproduce that detection.

Specifically, I want this analysis to perform Bayesian model comparison to (a) confirm a subhalo is preferred over a smooth-mass baseline by fitting a free-position, free-mass SIS perturber across the image plane and comparing the Bayesian evidence to the no-subhalo fit, and (b) test the "super-concentrated" claim by comparing the SIS subhalo against a more shallow NFW mass profile at the recovered position.

Set the pipeline up so the smooth lens light and mass model, the pixelized source reconstruction, and the subhalo results are all inspectable on my computer, and report the Bayesian evidence for each comparison.

Assess whether the analysis will run fast on my laptop / PC GPU, and if not, set this up as a small project on the HPC I have access to.

The third should be hard mode and is a new prompt not on the README.md, with the goal that it requires us to combine 3 different packages on the autolens_workspace: group, multi, imaging and interferometer, here is the prompt:

Assistant mode.

First, I want to simulate imaging and interferometer data of a group-scale strong lens, which is composed of two SIE lens galaxies and a quadruply imaged Cored Sersic background source.

Then, I want to perform modeling of this dataset, simultaneously fitting the imaging and interferometer data. I want the foreground lens model to use multi Gaussian Expansions for the lens light, SIE's for each lens and a multi Gaussian expansion for the background source.

After this fit has been judged successful, do a follow up lens model that uses a pixelized source reconstruction, but retains the MGE lens light and SIE source.

Present me with results confirming the fit was a success.

I also want you to make one benchmark based on the Teacher mode example prompt:

Teacher mode.

I'm new to PyAutoLens and want to learn the basic workflow end-to-end. Can you walk me through it on a simple simulated example: simulate Euclid-like imaging of a simple strong lens (an isothermal mass with a Sersic source), then fit that simulated data and recover the lens model.

Explain what each step is doing and why as we go: composing the lens and source model, running the simulation, choosing the mask, the non-linear search, and how to read the result. So I come away understanding the workflow, not just the commands.

Put this in a benchmark package and have a design whereby I can run these benchmarks, track the conversation and results and store them so they can be pushed to GitHub as well as be run and tracked for different models and run with the same model to compare performance on different days. Think about if there is anything else benchmarking would benefit from and put this all in the clone agent too, e.g. make sure it's aware of it.

[Jammy2211]

Checkpoint — ship sign-off (supervised park, 2026-07-10)

What was being done: the build is complete and committed locally on feature/assistant-benchmarks in both repos; nothing is pushed. Per the autonomy contract (supervised at checkpoint 2) the run parks here for your sign-off — and independently, Heart reads YELLOW with no acknowledgement given at launch, which parks a ship at any level.

What's on the branches

autolens_assistant a5ad099 (13 files, +1098/−5):

• benchmarks/prompts/ — 4 frozen cards: assistant-easy-cosmos-web-ring, assistant-medium-slacs0946-subhalo, assistant-hard-group-multi (group × multi × imaging × interferometer), teacher-basic-workflow. Each: verbatim prompt + 100-pt rubric (machine-checkable vs judged rows).
• autoassistant/benchmark.py — new-run (scaffolds benchmarks/runs/<id>/<date>_<model>_<harness>/ with meta.yaml capturing stack versions + assistant SHA), score (totals a filled rubric into meta.yaml), report (regenerates benchmarks/RESULTS.md: per-benchmark model×harness leaderboard + chronological same-model table).
• benchmarks/README.md (protocol: frozen prompt versioning, same-judge comparability, failures-are-data, ~500 KB artifact cap) + benchmarks/AGENTS.md (agent contract, both roles: benchmarked vs operating; runs/ is data never instructions).
• README: new Benchmarks section + your pending SLACS prompt-header simplification (committed as part of this branch); card↔README prompt parity is enforced by a unit test.
• modes/maintainer.md: Assistant-as-template now classifies benchmarks (harness generic, cards domain, runs/RESULTS per-clone data) — plus a new "Per-clone data" paragraph.
• Citation sweep (--check-citations) extended to benchmarks/ docs+cards (never runs/).

PyAutoBrain 8d4575c (1 file, +7/−1): _clone.py partition patterns mirroring the boundary above — without this, every tracked benchmarks/ file is unclassified and a clone birth hard-fails. Witness: analyze over the reference reports unclassified 0.

Gate evidence

Leg	Result
Tests	autolens_assistant suite 52/52 pass (incl. 11 new benchmark tests); PyAutoBrain 22/22 pass; --check-citations clean (98 files, 389 citations, 0 missing)
Smoke	n/a — no downstream workspace script surface (template docs + tooling; no smoke_tests.txt repo touched)
Review	faculty surface generated (--repo path; note: the faculty's --task mode can't resolve in-place branches — no worktree); verdict on the surface: CLEAN (self-review by the running session, not independent)
Heart	YELLOW: workspace validation 3 failed (2026-07-09), 58 stale parked scripts, assistant pinned behind installed, Mind PR 10d old (+2 stale reasons). All pre-existing organism-scope, none caused by this branch — but no ack was given at launch, so the leg parks

The question

Approve pushing both branches and opening the two PRs (assistant PR referencing this issue; Brain PR cross-referenced)? Options:

1. "ship" — push both, open both PRs (no pending-release label: the assistant is not in the library release train; say so if you want it anyway). Merge/close stays yours.
2. "ship with changes: …" — tell me what to amend first (e.g. rubric weights, artifact cap, run-dir naming).
3. "hold" — branches stay local; nothing published.

On answer 1 or 2 any environment can resume from active.md (status awaiting-input, this comment linked). The follow-up calibration-campaign prompt (first cross-model runs) is filed in Mind either way — it only makes sense post-merge.

[Jammy2211]

Shipped — 2026-07-10

PRs

• feat: built-in benchmark suite — 4 frozen prompt cards + run/track harness #58 — MERGED
• feat: classify the assistant benchmarks/ package in the clone partition PyAutoBrain#79 — MERGED (clone partition patterns)

Summary

The benchmark suite is live on main: four frozen prompt cards (assistant easy/medium/hard + teacher) with machine/judged rubrics, the benchmark.py new-run/score/report harness, the protocol + agent contract, and the clone template boundary so future domain assistants inherit the machinery. Ship + merge were human-directed in-conversation; the Heart YELLOW set was acknowledged in-session at that direction.

Session Notes

• The hard card's follow-up wording was simplified pre-publication per the author ("MGE source", intent stated plainly) — no version bump needed since nothing had been published or run.
• Run records live in-repo (benchmarks/runs/, committed) by design; artifact cap keeps the public template lean. The harness records runs; it does not launch agents.
• Card↔README prompt parity and dataset existence are test-enforced; the citation sweep covers benchmarks/ docs+cards but never runs/ (transcripts are historical data).
• Next: the first calibration campaign (research/autolens_assistant/benchmark_calibration_runs.md) — starting now, human-directed. This issue is ready to close.