feat: deep audit phase 2 — skill prose, tooling robustness, workflow skills

[Jammy2211]

Overview

Phase 2 of the reference-assistant audit (phase 1: #33 / PR #34). Deep pass over the assistant's substance: every mature skill body reviewed as instructions for a weaker model (a strong model — Fable — does the reviewing; users run the skills on whatever model they have), tooling robustness, workflow skills, and operational wiki spot-checks.

Autonomy: launched --auto, effective level supervised (feature cap). Plan recorded here per the autonomy contract; the run proceeds through mechanical stretches and parks at ship sign-off with a batched question. No Heart reasons were acknowledged at launch. Merge stays human.

Plan

Three phased PRs under this issue:

• PR A (this run) — skill prose review: read all 19 mature al_* skills + skills/_style.md end-to-end; tighten Orient → Ask → Branch → Combine arcs, surface buried decisions, add missing "when not to use this" and failure modes, fix stale pointers; judge each recipe by whether a weaker model could execute it without inference leaps. No renames, frontmatter contracts stable.
• PR B — tooling robustness: audit_skill_apis.py (incl. the triple-printed version-mismatch wall), the PreToolUse gate, activate.sh/Makefile/config/, version-baseline refresh vs the 2026.7 stack.
• PR C — workflow skills + wiki spot-checks: start-new-project.md (highest-stakes), contribute-upstream.md, init-slam.md, _bootstrap_skill.md; wiki/core/ operational pages vs installed stack.

Detailed implementation plan (PR A)

Affected Repositories

• autolens_assistant (primary, only)

Branch Survey

Repository	Current Branch	Dirty?
./autolens_assistant	main (e124ab1)	clean (stray untracked james.bib, untouched)

Suggested branch: feature/assistant-deep-audit

Review protocol per skill

1. Read the full body. 2. Check against _style.md (arc, adaptive depth, script style, description frontmatter). 3. Weaker-model test: are steps executable without leaps — explicit decisions, explicit checks, explicit failure modes? 4. Verify wiki/workspace pointers resolve; suspicious API reasoning (not symbols — the audit covers those) checked against installed source. 5. Edit in place, smallest diff that fixes the finding.

Skills in scope (19 mature)

Setup & maintenance: al_setup_environment, al_update_wiki, al_audit_skill_apis, al_refresh_api_docs, al_ingest_paper.
Data prep: al_prepare_imaging_data, al_simulate_dataset.
Model building: al_build_imaging_model, al_build_interferometer_model, al_custom_profile.
Fitting: al_configure_search, al_run_search, al_chain_searches, al_run_slam_pipeline, al_debug_fit_failure.
Results: al_load_results, al_plot_tracer, al_plot_fit_residuals, al_inspect_source_reconstruction.
Plus _style.md itself. Stubs untouched unless al_subhalo_detect turns out cheap after the pass.

Validation

Symbol audit (audit_skill_apis.py, expect 0 broken), make validate-literature-citations, link sweep over every edited file, no-regression grep for the two-mode model. Ship gate per AUTONOMY.md; supervised ⇒ park at ship sign-off with the ship summary as a batched question here.

Original Prompt

Click to expand starting prompt

See PyAutoMind/issued/deep_audit_skills_tooling.md (filed 2026-07-08; phased PR A/B/C cut, constraints, and per-PR scope).

[Jammy2211]

Checkpoint — PR A (skill prose review) complete; parked at ship sign-off per the supervised autonomy contract.

What was done: all 19 mature skills read end-to-end against _style.md and every code recipe's symbols introspected against the installed 2026.7.6 stack (the repo-wide symbol audit only covers cited symbols, not code-block claims — that gap is exactly where the real breakage was). Five commits on feature/assistant-deep-audit:

• al_custom_profile: recipe used three nonexistent APIs (Dictable mixin, _radii_from_grid, _b_n) and told users to write priors YAML into the installed PyAutoGalaxy config (source-edit-boundary violation). Rewritten around real base-class helpers; corrected recipe executed successfully; no-default-priors gotcha now explicit.
• al_run_slam_pipeline: recipe imported a slam module + slam_pipeline_main driver that have never existed. Replaced with the real copy-the-template pattern grounded in slam_start_here.py.
• al_configure_search: iterations_per_update renamed upstream; UltraNest/PySwarms no longer shipped — menu now matches dir(af).
• al_load_results: tracer.einstein_mass_angular_from moved to al.LensCalc.
• al_inspect_source_reconstruction: invented inversion attribute names replaced with verified ones; al_chain_searches phase 2 used exactly the Delaunay+ConstantSplit combination that open regression PyAutoArray#332 crashes on — switched to RectangularUniform+Constant with the regression noted inline.
• al_debug_fit_failure: mixed-generation aggregator snippet replaced with the load-results-consistent pattern.
• al_prepare_imaging_data: recipe ended on a bare interactive plot call; now saves + prints the path per the plot rules.
• _style.md + AGENTS.md: macOS-only open offer made platform-aware; typo.
• Clean after full verification: al_run_search, al_build_imaging_model, al_simulate_dataset, al_plot_tracer (1 fix), al_plot_fit_residuals, al_ingest_paper, al_setup_environment, al_update_wiki, al_refresh_api_docs, al_audit_skill_apis (claims skimmed; it is PR B's subject).

Validation: symbol audit 55 files / 116 symbols / 0 broken; citation metadata valid; relative-link sweep clean; corrected custom-profile recipe run end-to-end against the installed stack.

Recorded for PR B (tooling): (1) the PreToolUse gate reads PYAUTO_SKIP_API_GATE from its own process env, so the documented per-command bypass can never work; (2) when the stack import fails (e.g. version check), the gate marks every symbol STALE and prints the full multi-paragraph error once per symbol; (3) audit_skill_apis --check-version prints the same version-mismatch wall three times (once per library import). For PR C (wiki): wiki/core/concepts/non_linear_search.md, wiki/core/api/searches.md decision tables and wiki/core/index.md still recommend UltraNest/PySwarms that the stack doesn't expose (flagged, not rewritten — wiki/core edits go through al_update_wiki).

The question: approve opening the PR for this branch (I recommend a draft PR titled "fix: deep audit PR A — repair broken skill recipes against the installed stack", body carrying this report)? On approval the PR opens and PR B (tooling) starts as the next independent step; if you want changes, comment here and the branch amends first. Merge stays human either way.

[Jammy2211]

Human approved the phase-A ship in session — PR A is open as draft: #36. Proceeding to PR B (tooling robustness) on a fresh branch off main; the three confirmed gate/audit bugs recorded above are its worklist. Merge of #36 stays human.

[Jammy2211]

Checkpoint — PR B (tooling robustness) complete; parked at ship sign-off.

Branch feature/assistant-deep-audit-tooling (off main, independent of PR A / #36), one commit 25422cd:

1. Gate bypass fixed — PYAUTO_SKIP_API_GATE=1 as a command prefix now works (the hook previously read only its own process env, so the documented escape hatch was dead).
2. Broken stack ≠ symbol drift — when the stack fails to import (e.g. the workspace version check), the validator now reports the import failure once (truncated) and exits 3; the hook fails open on exit 3 since the command surfaces the same error itself. Previously every symbol was marked STALE with the full multi-paragraph wall repeated per symbol.
3. Deduplicated import walls — --check-install/--check-version group identical errors ("autofit, autogalaxy, autolens import failed: …" once, not three walls).
4. Makefile audit/test targets; gate skill documents the exit-3 contract and both bypass forms.

Validation: full tooling suite 39 passed (3 new regression tests pin the fixes); both failure modes re-verified live against the current mismatched stack. Deliberately untouched: version.txt pin — release-owned (PyAutoBuild stamps it); hand-bumping against a main dev build would fake a release state. activate.sh/config/ reviewed, no changes needed.

The question: approve opening PR B (draft, this report as body)? On approval it opens and PR C (workflow skills + wiki spot-checks: start-new-project, UltraNest/PySwarms wiki tables via al_update_wiki) starts as the final phase.

[Jammy2211]

Checkpoint — PR C (workflow skills + wiki spot-checks) complete; parked at ship sign-off. This is the final phase.

Branch feature/assistant-deep-audit-wiki (off main, independent of #36/#37), one commit 33120a5:

Workflow skills: start-new-project (the highest-stakes skill) reviewed end-to-end — one real contradiction fixed: project datasets were dataset/<sample>/ in steps 3/5 but data/ in the scaffold tree, .gitignore, manifests and the Publish gate (which audits git ls-files data/); aligned to data/. contribute-upstream hardcoded "Claude Opus 4.7" in the co-author trailer — now the current session's model per AGENTS.md. init-slam's 11 workspace source paths all verified present. _bootstrap_skill sound; doubled-article mangles fixed in it and _style.md.

Wiki (per al_update_wiki discipline, verified against installed dir(af)): four pages still recommended UltraNest/PySwarms, which the installed autofit does not expose — families lists, both pick-a-search tables and the index now name what ships (Nautilus/Dynesty, BFGS/LBFGS) with not-exposed notes. searches.md shared-knobs had two stale kwargs (iterations_per_update, force_pickle_overwrite) → replaced with the verified shared set. samples_and_posteriors no longer names the removed SamplesMLE (af.Samples claim verified). Operational pages (sandbox env vars) spot-checked — all real. Provenance stamping n/a (legacy-pinned pages).

Validation: symbol audit 0 broken; idiom lint clean (181 files); citations valid; link sweep clean.

The question: approve opening PR C (draft, this report as body)? That completes the deep audit — after it opens, the remaining human acts are merging #36, #37 and PR C, then I close out the Mind state and the issue.

[Jammy2211]

PR C is open: #38. All three phases of the deep audit are now at the merge gate — #36 (skill recipes), #37 (tooling), #38 (workflow skills + wiki). Independent branches off main; #36 and #38 both touch _style.md on different lines, so whichever merges second may need a trivial rebase. Merge order suggestion: #36 → #38 → #37 (or any — conflicts at most trivial). Merge and close stay human; on merge I'll clean up branches, complete the Mind state, and append the calibration-log outcomes.

[Jammy2211]

Checkpoint — Phase D (AGENTS.md + general tooling) complete; parked at ship sign-off.

Branch feature/assistant-deep-audit-general (off main), one commit 1f4a9f7, 38 files:

• Stale-org URL sweep: 40 URLs still pointed at github.com/Jammy2211|rhayes777 for repos that are canonically PyAutoLabs/ per sources.yaml — all rewritten (PyAutoLogo correctly left). Worst case: al_setup_environment's editable-clone recipe cloned all five libraries from the old owners while telling the agent to 'resolve URLs from sources.yaml'.
• AGENTS.md cross-harness gap: the code gate was described as ambient, but the PreToolUse hook only exists on Claude Code — Codex/Gemini/OpenCode/Copilot sessions got no enforcement and no fallback instruction. The invariant now tells non-hook harnesses to self-enforce via audit_skill_apis --code/--file before executing generated PyAuto* code. Rest of AGENTS.md re-verified with fresh eyes: plot-API claim true (MatPlot2D/Output gone), --check-version exit codes accurate, llms.txt + .gemini/settings.json wiring current.
• Tooling: refresh_api_docs.py --scope gains scripts (parity with the audit). literature.py, hpc/, activate.sh reviewed — no changes needed.

Validation: symbol audit 0 broken; citations valid; refresh_api_docs.py parses; URL remainder check shows only the unmoved PyAutoLogo.

Conflict note: this branch touches many of the same files as #36/#38 but different lines (URL lines vs recipe/table hunks); merges should be clean or trivially rebased. Suggested merge order: #36 → #38 → #37 → this.

The question: approve opening phase D as draft PR? That truly completes the audit series.

[Jammy2211]

Phase D is open: #39. The deep-audit series is complete — four PRs at the merge gate: #36 (skill recipes) · #37 (tooling) · #38 (workflow + wiki) · #39 (org URLs + cross-harness gate + tooling parity). Suggested merge order #36 → #38 → #37 → #39 (shared files on different lines; at most trivial rebases). Merge and close stay human — on merge I'll delete the four branches, complete the Mind state with calibration outcomes, and close this issue.

[Jammy2211]

Deep audit complete — all four phases merged to main: #36 (skill recipes, squash 46cde1b) → #38 (workflow + wiki, ee40372) → #37 (tooling, b0ff0f1) → #39 (org URLs + cross-harness gate, 94c3c3e; one mechanical merge-conflict resolution in the #332 warning block, combining #38's wording with the canonical org URL). Branches deleted. Outstanding items carried forward: the version-pin skew stays a release-time action (Heart tracks it); skill-stub completion (phase 2 of the original audit prompt) remains in the Mind backlog.