Skip to content

maint: vram-first validation — probe override, record-and-flag pins (polish phase 2)#55

Merged
Jammy2211 merged 10 commits into
mainfrom
feature/profiling-vram-validation
Jul 8, 2026
Merged

maint: vram-first validation — probe override, record-and-flag pins (polish phase 2)#55
Jammy2211 merged 10 commits into
mainfrom
feature/profiling-vram-validation

Conversation

@Jammy2211

Copy link
Copy Markdown
Contributor

Summary

Phase 2 of the polish series (closes the local leg of #54): validate the VRAM/vmap story before the PreOptimizationTimes campaign burns A100 hours. All 9 campaign cells now run green end-to-end on CPU (durable record: results/notes/vram_validation_2026_07_08.md); the sweep flushed three real defects, all fixed here. The A100 probe leg is staged (submits + ingest workflow) for when RAL is back.

Scripts Changed

  • vram/config.pyresolve_vmap_batch(): runtime cells prefer a fresh cell-matching vmap_probe_<model>[_sparse].json over the static table (a stale table can no longer OOM a job whose submit re-probed); VMAP_BATCH_SPARSE (w-tilde rows, unprobed cells fall back to the dense value); PROVENANCE (dense table dates to 2026-05-24 — recorded explicitly); non-GPU backends clamp resolved batches to 3 (the A100 table OOM-killed laptop-CPU runs at batch 64)
  • vram/probe.py — probe JSONs record the backend that produced them; resolution ignores probes from a different backend (CPU memory analysis says nothing about A100)
  • _profile_cli.pycheck_pinned() / record_pinned_check(): pinned likelihood/evidence drift warns and records (pinned_expected + pinned_drift in the result JSON — the surface PyAutoHeart's filed vitals leg scans) instead of crashing a multi-hour job; boundary rule in results/notes/design_lock_in.md (profiling records and flags; autolens_workspace_test adjudicates library correctness)
  • likelihood_runtime/<class>/<model>.py (all 9 cells) — wired to resolve_vmap_batch with the winning source logged; pins converted to record-and-flag (eager ≡ JIT ≡ vmap cross-consistency asserts stay hard); imaging/{mge,pixelization,delaunay} pins made per-instrument (the old module-level HST constants failed any non-HST run by construction); probe basenames normalised to vmap_probe_<model>[_sparse].json
  • hpc/batch_gpu/submit_probe_{fast,delaunay}_a100 — probe-only sweeps re-validating every campaign vmap cell (dense + sparse imaging) against current source; ingest workflow in vram/README.md
  • test/test_vram_config.py — 17 new tests (sparse fallback, probe override, mismatch/corrupt/backend handling); 25/25 total
  • results/notes/vram_validation_2026_07_08.md — sweep verdict, timings, defects; dataset/imaging/euclid/ + simulators dashboard row (auto-simulated during the sweep)

Test Plan

  • 25/25 unit tests; ruff check + format --check clean; build_readme.py --check idempotent
  • Local CPU validation sweep: all 9 campaign cells green (incl. sparse variants); probe pipeline + probe-over-table resolution verified live; drift fields verified in real artifacts

Validation checklist (--auto run — plan approved in-session)

  • Effective level: supervised (header: supervised, cap: maintenance → safe)
  • Plan: on the issue (maint: vram-first validation sweep + batch-size provenance (polish phase 2) #54), approved interactively at start
  • Gate: tests 25/25 (downstream n/a — standalone leaf) · smoke repo-gate pass + full live sweep (six-workspace smoke n/a — no downstream script surface) · review CLEAN (every changed path exercised by the sweep) · Heart YELLOW, human-acknowledged at ship sign-off (issue comment): reason set organism-wide, none touch this repo
  • Human: plan sound in hindsight?
  • Human: diff matches plan (no scope creep)?
  • Human: merge, amend, or reject — then log the outcome

🤖 Generated with Claude Code

Jammy2211 and others added 3 commits July 8, 2026 16:19
…ce (#54)

- vram/config.py: VMAP_BATCH_SPARSE table (w-tilde rows; unprobed cells
  fall back dense), PROVENANCE record (dense table probed 2026-05-24 —
  stale vs current source, re-probe submits staged), and
  resolve_vmap_batch(): runtime cells now prefer a fresh cell-matching
  vmap_probe_<model>[_sparse].json over the static table, so a stale
  table cannot OOM a job whose submit re-probed in Phase A
- all 8 vmap-capable runtime cells wired to resolve_vmap_batch with the
  winning source logged; probe JSON basenames normalised to
  vmap_probe_<model>[_sparse].json across interferometer/point_source/
  datacube (imaging cells already followed it)
- hpc/batch_gpu/submit_probe_{fast,delaunay}_a100: probe-only sweeps
  re-validating every campaign vmap cell (dense + sparse imaging) against
  current source; ingest workflow documented in vram/README.md, which
  also loses its stale z_projects path
- test/test_vram_config.py: 11 new tests (sparse fallback, probe
  override, mismatch/corrupt-JSON handling)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Design decision (user-directed): autolens_profiling's pinned values guard
benchmark comparability, not library correctness — that remit belongs to
autolens_workspace_test. A hard assert conflated "computation changed"
with "profiling run failed" and could kill a multi-hour A100 job at the
final assertion.

- _profile_cli.check_pinned(): compare against the pin, warn loudly on
  drift, return a drift record; record_pinned_check() merges
  pinned_expected + pinned_drift into the already-written result JSON —
  the machine surface PyAutoHeart's vitals scan will read
  (feature/pyautoheart/profiling_drift_check.md filed)
- all 9 runtime cells converted; eager ≡ JIT ≡ vmap cross-consistency
  assertions stay hard (they invalidate the measurement itself)
- imaging mge/pixelization/delaunay pins made per-instrument first (the
  old module-level HST constants were asserted for every --instrument,
  failing any non-hst run by construction — flushed by this sweep)
- vram: non-GPU backends clamp resolved batches to 3 (A100 table applied
  on laptop CPU OOM-killed the sweep); probes record their backend and
  resolution ignores probes from a different backend; +12 tests (25/25)
- boundary rule recorded in results/notes/design_lock_in.md

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All 9 campaign cells green on CPU under the fixed code (results/notes/
vram_validation_2026_07_08.md is the durable record — sweep artifacts were
scratch-tier). Simulators dashboard row for the auto-simulated euclid
dataset (v2026.7.6.649).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Jammy2211 and others added 7 commits July 8, 2026 17:44
Two lint-gate failures, one root cause each:

- PR gate: build_readme --check diverged because results/simulators/ was
  still gitignored (a pre-phase-1 "until we decide" rule) — the local
  dashboard baked a euclid row from an artifact CI never received. The
  phase-1 design settled that results live in-repo: un-ignore
  results/simulators/ + results/searches/ (results/likelihood/ stays
  ignored as the retired legacy section) and commit the euclid artifact
  pair.
- main gate (lychee): simulators/README.md still linked the retired
  likelihood/README.md; retargeted to likelihood_runtime/breakdown.
  Repo-wide relative-link scan: no other dead links.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…me/breakdown split; exposed once lychee passed)
…picked pip list lacked pyyaml etc; adopt the lib-tests.yml pattern
@Jammy2211 Jammy2211 merged commit 32fc5cd into main Jul 8, 2026
1 check passed
@Jammy2211 Jammy2211 deleted the feature/profiling-vram-validation branch July 8, 2026 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pending-release Queued for the next release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant