From 3a6ca92227bc64315769a9abc43b9f534a4557bb Mon Sep 17 00:00:00 2001 From: Jammy2211 Date: Wed, 8 Jul 2026 15:32:44 +0100 Subject: [PATCH 1/3] maint: design lock-in + results/dashboard groundwork (#52) - results live wholly in-repo: sweep.py/aggregate.py default --output-root is now results/runtime/ (workspace_developer tree stays as history) - sweep.py: sys.executable default (no machine-specific path), imaging/mge joins CELLS, --sparse passthrough for the w-tilde rows - build_readme.py retargeted from the retired likelihood/ layout to the real results sections (breakdown/runtime/simulators/searches), with comparison.json support and a named-baseline column hook - dashboard sentinels added to README + runtime/breakdown/searches READMEs - PreOptimizationTimes baseline convention defined (results/notes/design_lock_in.md); results/README.md rewritten - stale docs fixed (retired likelihood/ refs, z_projects HPC refs, PyAutoPrompt roadmap ref); quick_update/hpc/scripts READMEs added Co-Authored-By: Claude Fable 5 --- AGENTS.md | 13 +- README.md | 57 +++-- _profile_cli.py | 8 +- hpc/README.md | 33 +++ likelihood_breakdown/README.md | 14 ++ likelihood_runtime/README.md | 10 +- likelihood_runtime/aggregate.py | 19 +- likelihood_runtime/sweep.py | 42 +++- quick_update/README.md | 22 ++ results/README.md | 48 +++- results/notes/design_lock_in.md | 93 +++++++ scripts/README.md | 8 + scripts/build_readme.py | 426 ++++++++++++++++++-------------- searches/README.md | 8 + 14 files changed, 551 insertions(+), 250 deletions(-) create mode 100644 hpc/README.md create mode 100644 quick_update/README.md create mode 100644 results/notes/design_lock_in.md create mode 100644 scripts/README.md diff --git a/AGENTS.md b/AGENTS.md index bee232c..743afbd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -10,18 +10,19 @@ overview (vision, latest run-times, roadmap); this file is the operational guide ## Repository Structure ``` -likelihood/ Per-instrument likelihood profile scripts (imaging, interferometer, - datacube, point_source) — import the shared _profile_cli helper likelihood_runtime/ Full-pipeline JIT runtime, driven by sweep.py across CPU/GPU/A100 × fp64/mp + (per-cell scripts under /.py import the shared _profile_cli helper) likelihood_breakdown/ Per-step JIT decomposition of a single likelihood config -vram/ GPU memory-usage profiling +vram/ GPU memory-usage profiling + the per-cell A100 vmap batch-size table instruments/ Instrument definitions (pixel scale, shape) used to frame results searches/ Sampler / search profiling (Nautilus first) simulators/ Run-time tracking for the PyAutoLens simulators latent/ Latent-variable profiling -quick_update/ Fast incremental re-profiling helpers -hpc/ SLURM submit scripts for the RAL HPC -results/ Versioned JSON + PNG artifacts (`*_v....{json,png}`) +quick_update/ Fast incremental re-profiling helpers (unversioned scratch tier) +hpc/ SLURM submit scripts for the RAL HPC (A100 rows of the sweep matrix) +scripts/ Repo tooling (build_readme.py dashboard generator) +results/ JSON + PNG artifacts: versioned summaries, sweep comparisons, named + baselines (results/README.md defines the shapes; sweeps default here) config/ dataset/ output/ Config, input data, runtime output ``` diff --git a/README.md b/README.md index be3cc05..7997cb7 100644 --- a/README.md +++ b/README.md @@ -24,9 +24,28 @@ Results are framed by **astronomy instrument** (HST, Euclid, JWST, …) rather t ## Latest run-times -Cell-level full-pipeline numbers live in [`likelihood_runtime/OPTIMIZATION_NOTES.md`](./likelihood_runtime/OPTIMIZATION_NOTES.md), which carries the latest CPU + local-GPU per-call costs together with per-cell "where to optimize next" recommendations and the mp-vs-fp64 verdicts. The detailed multi-config `comparison.json` artifacts are committed under [`autolens_workspace_developer/jax_profiling/results/jit///`](https://github.com/PyAutoLabs/autolens_workspace_developer/tree/main/jax_profiling/results/jit) and are re-aggregated by `likelihood_runtime/aggregate.py` whenever a new sweep finishes. + -(The previous auto-generated table in this README was retired when the likelihood profiling was split into `likelihood_breakdown/` + `likelihood_runtime/`; `scripts/build_readme.py` is queued for a path-update follow-up.) +**Likelihood runtime** — full-pipeline per-call cost per cell × config: + +_No data yet — run `likelihood_runtime/sweep.py` then `aggregate.py` to populate._ + +**Likelihood breakdown** — latest per-step decompositions: + +| Cell | Instrument | Inversion path | Step-sum total | PyAutoLens version | +|------|------------|----------------|----------------|--------------------| +| `imaging/delaunay` | hst | dense (mapping) | 11.29 s | v2026.5.29.4 | +| `imaging/delaunay` | hst | sparse (w-tilde) | 7.96 s | v2026.5.29.4 | +| `imaging/mge` | hst | dense (mapping) | 178.2 ms | v2026.5.29.4 | +| `imaging/pixelization` | hst | dense (mapping) | 7.79 s | v2026.5.29.4 | +| `imaging/pixelization` | hst | sparse (w-tilde) | 8.94 s | v2026.5.29.4 | + + +The tables above are auto-generated by `scripts/build_readme.py` from the artifacts under [`results/`](./results/README.md) — never edit them by hand; run `python scripts/build_readme.py` after a profiling run and commit the result (CI checks idempotence via `--check`). Narrative context — per-cell "where to optimize next" recommendations and the mp-vs-fp64 verdicts — lives in [`likelihood_runtime/OPTIMIZATION_NOTES.md`](./likelihood_runtime/OPTIMIZATION_NOTES.md). + +**PreOptimizationTimes** is the named baseline the upcoming optimization work is measured against: a frozen snapshot of the full campaign (laptop CPU, HPC CPU, HPC A100 × fp64/mp) under `results/baselines/PreOptimizationTimes/`, rendered as a baseline column in the dashboard once populated. The convention is defined in [`results/notes/design_lock_in.md`](./results/notes/design_lock_in.md). + +(Historical multi-config sweeps up to 2026-07 were committed under [`autolens_workspace_developer/jax_profiling/results/jit/`](https://github.com/PyAutoLabs/autolens_workspace_developer/tree/main/jax_profiling/results/jit); sweeps now write in-repo to `results/runtime/` by default.) ## JAX gradients — currently out of scope @@ -34,34 +53,36 @@ Gradient profiling (`jax.grad` of the likelihood, autodiff-based optimisers) is ## How to read this repo -Each profiling script writes a **versioned artifact pair** under `results/`: +Profiling scripts write two artifact shapes under `results/` (full reference: [`results/README.md`](./results/README.md)): ``` -results/
//_summary_v....json -results/
//_summary_v....png -``` - -The version string matches the PyAutoLens release that produced the numbers (e.g. `v2026.5.1.4`). Older versions are retained alongside newer ones, so trends across releases stay inspectable. The JSON carries structured timings; the PNG is the at-a-glance plot. +# Versioned summaries — standalone runs; history retained side-by-side +results/
//___v...[_sparse].{json,png} -Examples that already exist in the source-of-truth repo: +# Per-config sweeps — sweep.py + aggregate.py; latest sweep per cell +results/runtime//[/]/[_sparse].{json,png,log} + comparison.{json,png} +``` -- `imaging_summary_v2026.5.1.4.json` / `.png` -- `point_source_summary_v2026.5.1.4.json` -- `delaunay_sparse_cpu_likelihood_summary_hst_v2026.5.1.4.json` +The version string matches the PyAutoLens release that produced the numbers (e.g. `v2026.5.29.4`). The JSON carries structured timings; the PNG is the at-a-glance plot. Cross-release **trend** questions read the versioned summaries; cross-hardware **comparison** questions read `comparison.json`. ## Section index | Folder | Contents | |--------|----------| -| [`likelihood_breakdown/`](./likelihood_breakdown/README.md) | Per-step JIT decomposition. Single config. *Where does time go inside the likelihood?* | | [`likelihood_runtime/`](./likelihood_runtime/README.md) | Full-pipeline JIT only, driven by `sweep.py` across CPU/GPU/A100 × fp64/mp. *How long will this likelihood take on this hardware?* | +| [`likelihood_breakdown/`](./likelihood_breakdown/README.md) | Per-step JIT decomposition. Single config. *Where does time go inside the likelihood?* | +| [`vram/`](./vram/README.md) | GPU memory profiling + the per-cell vmap batch-size table for the A100. | +| [`latent/`](./latent/README.md) | Latent-variable profiling. | +| [`instruments/`](./instruments/README.md) | Instrument presets (pixel scale, shape) that frame every result. | | [`simulators/`](./simulators/README.md) | Run-time tracking for the PyAutoLens simulators. | | [`searches/`](./searches/README.md) | Sampler / search profiling, Nautilus first. | -| [`results/`](./results/README.md) | Versioned JSON + PNG artifacts written by the above scripts. | +| [`quick_update/`](./quick_update/README.md) | Fast incremental re-profiling helpers. | +| [`hpc/`](./hpc/README.md) | SLURM submit scripts for the RAL HPC (A100 rows of the sweep matrix). | +| [`results/`](./results/README.md) | JSON + PNG artifacts written by the above scripts; named baselines. | ## Roadmap -This repo is being built in phases. Phase numbers correspond to internal sub-prompts under `PyAutoLabs/PyAutoPrompt/z_features/autolens_profiling.md`. +This repo is being built in phases (bootstrap history now archived in `PyAutoMind`). | Phase | Title | Status | |-------|-------|--------| @@ -70,7 +91,9 @@ This repo is being built in phases. Phase numbers correspond to internal sub-pro | 2 | Mirror simulator profiling scripts + run-time tracking | ✓ shipped | | 3 | Nautilus profiling, design for sampler expansion | ✓ shipped | | 4 | Top-level + per-section README dashboard with instrument framing | ✓ shipped | -| 5 | GitHub Actions for lint + profile re-runs + README refresh | queued | +| 5 | GitHub Actions for lint + profile re-runs + README refresh | ✓ shipped (`lint.yml` per-PR; `profile.yml` manual/on-release) | +| 6 | Design lock-in + results/dashboard groundwork ([#52](https://github.com/PyAutoLabs/autolens_profiling/issues/52)) | in progress | +| 7 | **PreOptimizationTimes** baseline campaign (vram-first, then runtime + breakdown) | queued | ### Future enhancements (Phase 4 follow-ups) @@ -87,7 +110,7 @@ Dashboards can grow in many directions. The list below captures candidate improv - [`PyAutoLabs/PyAutoLens`](https://github.com/PyAutoLabs/PyAutoLens) — the library being profiled. - [`PyAutoLabs/autolens_workspace`](https://github.com/PyAutoLabs/autolens_workspace) — user-facing science scripts and tutorials. -- [`PyAutoLabs/autolens_workspace_developer`](https://github.com/PyAutoLabs/autolens_workspace_developer) — the developer workspace; **source of truth during the migration**. Each phase mirrors the relevant subdirectories from here into this repo. +- [`PyAutoLabs/autolens_workspace_developer`](https://github.com/PyAutoLabs/autolens_workspace_developer) — the developer workspace this repo's scripts were migrated from; still hosts the pre-2026-07 sweep history and the gradient-profiling work. - [`Jammy2211/autolens_colab_profiling`](https://github.com/Jammy2211/autolens_colab_profiling) — sibling repo, Colab-specific scope. Not yet migrated to PyAutoLabs. ## Package vs scripts diff --git a/_profile_cli.py b/_profile_cli.py index 4216b05..2987f42 100644 --- a/_profile_cli.py +++ b/_profile_cli.py @@ -1,7 +1,7 @@ """Shared CLI / JSON / auto-simulate helpers for the likelihood scripts. -Used by every script under ``likelihood/{imaging,interferometer, -datacube,point_source}/`` so the per-script boilerplate stays minimal +Used by every per-cell script under ``likelihood_runtime/`` and +``likelihood_breakdown/`` so the per-script boilerplate stays minimal and the sweep-driver flags (``--config-name``, ``--output-dir``, ``--use-mixed-precision``) and dataset auto-simulate hook are defined in one place. @@ -69,8 +69,8 @@ def parse_profile_cli(default_config_name: Optional[str] = None) -> ProfileCLI: "--output-dir", default=None, help=( - "Override results dir. Defaults to " - "/results/likelihood//." + "Override results dir. Each per-cell script defaults to its " + "package's section under /results/." ), ) parser.add_argument( diff --git a/hpc/README.md b/hpc/README.md new file mode 100644 index 0000000..1ac1b3e --- /dev/null +++ b/hpc/README.md @@ -0,0 +1,33 @@ +# hpc + +SLURM submit scripts for the RAL HPC — the `hpc_a100_fp64` / `hpc_a100_mp` +rows of the sweep matrix (and HPC-CPU rows) that the local +`likelihood_runtime/sweep.py` driver cannot run itself. + +## Layout + +``` +batch_gpu/ + submit____a100__[_sparse] # one submit per cell/config + output/ error/ # SLURM stdout/stderr (gitignored) +``` + +Submit names follow the same `/` cell grid as the rest of the +repo; `runtime_` prefixed submits drive `likelihood_runtime/` cells, `nss_` +prefixed ones drive `searches/` cells. + +## Running + +On the HPC login node: + +```bash +source activate.sh # repo-root helper: venv + PYTHONPATH at the canonical checkouts +sbatch hpc/batch_gpu/submit_runtime_imaging_mge_a100_hst_fp64 +``` + +Each job writes its per-config JSON into the same `results/` layout as a local +sweep (`--config-name hpc_a100_fp64` etc.), so `likelihood_runtime/aggregate.py` +merges local and A100 rows into one `comparison.json`. Copy/commit the result +JSONs from the HPC checkout back via the normal git flow. The PyAuto* +libraries resolve from sibling source checkouts on `PYTHONPATH` — never +pip-install them into the venv (`HPCPullPyAuto` is the update story). diff --git a/likelihood_breakdown/README.md b/likelihood_breakdown/README.md index 13a15cf..39e0922 100644 --- a/likelihood_breakdown/README.md +++ b/likelihood_breakdown/README.md @@ -8,6 +8,20 @@ Run one of these scripts when you want to find the **next source-code-level opti For *how long the likelihood actually takes* on production hardware — i.e. a single end-to-end number per (hardware, precision) config — use the sibling package [`likelihood_runtime/`](../likelihood_runtime/) instead. The two packages are deliberately disjoint so neither has to pay the other's cost. +## Latest results + + +| Cell | Instrument | Inversion path | Step-sum total | PyAutoLens version | +|------|------------|----------------|----------------|--------------------| +| `imaging/delaunay` | hst | dense (mapping) | 11.29 s | v2026.5.29.4 | +| `imaging/delaunay` | hst | sparse (w-tilde) | 7.96 s | v2026.5.29.4 | +| `imaging/mge` | hst | dense (mapping) | 178.2 ms | v2026.5.29.4 | +| `imaging/pixelization` | hst | dense (mapping) | 7.79 s | v2026.5.29.4 | +| `imaging/pixelization` | hst | sparse (w-tilde) | 8.94 s | v2026.5.29.4 | + + +Auto-generated by `scripts/build_readme.py` from the versioned artifacts under `results/breakdown/`. + ## Methodology For each pipeline step (e.g. *ray-trace grids* → *blurred mapping matrix* → *curvature matrix F* → *NNLS reconstruction* → *log-evidence*), the script: diff --git a/likelihood_runtime/README.md b/likelihood_runtime/README.md index fc2456f..92e1c7b 100644 --- a/likelihood_runtime/README.md +++ b/likelihood_runtime/README.md @@ -10,6 +10,14 @@ For *where the time goes inside the likelihood*, use the sibling package [`likel The empirical findings from previous sweeps — per-cell timings, mp verdicts, the GPU-NUFFT regression, the upstream blockers — live in [`OPTIMIZATION_NOTES.md`](OPTIMIZATION_NOTES.md) in this directory. +## Latest results + + +_No data yet — run `likelihood_runtime/sweep.py` then `aggregate.py` to populate._ + + +Auto-generated by `scripts/build_readme.py` from `results/runtime/**/comparison.json`; baseline columns appear once `results/baselines/` is populated. + ## Methodology Each script measures **one** quantity per run: the steady-state cost of the entire likelihood as a single JIT-compiled JAX program. There is no per-step decomposition. The measurement is: @@ -30,7 +38,7 @@ The sweep harness drives every in-scope cell through this matrix: | `local_cpu_mp` | CPU | mixed (fp32 inversion) | same + `--use-mixed-precision` | | `local_gpu_fp64` | RTX 2060 (consumer) | fp64 | `JAX_PLATFORM_NAME=cuda JAX_PLATFORMS=cuda,cpu` | | `local_gpu_mp` | RTX 2060 | mixed | same + `--use-mixed-precision` | -| `hpc_a100_fp64` | A100 (80 GB) | fp64 | SLURM-dispatched via `z_projects/profiling/hpc/sync` | +| `hpc_a100_fp64` | A100 (80 GB) | fp64 | SLURM-dispatched via the submit scripts under [`hpc/`](../hpc/README.md) | | `hpc_a100_mp` | A100 | mixed | same + `--use-mixed-precision` | The `cuda,cpu` listing on GPU configs is load-bearing: the Delaunay + datacube paths use `jax.pure_callback` for Hilbert-curve mesh generation, which needs a CPU device available even when the primary platform is CUDA. Without the trailing `cpu` the callback raises `pure_callback failed to find a local CPU device`. diff --git a/likelihood_runtime/aggregate.py b/likelihood_runtime/aggregate.py index f71a837..4fde4e1 100644 --- a/likelihood_runtime/aggregate.py +++ b/likelihood_runtime/aggregate.py @@ -1,11 +1,11 @@ """Aggregate per-config JSONs for a swept likelihood cell into comparison.{json,png}. -Reads every ``.json`` under a cell's output dir (see -``sweep.py``) and produces a single ``comparison.json`` whose schema -mirrors the existing -``autolens_workspace_developer/jax_profiling/results/jit/imaging/{mge, -pixelization,delaunay}/comparison.json`` artifacts so the existing -readers (and the OPTIMIZATION_NOTES doc) continue to work. +Reads every ``[_sparse].json`` under a cell's output dir (see +``sweep.py``; default root is ``results/runtime/`` in this repo) and +produces a single ``comparison.json`` whose schema mirrors the historical +``autolens_workspace_developer/jax_profiling/results/jit`` artifacts so the +existing readers (and the OPTIMIZATION_NOTES doc) continue to work. +``_sparse`` rows order after the canonical configs. The ``comparison.png`` is a log-scale grouped bar chart: one bar per (step, config), sorted by step cost on the slowest config. The @@ -36,8 +36,7 @@ _REPO_ROOT = Path(__file__).resolve().parents[1] -_WT_ROOT = _REPO_ROOT.parent -_DEFAULT_OUTPUT_ROOT = _WT_ROOT / "autolens_workspace_developer" / "jax_profiling" / "results" / "jit" +_DEFAULT_OUTPUT_ROOT = _REPO_ROOT / "results" / "runtime" # Stable ordering — keep the same row order as sweep_likelihood + the prior @@ -86,7 +85,9 @@ def _discover_cells(output_root: Path) -> list[tuple[str, ...]]: def _has_config_json(d: Path) -> bool: return any( - p.stem in _CONFIG_ORDER or p.stem.endswith("_pre_fix") + p.stem in _CONFIG_ORDER + or p.stem.removesuffix("_sparse") in _CONFIG_ORDER + or p.stem.endswith("_pre_fix") for p in d.glob("*.json") ) diff --git a/likelihood_runtime/sweep.py b/likelihood_runtime/sweep.py index 1d23edc..2a56496 100644 --- a/likelihood_runtime/sweep.py +++ b/likelihood_runtime/sweep.py @@ -1,8 +1,8 @@ """Multi-config likelihood profiling driver. Runs each in-scope cell across the CPU/GPU x fp64/mp matrix (4 configs per -cell locally; HPC A100 configs are dispatched separately via -`z_projects/profiling/hpc/sync`). +cell locally; HPC A100 configs are dispatched separately via the SLURM +submit scripts under ``hpc/``). Each subprocess invokes the existing per-cell likelihood script under ``autolens_profiling/likelihood_runtime//.py`` with the @@ -13,10 +13,11 @@ ///.png ///.log (captured stdout/stderr) -Default ``--output-root`` is -``autolens_workspace_developer/jax_profiling/results/jit`` — matches the -existing imaging precedent and is read by ``aggregate.py`` to produce -``comparison.json`` / ``comparison.png``. +Default ``--output-root`` is ``results/runtime/`` in this repo, read by +``aggregate.py`` to produce ``comparison.json`` / ``comparison.png``. +(Earlier sweeps wrote to ``autolens_workspace_developer/jax_profiling/ +results/jit`` — that tree remains readable history but is no longer the +default; see ``results/notes/design_lock_in.md``.) Usage:: @@ -28,6 +29,9 @@ # Single cell, single backend python likelihood_runtime/sweep.py --only interferometer/mge --skip-cpu + + # Imaging sparse-operator (w-tilde) rows — filenames gain a _sparse suffix + python likelihood_runtime/sweep.py --only imaging/pixelization imaging/delaunay --sparse """ from __future__ import annotations @@ -42,14 +46,14 @@ _REPO_ROOT = Path(__file__).resolve().parents[1] # autolens_profiling/ -_WT_ROOT = _REPO_ROOT.parent # PyAutoLabs-wt// (or PyAutoLabs/) -_DEFAULT_OUTPUT_ROOT = _WT_ROOT / "autolens_workspace_developer" / "jax_profiling" / "results" / "jit" -_DEFAULT_PYTHON = "/home/jammy/venv/PyAutoGPU/bin/python" +_DEFAULT_OUTPUT_ROOT = _REPO_ROOT / "results" / "runtime" +_DEFAULT_PYTHON = sys.executable # (dataset_class, model). Order is roughly cheapest -> heaviest so failures # surface quickly during iteration. CELLS: list[tuple[str, str]] = [ + ("imaging", "mge"), ("imaging", "pixelization"), ("imaging", "delaunay"), ("interferometer", "mge"), @@ -121,6 +125,16 @@ def _parse_args() -> argparse.Namespace: action="store_true", help="Skip the use_mixed_precision rows (just fp64).", ) + p.add_argument( + "--sparse", + action="store_true", + help=( + "Pass --sparse to every selected cell (w-tilde sparse-operator " + "inversion path; imaging cells only — combine with --only). " + "Result filenames gain a _sparse suffix so dense and sparse " + "rows coexist." + ), + ) p.add_argument( "--output-root", type=Path, @@ -170,10 +184,12 @@ def _run_one( config: SweepConfig, out_dir: Path, dry_run: bool, + sparse: bool = False, ) -> tuple[bool, float, str]: """Run one (cell, config) pair as a subprocess. Returns (ok, elapsed, log_path).""" out_dir.mkdir(parents=True, exist_ok=True) - log_path = out_dir / f"{config.name}.log" + log_suffix = "_sparse" if sparse else "" + log_path = out_dir / f"{config.name}{log_suffix}.log" cmd = [ python, @@ -181,6 +197,7 @@ def _run_one( "--config-name", config.name, "--output-dir", str(out_dir), *config.extra_args, + *(("--sparse",) if sparse else ()), ] env = dict(os.environ) @@ -213,7 +230,7 @@ def _run_one( # Verify the device.backend in the JSON matches expectations. if ok: import json - json_path = out_dir / f"{config.name}.json" + json_path = out_dir / f"{config.name}{log_suffix}.json" if json_path.exists(): try: data = json.loads(json_path.read_text()) @@ -263,7 +280,8 @@ def main() -> int: for cfg in configs: try: ok, elapsed, _log = _run_one( - args.python, script_path, cfg, out_dir, args.dry_run + args.python, script_path, cfg, out_dir, args.dry_run, + sparse=args.sparse, ) except KeyboardInterrupt: print("\n\nsweep interrupted by user") diff --git a/quick_update/README.md b/quick_update/README.md new file mode 100644 index 0000000..f5c2758 --- /dev/null +++ b/quick_update/README.md @@ -0,0 +1,22 @@ +# quick_update + +Fast incremental re-profiling helpers: one thin script per dataset class that +re-runs the headline likelihood measurement in seconds, for tight +edit-profile-edit loops while iterating on the source libraries. + +These are the **scratch tier** of the repo: + +- Outputs land in `results/quick_update/` as **unversioned** JSONs + (`_quick_update_.json`) that are overwritten on every run — + no history, no dashboard row. +- Numbers here are for steering a source-code change, never for citing: when a + result matters, re-run the real cell under + [`likelihood_runtime/`](../likelihood_runtime/README.md) or + [`likelihood_breakdown/`](../likelihood_breakdown/README.md). + +## Running + +```bash +python quick_update/imaging.py +python quick_update/interferometer_delaunay.py +``` diff --git a/results/README.md b/results/README.md index f244ee6..ef3a0e6 100644 --- a/results/README.md +++ b/results/README.md @@ -1,22 +1,50 @@ # results -Versioned profiling artifacts written by scripts under [`likelihood/`](../likelihood/README.md), [`simulators/`](../simulators/README.md), and [`searches/`](../searches/README.md). Layout mirrors the source folders. +Profiling artifacts written by the packages above. Layout mirrors the source +packages; the dashboard tables in every README are rendered from this tree by +`scripts/build_readme.py`. -## Filename convention +## Sections + +| Folder | Written by | Contents | +|--------|-----------|----------| +| `runtime/` | [`likelihood_runtime/`](../likelihood_runtime/README.md) sweeps | Per-config sweep outputs + `comparison.{json,png}` per cell; A100 logs/probes | +| `breakdown/` | [`likelihood_breakdown/`](../likelihood_breakdown/README.md) | Versioned per-step decompositions | +| `simulators/` | [`simulators/`](../simulators/README.md) | Versioned simulator run-time summaries | +| `searches/` | [`searches/`](../searches/README.md) | Versioned sampler profiling summaries | +| `quick_update/` | [`quick_update/`](../quick_update/README.md) | Unversioned fast re-profiling snapshots (scratch tier) | +| `notes/` | humans + agents | Narrative findings and design notes (e.g. [`design_lock_in.md`](./notes/design_lock_in.md)) | +| `baselines/` | campaign snapshots | Named, frozen baselines (e.g. `PreOptimizationTimes/`) — see below | + +## The two artifact shapes + +**Versioned summaries** — written by per-cell scripts run standalone; history +is retained side-by-side so cross-release trends stay inspectable: ``` -_summary_v....json -_summary_v....png +___v...[_sparse].json # purpose = summary | breakdown +___v...[_sparse].png ``` -The version string is the PyAutoLens release that produced the numbers (e.g. `v2026.5.1.4`). Older versions are kept alongside newer ones so cross-release trends stay inspectable. +The version string is the PyAutoLens release that produced the numbers +(e.g. `v2026.5.29.4`). -Example: +**Per-config sweeps** — written by `likelihood_runtime/sweep.py` and +aggregated by `aggregate.py`; each cell dir holds the *latest* sweep: ``` -results/likelihood/imaging/imaging_summary_v2026.5.1.4.json -results/likelihood/imaging/imaging_summary_v2026.5.1.4.png -results/likelihood/imaging/mge/delaunay_sparse_cpu_likelihood_summary_hst_v2026.5.1.4.json +runtime//[/]/[_sparse].{json,png,log} +runtime//[/]/comparison.{json,png} ``` -Populated as Phases 1–3 land. See the top-level [README](../README.md) for the full phase plan. +Config names: `local_cpu_fp64 | local_cpu_mp | local_gpu_fp64 | local_gpu_mp | +hpc_a100_fp64 | hpc_a100_mp`, with `_sparse` as a filename suffix. + +## Named baselines + +A **baseline** is a frozen snapshot of a full campaign under +`baselines//`, mirroring the `runtime/` layout plus a rendered +`.md` with every headline number on one page. Baselines are +append-only — never edited after the campaign closes. The first baseline is +**`PreOptimizationTimes`** (the pre-optimization reference). Full convention: +[`notes/design_lock_in.md`](./notes/design_lock_in.md). diff --git a/results/notes/design_lock_in.md b/results/notes/design_lock_in.md new file mode 100644 index 0000000..bbc0be2 --- /dev/null +++ b/results/notes/design_lock_in.md @@ -0,0 +1,93 @@ +# Design lock-in — pre-PreOptimizationTimes review (2026-07-08) + +One final holistic review of the repo before the **PreOptimizationTimes** +baseline campaign — the last full profiling sweep before the optimization +push. This note records what is locked in, what changed, and why, so future +extensions (more datasets, instruments, packages) build on a settled core. +Tracked by [autolens_profiling#52](https://github.com/PyAutoLabs/autolens_profiling/issues/52); +parent intent: PyAutoMind `maintenance/autolens_profiling/polish.md`. + +## What is locked in (unchanged — the design is right) + +- **The package split.** `likelihood_runtime/` (how long per call?) vs + `likelihood_breakdown/` (where does the time go?) vs `vram/` (does it fit, + and at what vmap batch size?) are deliberately disjoint questions with + deliberately disjoint packages. `latent/`, `searches/`, `simulators/`, + `quick_update/` follow the same one-question-per-package rule. New profiling + concerns get a **new package**, not a flag on an existing one. +- **The cell grid.** Work is addressed as `/` cells + (e.g. `imaging/mge`, `datacube/delaunay`), optionally deepened by + `/`. Sweep drivers, aggregators, output dirs and HPC submit + scripts all speak this grid. New datasets/models slot in as new cells. +- **`_profile_cli.py` as the single CLI surface.** Every per-cell script takes + `--config-name / --output-dir / --use-mixed-precision / --instrument / + --vmap-probe / --sparse` through the shared helper. New flags go here, once, + never per-script. +- **Instrument framing** via `instruments/` presets; headline numbers are + named for observing programmes (HST, ALMA, JWST…), not pixel counts. +- **Config axis names**: `local_cpu_fp64 | local_cpu_mp | local_gpu_fp64 | + local_gpu_mp | hpc_a100_fp64 | hpc_a100_mp` (the 6-config matrix), with + `_sparse` as a filename suffix, not a seventh config. +- **Correctness gates inside every timing script** (eager ≡ JIT ≡ vmap at + documented rtol) stay mandatory for new cells. + +## What changed in this review + +1. **Results live wholly in this repo.** `sweep.py` / `aggregate.py` + previously defaulted `--output-root` to + `../autolens_workspace_developer/jax_profiling/results/jit` — a migration + leftover. The default is now **`results/runtime/`** in-repo. The + workspace_developer tree remains readable history; nothing new is written + there. (`--output-root` still overrides for scratch runs.) +2. **No machine-specific defaults.** `sweep.py`'s hard-coded + `/home/jammy/venv/PyAutoGPU/bin/python` default became `sys.executable` + (violated the repo's own "no machine-specific absolute paths" rule). +3. **`imaging/mge` joined the sweep `CELLS` grid** so the runtime campaign + covers the same imaging cells as the breakdown package; `sweep.py` also + gained `--sparse` passthrough so the imaging sparse-vs-mapping comparison + runs through the same driver as everything else. +4. **The README dashboard was revived and retargeted.** + `scripts/build_readme.py` still pointed at the retired `likelihood/` + package (deleted in the runtime/breakdown split) and scanned 0 artifacts. + It now scans the real `results/` sections (`breakdown`, `runtime`, + `simulators`, `searches`), understands both artifact shapes (see below), + and renders auto-tables into the top-level README + per-package READMEs. + CI's `--check` idempotence gate is unchanged. +5. **Stale docs fixed**: retired-`likelihood/` references in `AGENTS.md`, + `README.md`, `results/README.md` and `_profile_cli.py`; the + `z_projects/profiling/hpc/sync` references (HPC dispatch lives in-repo + under `hpc/`); the roadmap's retired `PyAutoPrompt` registry pointer. + `quick_update/`, `hpc/` and `scripts/` gained READMEs. + +## The two artifact shapes (both canonical) + +| Shape | Written by | Pattern | Versioning | +|-------|-----------|---------|------------| +| **Versioned summary** | per-cell scripts run standalone | `___v[_sparse].{json,png}` | in the filename; history retained side-by-side | +| **Per-config sweep** | `sweep.py` → `aggregate.py` | `/[/]/[_sparse].{json,png,log}` + `comparison.{json,png}` | in each JSON's metadata; dirs hold the *latest* sweep | + +Rule of thumb: **cross-release trend** questions read versioned summaries; +**cross-hardware comparison** questions read `comparison.json`. + +## The PreOptimizationTimes convention (for phases 2–4) + +- A **baseline** is a named, frozen snapshot of campaign results: + `results/baselines//` containing (a) the `comparison.json` + per swept cell, (b) the versioned summary JSONs the campaign produced, and + (c) a rendered **`.md`** — every headline number in one + browsable page (cells × configs × instruments). +- `PreOptimizationTimes` is the first such baseline: laptop CPU, HPC CPU and + HPC A100 (+ mp where supported); vmap-only runtime numbers using the + `vram/` batch-size table; laptop GPU appended later by hand. +- The top-level README dashboard grows a baseline column once + `results/baselines/PreOptimizationTimes/` exists — the "compare against + this" anchor for all optimization work that follows. +- Baselines are **append-only**: never edited after the campaign closes; a + post-optimization campaign snapshots a new name next to it. + +## Deliberately out of scope here + +Profiling runs (phases 2–4); searches profiling; `point_source` cells (in the +grid, excluded from this campaign); laptop-GPU rows (human-run follow-up); +gradient profiling (still in `autolens_workspace_developer`, folds in later); +the future PyAutoBrain profiling agent (separate `feature/pyautobrain/` task). diff --git a/scripts/README.md b/scripts/README.md new file mode 100644 index 0000000..b52c487 --- /dev/null +++ b/scripts/README.md @@ -0,0 +1,8 @@ +# scripts + +Repo tooling (not profiling scripts). + +- **`build_readme.py`** — renders the auto-generated dashboard tables in the + top-level and per-package READMEs from the artifacts under `results/`. + Run `python scripts/build_readme.py` after a profiling run and commit the + result; CI's `lint.yml` runs `--check` to enforce idempotence. diff --git a/scripts/build_readme.py b/scripts/build_readme.py index 9ce91d1..4330266 100644 --- a/scripts/build_readme.py +++ b/scripts/build_readme.py @@ -1,6 +1,6 @@ """ build_readme.py — refresh auto-generated tables in every README from the -latest versioned artifacts under `results/`. +latest artifacts under `results/`. Run from the repo root: @@ -10,42 +10,40 @@ Each table region in a README is delimited by sentinel comments, e.g. - + | ... | - + This script: - 1. Scans `results/**/*_summary_v.json`. - 2. Parses filenames into (section, sub-folder, script, instrument, version). - 3. Picks the latest version per group via PEP 440-ish dotted-version sort. - 4. Generates a markdown table per known region type and replaces the - content inside the matching sentinel block. - -Sections covered today: - - - top-level README.md - ... - - likelihood/README.md (section overview) - - likelihood/imaging/README.md | likelihood-imaging - - likelihood/interferometer/README.md | likelihood-interferometer - - likelihood/point_source/README.md | likelihood-point_source - - likelihood/datacube/README.md | likelihood-datacube - - simulators/README.md | simulators - - searches/nautilus/README.md | searches-nautilus - -Hardware-tier columns (CPU / laptop GPU / HPC GPU) are deferred — every -artifact today is implicitly CPU and the table shows a single "Latest" -column. Once future artifacts encode hardware in the filename or JSON -(`*_summary_v_.json` or `{"hardware": "a100"}`), the -column logic in `_render_*_table` will be extended without touching the -sentinel layout. + 1. Scans `results/{breakdown,simulators,searches}/**` for **versioned + artifacts** (`