refactor(zoo): natural-language format (drop front-matter) + overview page

arbor-zoo/README.md

@@ -9,14 +9,12 @@ Full format spec and the verifier's check list live in the docs:
 
 ## Format in one line
 
-Each `arbor-zoo/<name>/` folder is one benchmark: a **README.md** (a small YAML
-front-matter contract — metric, dev/test split, baseline, edit surface — plus a prose
-body: task, metric, how to run & optimize), a **PROVENANCE.md** card (setup &
-environment, baseline implementation, baseline reproduction, source, license,
-contamination, caveats), a runnable **baseline** (one or more code files), and a
-protected **eval entrypoint** (`eval.sh` or `eval.py`) that prints one `score: <float>`
-line for `dev` and `test`. The contract lives in the README front-matter — there is no
-separate manifest file.
+Each `arbor-zoo/<name>/` folder is one benchmark: a **README.md** (a plain-language
+description Arbor reads — the task, the metric, what it may edit, how dev/test differ), a
+**PROVENANCE.md** card for humans (source, setup, how the baseline works, contamination,
+caveats), a runnable **baseline** (one or more code files), and a protected **eval
+entrypoint** (`eval.sh` or `eval.py`) that prints one `score: <float>` line for `dev` and
+`test`. The format is documentation-first — there is no machine manifest.
 
 ## Packs
 

arbor-zoo/_template/PROVENANCE.md

@@ -1,42 +1,29 @@
 # Provenance — TODO_pack_name
 
-All seven headings below are required. The verifier checks they are present; a maintainer
-reads and accepts the content before the benchmark ships. The README front-matter holds
-the machine facts (metric, splits, baseline, edit surface); this card holds the human
-trust + setup story.
+This card is for humans. Fill in every section; a maintainer reads it before the benchmark
+is accepted. (`arbor benchmark verify` checks these headings are present.)
 
 ## Source
 
-Where the benchmark comes from — the paper, repo, or competition, with a link, and how
-the data was collected or generated.
+Where the benchmark comes from — paper, repo, or competition, with a link, and how the data
+was collected or generated.
 
 ## Setup & environment
 
-Hardware (CPU / GPU), Python version, install command, env vars the eval expects, and any
-API keys, downloads, or services the user must provision. State whether it is offline.
+Hardware (CPU / GPU), Python version, install command, env vars, and any API keys, downloads,
+or services the user must provision. State whether it's offline. License of the code and data,
+and whether the data may be redistributed.
 
-## Data source & license
+## Baseline
 
-Where the data comes from, its license, and whether it may be redistributed. If it may not
-be bundled, ship `data/download.sh` instead of the data.
-
-## Baseline implementation
-
-**How the shipped baseline works** — the algorithm/approach, why it scores what it does,
-and what headroom it deliberately leaves for Arbor.
-
-## Baseline reproduction
-
-The number `eval dev` prints today (this must match `baseline.score` in the front-matter)
-and any gap from the published number.
+How the shipped baseline works and what score it tends to produce. **Results vary** by user,
+hardware, and (for API tasks) model — note the range you saw rather than a single fixed number.
 
 ## Contamination assessment
 
-**Mandatory.** Could the test split already be in a model's pre-training data? Is the
-held-out split truly disjoint from dev? Explain why a high score reflects real capability
-and not memorisation.
+**Mandatory.** Could the test split be in a model's pre-training data? Is the held-out split
+truly disjoint from dev? Explain why a high score reflects real capability.
 
 ## Caveats
 
-Known limitations — hardware sensitivity, metric noise, scope, anything a user should know
-before trusting the number.
+Known limitations — hardware sensitivity, metric noise, scope.

arbor-zoo/_template/README.md

@@ -1,64 +1,34 @@
----
-name: TODO_pack_name
-metric:
-  direction: maximize          # maximize | minimize
-eval:
-  cmd: "bash eval.sh"          # optional; omit to use the eval.sh/eval.py convention.
-                               # the verifier appends `dev` / `test`
-splits:                        # how dev/test differ — lets the verifier prove they're disjoint
-  kind: seed_range             # seed_range | path
-  dev:  { base: 1000, count: 3 }
-  test: { base: 9000, count: 3 }
-  # path example:
-  #   kind: path
-  #   dev:  ["data/dev/**"]
-  #   test: ["data/test/**"]
-baseline:
-  score: 0.0                   # what `eval dev` prints today (verifier checks reality matches)
-  tolerance: 0.0               # relative margin for reproduce + determinism
-  kind: exact                  # exact | timing  (timing uses a ratio tolerance)
-edit: [solution.py]            # the editable surface (1+ files/globs); everything else is protected
-# frozen:                      # OPTIONAL — the freeze axis (what's held fixed for comparability)
-#   model: gpt-x               #   freeze the model (edit = scaffold/prompt), OR
-#   budget: "wall-clock 1h"    #   freeze only a budget (edit spans training+scaffold)
----
-
 # TODO_pack_name
 
-One-line summary of the benchmark. Fill in the front-matter above (the machine contract)
-and the four sections below — the verifier checks both, and they are what users and Arbor
-read.
+One-line summary of the benchmark — this first line shows up in `arbor benchmark list`.
 
-## Task & metric
+This README is what **Arbor reads** at intake to understand the task. Write it in plain
+language; there is no rigid format. Cover the four things below.
 
-What is the task, what does a solution look like, what number is optimized, and which
-direction is better (maximize / minimize)? Name the **edit surface** — the baseline
-code Arbor may change (one file like `solution.py`, or a whole set of files / a
-directory) — and what is off-limits (the eval harness and any ground-truth files).
+## The task
 
-## Run the baseline
+What the task is and what a solution looks like.
 
-The exact commands and the score the shipped baseline prints. The eval entrypoint is
-`eval.sh` (or `eval.py`), invoked with a split:
+## Metric
 
-```bash
-bash eval.sh dev     # iterate here
-bash eval.sh test    # held-out gate
-```
+What `eval.sh` / `eval.py` prints (one `score:` line) and whether **higher or lower** is
+better.
 
-Point to [`PROVENANCE.md`](PROVENANCE.md) → Setup & environment for install / hardware
-/ keys.
+## What Arbor may edit
 
-## Optimize with Arbor
+Which file(s) are the editable baseline (e.g. `solution.py`), and what's off-limits (the
+eval harness and any ground-truth / data).
 
-How to point Arbor at this benchmark (copy it out of the Arbor checkout first, it uses
-git worktrees) and the suggested research contract — metric, dev/test discipline, what
-may and may not be edited.
+## Dev / test
 
-## Provenance
+How dev and test differ, so the held-out split is clear — disjoint seeds, or separate
+`data/dev/` and `data/test/` folders.
 
-See [`PROVENANCE.md`](PROVENANCE.md). Verify the benchmark with:
+## Run it
 
 ```bash
-arbor benchmark verify arbor-zoo/<name>
+python eval.py --split dev    # iterate here
+python eval.py --split test   # held-out gate
 ```
+
+See [`PROVENANCE.md`](PROVENANCE.md) for source, setup, and the baseline write-up.

arbor-zoo/_template/eval.py

@@ -6,17 +6,16 @@
     score: <float>
 
 A candidate that fails correctness should still print a score (e.g. `score: 0.0`)
-so the metric is always parseable. dev and test MUST use disjoint data — declare the
-split in the README front-matter (`splits:`) so the verifier can prove disjointness, and
-keep these constants in sync with it.
+so the metric is always parseable. dev and test MUST use disjoint data — describe the
+split in the README and PROVENANCE so a reviewer can confirm it's truly held out.
 """
 
 from __future__ import annotations
 
 import argparse
 
-# The held-out split. Keep in sync with `splits:` in the README front-matter — the
-# verifier proves disjointness against that declaration.
+# The held-out split. Describe it in the README (Dev / test) and PROVENANCE so a
+# reviewer can confirm dev and test never overlap.
 DEV_SEED_BASE = 1000
 TEST_SEED_BASE = 9000
 N_INSTANCES = 3

arbor-zoo/algotune_knn/PROVENANCE.md

@@ -3,83 +3,47 @@
 ## Source
 
 Modelled on [AlgoTune](https://algotune.io/) (NeurIPS 2025), a benchmark of
-single-editable-solver optimization tasks. This benchmark is **not** a verbatim copy
-of any AlgoTune task — it is an independent, self-contained re-implementation of the
-same *structure* (one editable solver + a fixed problem generator, reference solver,
-and correctness verifier) for a brute-force k-nearest-neighbours task. It originated
-as the worked example in [`examples/algotune_knn`](../../examples/algotune_knn).
+single-editable-solver optimization tasks. This is **not** a verbatim copy of any AlgoTune
+task — it's an independent re-implementation of the same *structure* (one editable solver +
+a fixed problem generator, reference solver, and correctness verifier) for a brute-force
+k-nearest-neighbours task. The data is generated deterministically from integer seeds via
+`numpy.random.default_rng(seed)`; nothing is downloaded or scraped.
 
 ## Setup & environment
 
 - **Hardware:** CPU only — no GPU.
-- **Python:** ≥ 3.10.
-- **Install:** `pip install -r requirements.txt` (just NumPy).
-- **Services / network:** none — fully offline; nothing is downloaded at eval time.
-- **Threads:** `eval.sh` exports `OMP_NUM_THREADS=1` (and the BLAS equivalents) so the
-  measured time reflects the *algorithm*, not the core count. Do not remove this — the
-  speedup ratio is only stable run-to-run under single-thread pinning.
+- **Python:** ≥ 3.10. **Install:** `pip install -r requirements.txt` (just NumPy).
+- **Offline:** yes — no network, no services.
+- **License:** MIT (code and the synthetic, seed-generated data); freely redistributable.
+- **Threads:** `eval.sh` pins `OMP_NUM_THREADS=1` (and BLAS equivalents) so the measured
+  time reflects the algorithm, not the core count — do not remove it.
 
-## Data source & license
+## Baseline
 
-There is **no external dataset**. Every problem instance is generated deterministically
-at eval time from an integer seed via `numpy.random.default_rng(seed)` (standard-normal
-point clouds). Nothing is downloaded, scraped, or redistributed.
+The shipped `solution.py` is the **naive brute-force baseline**, identical to
+`task.reference_solver`: full pairwise distances by broadcasting, then `argsort` and take the
+first `k`. Correct but unoptimised, so its speedup over the reference is **~1.0x** by
+construction — that's the number to beat. The headroom (partial selection via `argpartition`,
+the `|x−y|² = |x|² − 2x·y + |y|²` GEMM expansion, dtype/blocking) is what makes the search
+interesting.
 
-- Code license: **MIT** (same as the Arbor repository).
-- Data license: **MIT** — the data is synthetic and produced by the bundled `task.py`;
-  bundling is trivially allowed because there is nothing to redistribute beyond
-  generator code.
-
-The "data" is the generator: `task.generate_problem(seed, n_db, n_query, dim)` returns a
-database/query pair; `eval.py` builds dev instances from seeds `1000..1002` and test
-instances from `9000..9002`. No collection, labelling, or curation is involved —
-re-running the generator reproduces the exact bytes.
-
-## Baseline implementation
-
-The shipped `solution.py` is the **naive brute-force baseline**, intentionally
-identical to `task.reference_solver`:
-
-1. Compute the full pairwise distance matrix between every query and every database
-   point by broadcasting — `diff = queries[:, None, :] - database[None, :, :]`, then
-   `d2 = (diff * diff).sum(axis=2)` — an `O(n_query · n_db · dim)` operation.
-2. `argsort` every row of `d2` in full and slice the first `k` columns.
-
-It is correct but deliberately unoptimised, so its speedup over the reference is **1.0x
-by construction** — that is the number Arbor has to beat. The headroom (partial
-selection via `argpartition`, the `|x−y|² = |x|² − 2x·y + |y|²` GEMM expansion, dtype
-and blocking tweaks) is what makes the optimization loop interesting; none of it is
-applied in the baseline.
-
-## Baseline reproduction
-
-- **Published baseline:** ~1.0x — the baseline equals the reference by construction.
-- **Bundled baseline (measured):** `bash eval.sh dev` prints `score:` around
-  **0.99–1.01x**, fluctuating a couple of percent with timing noise on a quiet
-  single-thread CPU (the numerator and denominator run identical code).
-- **Gap:** none. Any deviation is pure timing jitter, not an algorithmic difference.
+**Results vary:** the score is a ratio of medians, so it wobbles a couple of percent
+run-to-run and depends on the machine — the *ratio* is stable under single-thread pinning,
+the absolute speedup is not.
 
 ## Contamination assessment
 
-**No held-out leakage and no pre-training contamination.**
-
-- **Dev/test isolation:** dev seeds `1000..1002` and test seeds `9000..9002` are
-  disjoint integer ranges (see `eval.py` `DEV_SEED_BASE` / `TEST_SEED_BASE`). A solution
-  tuned on dev has never seen the test instances.
-- **No web/pre-training contamination:** the instances are random point clouds generated
-  from seeds, not text or images from any corpus, so there is nothing a model could have
-  memorised. The metric is *wall-clock speedup of a computation*, not recall of an
-  answer — "fast but wrong" scores 0 via the independent `is_solution` gate, which
-  recomputes the ground truth.
+No held-out leakage and no pre-training contamination. Dev seeds `1000..1002` and test seeds
+`9000..9002` are disjoint ranges (see `DEV_SEED_BASE` / `TEST_SEED_BASE` in `eval.py`), so a
+solution tuned on dev has never seen the test instances. The instances are random point
+clouds generated from seeds — not text or images from any corpus — so there is nothing a
+model could have memorised, and the metric is wall-clock speedup, not recall of an answer
+(`is_solution` recomputes the ground truth, so "fast but wrong" scores 0).
 
 ## Caveats
 
-- **Timing metric is hardware-dependent.** The absolute speedup depends on the machine;
-  the *ratio* `median(reference)/median(solution)` is what is compared, and it is stable
-  run-to-run only under single-thread pinning (enforced by `eval.sh`).
-- **Determinism is not bit-exact.** Because the score is a ratio of medians, two runs
-  differ by a few percent; the verifier's `determinism` check is advisory for exactly
-  this reason — a human confirms the variation is timing noise, not a correctness bug.
-- **Seed bases live in `eval.py`.** `DEV_SEED_BASE` / `TEST_SEED_BASE` in
-  [`eval.py`](eval.py) define the held-out split; if you change one, change both and
-  re-confirm disjointness here.
+- **Timing metric is hardware-dependent** — compare the ratio, not the absolute number, and
+  keep the single-thread pinning.
+- **Not bit-exact** — two runs differ by a few percent; that's timing noise, not a bug.
+- **Seed bases live in `eval.py`** — if you change `DEV_SEED_BASE` / `TEST_SEED_BASE`, keep
+  dev and test disjoint.