From 115c313101b4a6ff6126f29ed18db4f19ede4cbe Mon Sep 17 00:00:00 2001 From: ignorejjj Date: Wed, 24 Jun 2026 14:32:15 +0800 Subject: [PATCH 1/3] docs(zoo): add Benchmark Zoo overview & roadmap page A user-facing big-picture page for the zoo: the vision (regression-harness-first, not a trophy board), the core idea (a pack is a locked optimization problem), the categorization (two layers + the freeze axis + task shapes), what's shipped vs planned, the four usage entry points (run / scaffold / author / add), and the discipline. Bilingual (EN + zh); nav groups it with the existing format reference. --- docs/zoo-overview.md | 115 ++++++++++++++++++++++++++++++++++++++++ docs/zoo-overview.zh.md | 103 +++++++++++++++++++++++++++++++++++ mkdocs.yml | 6 ++- 3 files changed, 223 insertions(+), 1 deletion(-) create mode 100644 docs/zoo-overview.md create mode 100644 docs/zoo-overview.zh.md diff --git a/docs/zoo-overview.md b/docs/zoo-overview.md new file mode 100644 index 0000000..9dacf7a --- /dev/null +++ b/docs/zoo-overview.md @@ -0,0 +1,115 @@ +# Benchmark Zoo — overview & roadmap + +This page is the **big picture** of the benchmark zoo: what it is, how we think about it, +what's built versus planned, and how you use it today. For the exact folder format and the +verifier's checks, see the [format reference](zoo.md). + +## What it is + +The **benchmark zoo** is a curated, verifiable set of optimization problems that Arbor can +be pointed at — packaged in one standard format so anyone can re-run them and check the +numbers. It lives in [`arbor-zoo/`](https://github.com/RUC-NLPIR/Arbor/tree/main/arbor-zoo), +one folder per benchmark. + +Two principles shape everything: + +- **Regression harness first, showcase second.** The zoo's first job is to keep Arbor + honestly tested on a small set of hand-checked, reproducible tasks. Being an outward + showcase is secondary. +- **Quality-capped, not a trophy board.** It is deliberately **not** a leaderboard of our + own results. We use representative work to *position a reusable, fairly-baselined task* — + never to catalogue how many papers we beat. + +## The one idea to internalize + +**A pack is not "a dataset" — it is a locked optimization problem:** + +> `data + frozen substrate + edit surface + metric + reference baseline` + +The same dataset climbed from a different angle (tune the prompt vs. fine-tune the model vs. +design a scaffold) is a **different pack**. The *angle is the pack*, encoded in the README +front-matter (what's editable, what's frozen, what's measured). This is why "how you climb a +benchmark" is decided when the pack is authored, not at run time. + +## How we categorize (the mental map) + +**Two layers.** + +| Layer | What | Purpose | +| --- | --- | --- | +| **Stable core** | synthetic / self-contained tasks (e.g. `algotune_knn`) | Arbor's regression harness — stable, cheap, deterministic | +| **Frontier shelf** | tasks anchored on representative *hot work* | track the field; the "climb real benchmarks" use case | + +**The freeze axis** — every pack declares what it holds fixed, which decides what it measures: + +- **Freeze the model** (edit a scaffold/prompt) → measures the *method*; cheap, clean + attribution, narrow coverage. +- **Freeze a budget** (compute/wall-clock; edit training + scaffold + data) → *any* mechanism + competes on equal footing (the MLE-bench style); broad coverage, needs the compute. + +**Task shapes that fit** — anything where you optimize an *artifact* against a held-out score: +algorithm/efficiency (kernels, AlgoTune), ML-engineering/tabular (Kaggle/MLE-bench), +coding/agents (SWE-bench), prompt/reasoning scaffolds, and training-efficiency. **Out of +scope**: tasks that only *evaluate a frozen model* (raw multimodal QA) or need hardware/sim +(embodied robotics) — there is no artifact for Arbor to optimize. + +## What's built vs. planned + +| Capability | Command / artifact | Status | +| --- | --- | --- | +| Pack format + front-matter contract (incl. `frozen:`) | `arbor-zoo//` | ✅ shipped | +| The gate that admits a pack | `arbor benchmark verify` | ✅ shipped | +| Index the packs | `arbor benchmark list` | ✅ shipped | +| Make a local dir Arbor-ready / author a pack | `arbor benchmark scaffold` (+ MCP tool, intake wiring) | ✅ shipped | +| Acquire a remote benchmark + scaffold a draft | `arbor benchmark add` (git / HF, global cache) | ✅ spine shipped | +| First verified dogfood pack | `arbor-zoo/algotune_knn` | ✅ shipped | +| **Collection intelligence** — survey a direction/work → harvest baseline → bring it up | the agent stages behind `add` | ⏳ planned | +| More curated packs across task shapes | `arbor-zoo/…` | ⏳ ongoing | +| Browsable zoo / leaderboard view | — | ⏳ deferred | +| Optimizing across many benchmarks at once | — | 🔭 far-future | + +The internal design behind the collection feature lives in the dev notes +(`docs/dev/benchmark-add.md`) along with a researched +[backlog of collectable benchmarks](https://github.com/RUC-NLPIR/Arbor/blob/main/docs/dev/benchmark-backlog.md). + +## How you use it today + +There are four entry points, from "just run it" to "add a new one": + +1. **Run Arbor on an existing pack.** Copy a pack out of the checkout (Arbor uses git + worktrees), then point Arbor at it: + ```bash + cp -r arbor-zoo/algotune_knn /tmp/algotune_knn + cd /tmp/algotune_knn && git init -q && git add -A && git commit -qm baseline + arbor # confirm the contract, then it iterates + ``` +2. **Make your own task Arbor-ready.** If you have code but no runnable eval / split, scaffold + the measurement plumbing (it never writes the solution): + ```bash + arbor benchmark scaffold ./my_task --style light # eval + split + solution stub + arbor benchmark scaffold ./my_task --style zoo # + README contract + PROVENANCE + ``` +3. **Author a zoo pack.** Scaffold with `--style zoo`, fill in the baseline + eval + + PROVENANCE, then gate it: `arbor benchmark verify arbor-zoo/` until it exits 0, and a + maintainer accepts it. Drafting may be automated; **acceptance is a human step**. +4. **Collect a benchmark** (Phase 1 today). Acquire a remote benchmark into the global cache + and scaffold a draft to complete: + ```bash + arbor benchmark add https://github.com/owner/repo --name my_bench + ``` + +## The discipline (why a green check means something) + +A pack only counts when these hold — partly machine-enforced by `verify`, partly human-reviewed: + +- **Held-out**: dev/test are provably disjoint; merges gate on the protected test split. +- **Frozen substrate**: what's fixed is declared, so an improvement is attributable and two + runs are comparable (no winning by swapping in a bigger model). +- **Provenance + human acceptance**: source, baseline, and a mandatory contamination + assessment are written down and read by a maintainer — never auto-accepted. + +That's the whole shape: the zoo marks out *locked optimization problems*, Arbor runs its +normal loop inside them, and the verifier + provenance keep the results trustworthy. + +See the [format reference](zoo.md) to author one, or the +[roadmap](roadmap.md) for where the wider effort is going. diff --git a/docs/zoo-overview.zh.md b/docs/zoo-overview.zh.md new file mode 100644 index 0000000..cf528bb --- /dev/null +++ b/docs/zoo-overview.zh.md @@ -0,0 +1,103 @@ +# 基准动物园 —— 总览与路线 + +这一页是基准动物园的**全景**:它是什么、我们怎么想、做了什么、还没做什么、现在怎么用。确切的 +文件夹格式和校验器检查见[格式参考](zoo.md)。 + +## 它是什么 + +**基准动物园**是一组经过筛选、可校验的优化题,可以让 Arbor 对准它们——用同一套标准格式打包, +让任何人都能复跑并核对数字。它放在 +[`arbor-zoo/`](https://github.com/RUC-NLPIR/Arbor/tree/main/arbor-zoo) 里,每个基准一个文件夹。 + +两条原则贯穿始终: + +- **回归 harness 优先,展示其次。** zoo 的首要职责是用一小批人工核对、可复跑的任务,让 Arbor + 始终被诚实地测着。对外展示是次要的。 +- **质量封顶,不做战绩榜。** 它**刻意不做**我们自己结果的排行榜。我们用代表性工作来*把一道 + 可复用、baseline 公平的任务定位准*——而不是去攒"我们超过了多少篇论文"。 + +## 要内化的那一个核心观念 + +**一个 pack 不是"一个数据集",而是一道锁死的优化题:** + +> `数据 + 冻结底座 + 编辑面 + 指标 + 参照基线` + +同一个数据集,从不同角度刷(调 prompt vs 微调模型 vs 设计 scaffold),就是**不同的 pack**。 +*角度即 pack*,编码在 README front-matter 里(什么可改、什么冻结、测什么)。所以"怎么刷一个 +benchmark"是在**写 pack 那一刻**决定的,不在运行时。 + +## 我们的分类思路(心智地图) + +**两层结构。** + +| 层 | 是什么 | 用途 | +| --- | --- | --- | +| **稳定核心层** | 合成/自包含任务(如 `algotune_knn`) | Arbor 的回归 harness——稳定、便宜、确定性 | +| **前沿货架层** | 锚定代表性*热门工作*的任务 | 追踪前沿;"刷真实 benchmark"的场景 | + +**冻结轴** —— 每个 pack 声明它固定什么,这决定了它测的是什么: + +- **冻模型**(改 scaffold/prompt)→ 测*方法*;便宜、归因干净、覆盖窄。 +- **冻预算**(算力/wall-clock;改训练+scaffold+数据)→ *任意*机制平等竞争(MLE-bench 式); + 覆盖广,但要算力撑得住。 + +**适合的任务形态** —— 凡是"优化一个 artifact、对着 held-out 分数刷"的都适合:算法/效率(kernel、 +AlgoTune)、ML 工程/表格(Kaggle/MLE-bench)、代码/agent(SWE-bench)、prompt/推理 scaffold、 +训练效率。**不在范围内**:只*评测一个冻结模型*的(纯多模态 QA)、或要硬件/仿真的(具身机器人) +——那里没有可供 Arbor 优化的 artifact。 + +## 做了什么 vs 还没做 + +| 能力 | 命令 / 产物 | 状态 | +| --- | --- | --- | +| pack 格式 + front-matter 契约(含 `frozen:`) | `arbor-zoo//` | ✅ 已交付 | +| 准入门 | `arbor benchmark verify` | ✅ 已交付 | +| 索引 pack | `arbor benchmark list` | ✅ 已交付 | +| 把本地目录补成 Arbor-ready / 写一个 pack | `arbor benchmark scaffold`(+ MCP 工具、intake 接线) | ✅ 已交付 | +| 获取远端 benchmark + 起草草稿 | `arbor benchmark add`(git / HF,全局缓存) | ✅ 骨架已交付 | +| 第一个过校验的 dogfood pack | `arbor-zoo/algotune_knn` | ✅ 已交付 | +| **收集智能** —— 综述一个方向/工作 → 收割 baseline → 跑通 | `add` 背后的 agent 阶段 | ⏳ 计划中 | +| 跨形态的更多策展 pack | `arbor-zoo/…` | ⏳ 进行中 | +| 可浏览的 zoo / leaderboard 视图 | — | ⏳ 暂缓 | +| 同时对多个 benchmark 优化 | — | 🔭 远期 | + +收集功能背后的内部设计在 dev 笔记(`docs/dev/benchmark-add.md`)里,并附了一份调研过的 +[可收集 benchmark 待办清单](https://github.com/RUC-NLPIR/Arbor/blob/main/docs/dev/benchmark-backlog.md)。 + +## 现在怎么用 + +四个入口,从"直接跑"到"加一个新的": + +1. **在现有 pack 上跑 Arbor。** 把 pack 拷出检出目录(Arbor 用 git worktree),再让 Arbor 对准它: + ```bash + cp -r arbor-zoo/algotune_knn /tmp/algotune_knn + cd /tmp/algotune_knn && git init -q && git add -A && git commit -qm baseline + arbor # 确认契约,然后它开始迭代 + ``` +2. **把你自己的任务补成 Arbor-ready。** 如果你有代码但没有能跑的 eval / 划分,补上测量管线 + (它绝不写解法): + ```bash + arbor benchmark scaffold ./my_task --style light # eval + 划分 + solution 占位 + arbor benchmark scaffold ./my_task --style zoo # + README 契约 + PROVENANCE + ``` +3. **写一个 zoo pack。** 用 `--style zoo` 脚手架,填好 baseline + eval + PROVENANCE,然后过门: + 反复 `arbor benchmark verify arbor-zoo/` 直到退出 0,再由维护者接受。起草可自动, + **接受是人工步骤**。 +4. **收集一个 benchmark**(目前是 Phase 1)。把远端 benchmark 拉进全局缓存并起草草稿待补全: + ```bash + arbor benchmark add https://github.com/owner/repo --name my_bench + ``` + +## 纪律(为什么一个绿勾是有分量的) + +一个 pack 只有满足这些才算数——一部分由 `verify` 机器强制,一部分人工审核: + +- **留出**:dev/test 可证明不相交;merge 以受保护的 test 划分为门。 +- **冻结底座**:固定了什么是声明出来的,所以"提升"可归因、两次 run 可比(不能靠换更大模型取胜)。 +- **来源 + 人工接受**:来源、baseline、以及一份必填的污染评估,都写下来并由维护者读过——绝不 + 自动接受。 + +整体就是这个形状:zoo 划出一道道*锁死的优化题*,Arbor 在里面跑它本来的循环,校验器 + 来源卡片 +保证结果可信。 + +写一个 pack 见[格式参考](zoo.md),整条线的去向见[路线图](roadmap.md)。 diff --git a/mkdocs.yml b/mkdocs.yml index 5d6fc61..a393138 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -58,7 +58,9 @@ nav: - Interaction Modes: interaction-modes.md - Web UI & Monitoring: web-ui.md - Plugins: plugins.md - - Benchmark Zoo: zoo.md + - Benchmark Zoo: + - Overview: zoo-overview.md + - Format & verifier: zoo.md - Skills: skills.md - Outputs & Resume: outputs-and-resume.md - Reference: @@ -120,6 +122,8 @@ plugins: Web UI & Monitoring: Web UI 与监控 Plugins: 插件 Benchmark Zoo: 基准动物园 + Overview: 概览 + Format & verifier: 格式与校验器 Skills: 技能 Outputs & Resume: 输出与续跑 Reference: 参考 From 4d826ca732bf1b0d8b3a89c9831f47ef462d6dac Mon Sep 17 00:00:00 2001 From: ignorejjj Date: Wed, 24 Jun 2026 15:48:51 +0800 Subject: [PATCH 2/3] =?UTF-8?q?refactor(zoo):=20natural-language=20format?= =?UTF-8?q?=20=E2=80=94=20drop=20the=20front-matter=20contract?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The README front-matter contract (metric / splits / baseline / tolerance / edit / frozen) proved too rigid, and a fixed baseline number isn't universal — the same baseline gives different scores by user, hardware, and model. So the format goes documentation-first: - **README is natural language, for Arbor.** It describes the task, the metric, what Arbor may edit, and how dev/test differ — in plain prose Arbor reads at intake. No YAML manifest. - **PROVENANCE is for humans.** Source, setup, how the baseline works (and that results vary), contamination, caveats. - **`verify` is a light structural lint** — README + PROVENANCE(+sections) + an eval entrypoint present. It no longer runs the eval (a score isn't a universal value), so the held-out/baseline/determinism machine checks are gone; those are stated in PROVENANCE prose and judged by a human. Changes: pack.py (drop Contract/front-matter parsing), verify.py (structural-only), scaffold.py (emit NL README + PROVENANCE; signature now name/metric_direction/style/ split_kind/eval_entrypoint/edit), collect.py + benchmark_cmd.py + mcp session_ops/server (drop front-matter/eval-run args), the algotune_knn pack + _template (NL README, PROVENANCE without a baseline number), and docs/zoo.{md,zh} + arbor-zoo/README. Also: a simpler, user-focused zoo overview page and "基准库" (not "基准动物园") in zh. Tests rewritten for the light format. ruff + mypy + full suite + mkdocs --strict green. Note: the internal dev design notes (docs/dev/benchmark-add.md, benchmark-backlog.md) still describe the old front-matter contract and need a later reconciliation pass when the collection agent stages are built. --- arbor-zoo/README.md | 14 +- arbor-zoo/_template/PROVENANCE.md | 39 +-- arbor-zoo/_template/README.md | 68 ++--- arbor-zoo/_template/eval.py | 9 +- arbor-zoo/algotune_knn/PROVENANCE.md | 98 +++---- arbor-zoo/algotune_knn/README.md | 97 +++---- docs/zoo-overview.md | 143 +++-------- docs/zoo-overview.zh.md | 123 +++------ docs/zoo.md | 187 +++++--------- docs/zoo.zh.md | 155 ++++------- mkdocs.yml | 2 +- src/cli/commands/benchmark_cmd.py | 50 +--- src/mcp/server.py | 22 +- src/mcp/session_ops.py | 13 +- src/zoo/__init__.py | 16 +- src/zoo/collect.py | 12 +- src/zoo/pack.py | 97 ++----- src/zoo/scaffold.py | 174 ++++++------- src/zoo/verify.py | 368 ++++----------------------- tests/test_benchmark_cli.py | 2 +- tests/test_mcp_session_ops.py | 16 +- tests/test_zoo_algotune_knn.py | 47 +--- tests/test_zoo_pack.py | 86 ++----- tests/test_zoo_scaffold.py | 89 ++----- tests/test_zoo_verify.py | 184 +++----------- 25 files changed, 543 insertions(+), 1568 deletions(-) diff --git a/arbor-zoo/README.md b/arbor-zoo/README.md index beff97f..9ee002e 100644 --- a/arbor-zoo/README.md +++ b/arbor-zoo/README.md @@ -9,14 +9,12 @@ Full format spec and the verifier's check list live in the docs: ## Format in one line -Each `arbor-zoo//` folder is one benchmark: a **README.md** (a small YAML -front-matter contract — metric, dev/test split, baseline, edit surface — plus a prose -body: task, metric, how to run & optimize), a **PROVENANCE.md** card (setup & -environment, baseline implementation, baseline reproduction, source, license, -contamination, caveats), a runnable **baseline** (one or more code files), and a -protected **eval entrypoint** (`eval.sh` or `eval.py`) that prints one `score: ` -line for `dev` and `test`. The contract lives in the README front-matter — there is no -separate manifest file. +Each `arbor-zoo//` folder is one benchmark: a **README.md** (a plain-language +description Arbor reads — the task, the metric, what it may edit, how dev/test differ), a +**PROVENANCE.md** card for humans (source, setup, how the baseline works, contamination, +caveats), a runnable **baseline** (one or more code files), and a protected **eval +entrypoint** (`eval.sh` or `eval.py`) that prints one `score: ` line for `dev` and +`test`. The format is documentation-first — there is no machine manifest. ## Packs diff --git a/arbor-zoo/_template/PROVENANCE.md b/arbor-zoo/_template/PROVENANCE.md index a7c7b70..0652d1f 100644 --- a/arbor-zoo/_template/PROVENANCE.md +++ b/arbor-zoo/_template/PROVENANCE.md @@ -1,42 +1,29 @@ # Provenance — TODO_pack_name -All seven headings below are required. The verifier checks they are present; a maintainer -reads and accepts the content before the benchmark ships. The README front-matter holds -the machine facts (metric, splits, baseline, edit surface); this card holds the human -trust + setup story. +This card is for humans. Fill in every section; a maintainer reads it before the benchmark +is accepted. (`arbor benchmark verify` checks these headings are present.) ## Source -Where the benchmark comes from — the paper, repo, or competition, with a link, and how -the data was collected or generated. +Where the benchmark comes from — paper, repo, or competition, with a link, and how the data +was collected or generated. ## Setup & environment -Hardware (CPU / GPU), Python version, install command, env vars the eval expects, and any -API keys, downloads, or services the user must provision. State whether it is offline. +Hardware (CPU / GPU), Python version, install command, env vars, and any API keys, downloads, +or services the user must provision. State whether it's offline. License of the code and data, +and whether the data may be redistributed. -## Data source & license +## Baseline -Where the data comes from, its license, and whether it may be redistributed. If it may not -be bundled, ship `data/download.sh` instead of the data. - -## Baseline implementation - -**How the shipped baseline works** — the algorithm/approach, why it scores what it does, -and what headroom it deliberately leaves for Arbor. - -## Baseline reproduction - -The number `eval dev` prints today (this must match `baseline.score` in the front-matter) -and any gap from the published number. +How the shipped baseline works and what score it tends to produce. **Results vary** by user, +hardware, and (for API tasks) model — note the range you saw rather than a single fixed number. ## Contamination assessment -**Mandatory.** Could the test split already be in a model's pre-training data? Is the -held-out split truly disjoint from dev? Explain why a high score reflects real capability -and not memorisation. +**Mandatory.** Could the test split be in a model's pre-training data? Is the held-out split +truly disjoint from dev? Explain why a high score reflects real capability. ## Caveats -Known limitations — hardware sensitivity, metric noise, scope, anything a user should know -before trusting the number. +Known limitations — hardware sensitivity, metric noise, scope. diff --git a/arbor-zoo/_template/README.md b/arbor-zoo/_template/README.md index f34b2b4..3277590 100644 --- a/arbor-zoo/_template/README.md +++ b/arbor-zoo/_template/README.md @@ -1,64 +1,34 @@ ---- -name: TODO_pack_name -metric: - direction: maximize # maximize | minimize -eval: - cmd: "bash eval.sh" # optional; omit to use the eval.sh/eval.py convention. - # the verifier appends `dev` / `test` -splits: # how dev/test differ — lets the verifier prove they're disjoint - kind: seed_range # seed_range | path - dev: { base: 1000, count: 3 } - test: { base: 9000, count: 3 } - # path example: - # kind: path - # dev: ["data/dev/**"] - # test: ["data/test/**"] -baseline: - score: 0.0 # what `eval dev` prints today (verifier checks reality matches) - tolerance: 0.0 # relative margin for reproduce + determinism - kind: exact # exact | timing (timing uses a ratio tolerance) -edit: [solution.py] # the editable surface (1+ files/globs); everything else is protected -# frozen: # OPTIONAL — the freeze axis (what's held fixed for comparability) -# model: gpt-x # freeze the model (edit = scaffold/prompt), OR -# budget: "wall-clock 1h" # freeze only a budget (edit spans training+scaffold) ---- - # TODO_pack_name -One-line summary of the benchmark. Fill in the front-matter above (the machine contract) -and the four sections below — the verifier checks both, and they are what users and Arbor -read. +One-line summary of the benchmark — this first line shows up in `arbor benchmark list`. -## Task & metric +This README is what **Arbor reads** at intake to understand the task. Write it in plain +language; there is no rigid format. Cover the four things below. -What is the task, what does a solution look like, what number is optimized, and which -direction is better (maximize / minimize)? Name the **edit surface** — the baseline -code Arbor may change (one file like `solution.py`, or a whole set of files / a -directory) — and what is off-limits (the eval harness and any ground-truth files). +## The task -## Run the baseline +What the task is and what a solution looks like. -The exact commands and the score the shipped baseline prints. The eval entrypoint is -`eval.sh` (or `eval.py`), invoked with a split: +## Metric -```bash -bash eval.sh dev # iterate here -bash eval.sh test # held-out gate -``` +What `eval.sh` / `eval.py` prints (one `score:` line) and whether **higher or lower** is +better. -Point to [`PROVENANCE.md`](PROVENANCE.md) → Setup & environment for install / hardware -/ keys. +## What Arbor may edit -## Optimize with Arbor +Which file(s) are the editable baseline (e.g. `solution.py`), and what's off-limits (the +eval harness and any ground-truth / data). -How to point Arbor at this benchmark (copy it out of the Arbor checkout first, it uses -git worktrees) and the suggested research contract — metric, dev/test discipline, what -may and may not be edited. +## Dev / test -## Provenance +How dev and test differ, so the held-out split is clear — disjoint seeds, or separate +`data/dev/` and `data/test/` folders. -See [`PROVENANCE.md`](PROVENANCE.md). Verify the benchmark with: +## Run it ```bash -arbor benchmark verify arbor-zoo/ +python eval.py --split dev # iterate here +python eval.py --split test # held-out gate ``` + +See [`PROVENANCE.md`](PROVENANCE.md) for source, setup, and the baseline write-up. diff --git a/arbor-zoo/_template/eval.py b/arbor-zoo/_template/eval.py index 9f3ab1f..fd3c6fc 100644 --- a/arbor-zoo/_template/eval.py +++ b/arbor-zoo/_template/eval.py @@ -6,17 +6,16 @@ score: A candidate that fails correctness should still print a score (e.g. `score: 0.0`) -so the metric is always parseable. dev and test MUST use disjoint data — declare the -split in the README front-matter (`splits:`) so the verifier can prove disjointness, and -keep these constants in sync with it. +so the metric is always parseable. dev and test MUST use disjoint data — describe the +split in the README and PROVENANCE so a reviewer can confirm it's truly held out. """ from __future__ import annotations import argparse -# The held-out split. Keep in sync with `splits:` in the README front-matter — the -# verifier proves disjointness against that declaration. +# The held-out split. Describe it in the README (Dev / test) and PROVENANCE so a +# reviewer can confirm dev and test never overlap. DEV_SEED_BASE = 1000 TEST_SEED_BASE = 9000 N_INSTANCES = 3 diff --git a/arbor-zoo/algotune_knn/PROVENANCE.md b/arbor-zoo/algotune_knn/PROVENANCE.md index 4c7485f..ca54db7 100644 --- a/arbor-zoo/algotune_knn/PROVENANCE.md +++ b/arbor-zoo/algotune_knn/PROVENANCE.md @@ -3,83 +3,47 @@ ## Source Modelled on [AlgoTune](https://algotune.io/) (NeurIPS 2025), a benchmark of -single-editable-solver optimization tasks. This benchmark is **not** a verbatim copy -of any AlgoTune task — it is an independent, self-contained re-implementation of the -same *structure* (one editable solver + a fixed problem generator, reference solver, -and correctness verifier) for a brute-force k-nearest-neighbours task. It originated -as the worked example in [`examples/algotune_knn`](../../examples/algotune_knn). +single-editable-solver optimization tasks. This is **not** a verbatim copy of any AlgoTune +task — it's an independent re-implementation of the same *structure* (one editable solver + +a fixed problem generator, reference solver, and correctness verifier) for a brute-force +k-nearest-neighbours task. The data is generated deterministically from integer seeds via +`numpy.random.default_rng(seed)`; nothing is downloaded or scraped. ## Setup & environment - **Hardware:** CPU only — no GPU. -- **Python:** ≥ 3.10. -- **Install:** `pip install -r requirements.txt` (just NumPy). -- **Services / network:** none — fully offline; nothing is downloaded at eval time. -- **Threads:** `eval.sh` exports `OMP_NUM_THREADS=1` (and the BLAS equivalents) so the - measured time reflects the *algorithm*, not the core count. Do not remove this — the - speedup ratio is only stable run-to-run under single-thread pinning. +- **Python:** ≥ 3.10. **Install:** `pip install -r requirements.txt` (just NumPy). +- **Offline:** yes — no network, no services. +- **License:** MIT (code and the synthetic, seed-generated data); freely redistributable. +- **Threads:** `eval.sh` pins `OMP_NUM_THREADS=1` (and BLAS equivalents) so the measured + time reflects the algorithm, not the core count — do not remove it. -## Data source & license +## Baseline -There is **no external dataset**. Every problem instance is generated deterministically -at eval time from an integer seed via `numpy.random.default_rng(seed)` (standard-normal -point clouds). Nothing is downloaded, scraped, or redistributed. +The shipped `solution.py` is the **naive brute-force baseline**, identical to +`task.reference_solver`: full pairwise distances by broadcasting, then `argsort` and take the +first `k`. Correct but unoptimised, so its speedup over the reference is **~1.0x** by +construction — that's the number to beat. The headroom (partial selection via `argpartition`, +the `|x−y|² = |x|² − 2x·y + |y|²` GEMM expansion, dtype/blocking) is what makes the search +interesting. -- Code license: **MIT** (same as the Arbor repository). -- Data license: **MIT** — the data is synthetic and produced by the bundled `task.py`; - bundling is trivially allowed because there is nothing to redistribute beyond - generator code. - -The "data" is the generator: `task.generate_problem(seed, n_db, n_query, dim)` returns a -database/query pair; `eval.py` builds dev instances from seeds `1000..1002` and test -instances from `9000..9002`. No collection, labelling, or curation is involved — -re-running the generator reproduces the exact bytes. - -## Baseline implementation - -The shipped `solution.py` is the **naive brute-force baseline**, intentionally -identical to `task.reference_solver`: - -1. Compute the full pairwise distance matrix between every query and every database - point by broadcasting — `diff = queries[:, None, :] - database[None, :, :]`, then - `d2 = (diff * diff).sum(axis=2)` — an `O(n_query · n_db · dim)` operation. -2. `argsort` every row of `d2` in full and slice the first `k` columns. - -It is correct but deliberately unoptimised, so its speedup over the reference is **1.0x -by construction** — that is the number Arbor has to beat. The headroom (partial -selection via `argpartition`, the `|x−y|² = |x|² − 2x·y + |y|²` GEMM expansion, dtype -and blocking tweaks) is what makes the optimization loop interesting; none of it is -applied in the baseline. - -## Baseline reproduction - -- **Published baseline:** ~1.0x — the baseline equals the reference by construction. -- **Bundled baseline (measured):** `bash eval.sh dev` prints `score:` around - **0.99–1.01x**, fluctuating a couple of percent with timing noise on a quiet - single-thread CPU (the numerator and denominator run identical code). -- **Gap:** none. Any deviation is pure timing jitter, not an algorithmic difference. +**Results vary:** the score is a ratio of medians, so it wobbles a couple of percent +run-to-run and depends on the machine — the *ratio* is stable under single-thread pinning, +the absolute speedup is not. ## Contamination assessment -**No held-out leakage and no pre-training contamination.** - -- **Dev/test isolation:** dev seeds `1000..1002` and test seeds `9000..9002` are - disjoint integer ranges (see `eval.py` `DEV_SEED_BASE` / `TEST_SEED_BASE`). A solution - tuned on dev has never seen the test instances. -- **No web/pre-training contamination:** the instances are random point clouds generated - from seeds, not text or images from any corpus, so there is nothing a model could have - memorised. The metric is *wall-clock speedup of a computation*, not recall of an - answer — "fast but wrong" scores 0 via the independent `is_solution` gate, which - recomputes the ground truth. +No held-out leakage and no pre-training contamination. Dev seeds `1000..1002` and test seeds +`9000..9002` are disjoint ranges (see `DEV_SEED_BASE` / `TEST_SEED_BASE` in `eval.py`), so a +solution tuned on dev has never seen the test instances. The instances are random point +clouds generated from seeds — not text or images from any corpus — so there is nothing a +model could have memorised, and the metric is wall-clock speedup, not recall of an answer +(`is_solution` recomputes the ground truth, so "fast but wrong" scores 0). ## Caveats -- **Timing metric is hardware-dependent.** The absolute speedup depends on the machine; - the *ratio* `median(reference)/median(solution)` is what is compared, and it is stable - run-to-run only under single-thread pinning (enforced by `eval.sh`). -- **Determinism is not bit-exact.** Because the score is a ratio of medians, two runs - differ by a few percent; the verifier's `determinism` check is advisory for exactly - this reason — a human confirms the variation is timing noise, not a correctness bug. -- **Seed bases live in `eval.py`.** `DEV_SEED_BASE` / `TEST_SEED_BASE` in - [`eval.py`](eval.py) define the held-out split; if you change one, change both and - re-confirm disjointness here. +- **Timing metric is hardware-dependent** — compare the ratio, not the absolute number, and + keep the single-thread pinning. +- **Not bit-exact** — two runs differ by a few percent; that's timing noise, not a bug. +- **Seed bases live in `eval.py`** — if you change `DEV_SEED_BASE` / `TEST_SEED_BASE`, keep + dev and test disjoint. diff --git a/arbor-zoo/algotune_knn/README.md b/arbor-zoo/algotune_knn/README.md index 11e4314..4a2e135 100644 --- a/arbor-zoo/algotune_knn/README.md +++ b/arbor-zoo/algotune_knn/README.md @@ -1,89 +1,50 @@ ---- -name: algotune_knn -metric: - direction: maximize # higher speedup is better -eval: - cmd: "bash eval.sh" # the verifier appends `dev` / `test` -splits: - kind: seed_range # dev/test are disjoint seed windows (proved by verify) - dev: { base: 1000, count: 3 } # must match DEV_SEED_BASE in eval.py - test: { base: 9000, count: 3 } # must match TEST_SEED_BASE in eval.py -baseline: - score: 1.0 # the shipped solution equals the reference (~1.0x) - tolerance: 0.30 # relative margin for reproduce + determinism (timing noise) - kind: timing # determinism uses a ratio tolerance, not bit-equality -edit: [solution.py] # the only file Arbor may change; everything else is protected ---- - # algotune_knn -An AlgoTune-style mini benchmark for Arbor: make a brute-force **k-nearest-neighbours** -computation *faster* without changing what it computes. CPU-only, no API key, fully -deterministic, sub-second — the simplest possible end-to-end benchmark of the -"edit the baseline → run eval → keep what improves the held-out score" loop. +An AlgoTune-style mini benchmark: make a brute-force **k-nearest-neighbours** computation +*faster* without changing what it computes. CPU-only, no API key, deterministic, sub-second. -## Task & metric +## The task Given a database of points and a batch of queries, return each query's **k nearest -neighbours** (Euclidean distance). The reference implementation is intentionally -naive — full pairwise distances followed by a full sort. Your job is to compute the -**same** neighbours **faster**. +neighbours** (Euclidean distance). The reference implementation is intentionally naive — +full pairwise distances followed by a full sort. The goal is to compute the **same** +neighbours **faster**. + +## Metric -- **Edit surface:** `solution.py` (the `solve(problem)` function). The protected - ground truth and harness — `task.py`, `eval.py`, `eval.sh` — must not change. -- **Metric:** `bash eval.sh` prints one line `score: `, where - `speedup = median(reference_time) / median(solution_time)` over a held-out set of - instances, **after a correctness gate**. **Higher is better (maximize).** -- The shipped `solution.py` equals the reference, so the baseline is ~**1.0x**. A - solution that fails the independent `is_solution` check on any instance scores - **0.0** — "fast but wrong" cannot win. +`bash eval.sh dev|test` prints one `score:` line — the speedup +`median(reference_time) / median(solution_time)`, measured after a correctness gate. +**Higher is better.** The shipped `solution.py` equals the reference, so the baseline is +about **1.0x**; a solution that fails the correctness check scores **0.0**. -| File | Role | Editable? | -| --- | --- | --- | -| `solution.py` | The solver Arbor optimises (`solve`). | **Yes — the edit surface.** | -| `task.py` | Problem generator, reference solver, `is_solution` verifier. | No — protected. | -| `eval.py` | Correctness gate + median-of-N timing; prints `score:`. | No — protected. | -| `eval.sh` | Pins a single BLAS/OpenMP thread, then runs `eval.py`. | No — protected. | +## What Arbor may edit -## Run the baseline +`solution.py` (the `solve(problem)` function) is the editable baseline. The protected +harness — `task.py` (problem generator, reference solver, independent verifier), `eval.py`, +and `eval.sh` — must not change. -No setup beyond NumPy (see [`PROVENANCE.md`](PROVENANCE.md) → Setup & environment). -Dev and test use **disjoint seed ranges** (`1000+` vs `9000+`), so the signal you -iterate on is never the data you are finally judged on: +## Dev / test + +Dev and test use **disjoint seed ranges** (`1000+` vs `9000+`), so the signal you iterate +on is never the data you're finally judged on: ```bash -bash eval.sh dev # iterate here -> score: ~1.0 -bash eval.sh test # held-out gate -> score: ~1.0 +bash eval.sh dev # iterate here +bash eval.sh test # held-out gate ``` -Problem size is tunable via env vars for a heavier benchmark — e.g. -`KNN_N_DB=8000 KNN_N_QUERY=500 KNN_DIM=32 bash eval.sh dev`. - ## Optimize with Arbor -Arbor runs experiments in git worktrees off the repo root, so work from a **copy -outside the Arbor checkout**: +Copy this folder out of the Arbor checkout (it uses git worktrees), then run `arbor`: ```bash cp -r arbor-zoo/algotune_knn /tmp/algotune_knn -cd /tmp/algotune_knn -git init -q && git add -A && git commit -qm baseline -arbor # confirm the contract in the intake chat +cd /tmp/algotune_knn && git init -q && git add -A && git commit -qm baseline +arbor ``` -Suggested research contract: *maximize the `score:` from `bash eval.sh dev`; iterate -on dev, gate merges on `bash eval.sh test`; only edit `solution.py`; never touch -`task.py`, `eval.py`, or `eval.sh`; output must keep passing `is_solution`.* Real -headroom exists — `argpartition` partial selection, the -`|x−y|² = |x|² − 2x·y + |y|²` GEMM distance expansion, dtype and blocking tweaks — -so the idea tree has several genuine branches, not one trick. +There's real headroom — `argpartition` partial selection, the +`|x−y|² = |x|² − 2x·y + |y|²` GEMM distance expansion, dtype and blocking tweaks — so the +search has several genuine branches, not one trick. -## Provenance - -See [`PROVENANCE.md`](PROVENANCE.md) for setup & environment, how the baseline is -implemented, baseline reproduction, source, license, and the contamination -assessment. Verify this benchmark with: - -```bash -arbor benchmark verify arbor-zoo/algotune_knn -``` +See [`PROVENANCE.md`](PROVENANCE.md) for source, setup, and how the baseline works. diff --git a/docs/zoo-overview.md b/docs/zoo-overview.md index 9dacf7a..deb3e22 100644 --- a/docs/zoo-overview.md +++ b/docs/zoo-overview.md @@ -1,115 +1,56 @@ -# Benchmark Zoo — overview & roadmap +# Benchmark Zoo -This page is the **big picture** of the benchmark zoo: what it is, how we think about it, -what's built versus planned, and how you use it today. For the exact folder format and the -verifier's checks, see the [format reference](zoo.md). +Arbor works by pointing at a task that prints a score, then repeatedly editing the code, +running the eval, and keeping the changes that improve the score. The **benchmark zoo** is a +collection of such tasks — each packaged in one standard format, so you can grab one and let +Arbor start optimizing it right away. It lives in +[`arbor-zoo/`](https://github.com/RUC-NLPIR/Arbor/tree/main/arbor-zoo), one folder per benchmark. -## What it is +## What it's for -The **benchmark zoo** is a curated, verifiable set of optimization problems that Arbor can -be pointed at — packaged in one standard format so anyone can re-run them and check the -numbers. It lives in [`arbor-zoo/`](https://github.com/RUC-NLPIR/Arbor/tree/main/arbor-zoo), -one folder per benchmark. +- **Ready-made tasks to optimize.** Each benchmark ships an eval and a baseline — point + `arbor` at it and it starts improving the score. +- **Turn your own task into one.** Have code but no runnable eval yet? One command adds the + eval scaffolding so Arbor can run it. +- **Collect new ones (in progress).** Have Arbor fetch a benchmark from GitHub / HuggingFace + and shape it into the zoo format. -Two principles shape everything: +## What a benchmark looks like -- **Regression harness first, showcase second.** The zoo's first job is to keep Arbor - honestly tested on a small set of hand-checked, reproducible tasks. Being an outward - showcase is secondary. -- **Quality-capped, not a trophy board.** It is deliberately **not** a leaderboard of our - own results. We use representative work to *position a reusable, fairly-baselined task* — - never to catalogue how many papers we beat. +One folder with three things: -## The one idea to internalize +- a **README** — what the task is and which score to optimize; +- **baseline code** — Arbor's starting point, and the only part it's allowed to edit; +- an **eval script** — run it and it prints one `score:` line; it's protected, so Arbor can't + game it. -**A pack is not "a dataset" — it is a locked optimization problem:** +Arbor's loop is just: edit the baseline → run the eval → keep the change if the score went up, +and repeat. -> `data + frozen substrate + edit surface + metric + reference baseline` +## Entry points -The same dataset climbed from a different angle (tune the prompt vs. fine-tune the model vs. -design a scaffold) is a **different pack**. The *angle is the pack*, encoded in the README -front-matter (what's editable, what's frozen, what's measured). This is why "how you climb a -benchmark" is decided when the pack is authored, not at run time. +| What you want to do | Command | +| --- | --- | +| See what's in the zoo | `arbor benchmark list` | +| Optimize a benchmark with Arbor | copy it out, `git init`, then run `arbor` in it | +| Check a benchmark is valid | `arbor benchmark verify ` | +| Make your own code Arbor-ready | `arbor benchmark scaffold ` | +| Pull a benchmark from GitHub / HF | `arbor benchmark add --name ` | -## How we categorize (the mental map) +Running one, start to finish: -**Two layers.** +```bash +cp -r arbor-zoo/algotune_knn /tmp/algotune_knn # copy out of the Arbor checkout +cd /tmp/algotune_knn && git init -q && git add -A && git commit -qm baseline +arbor # confirm the task, then it iterates +``` -| Layer | What | Purpose | -| --- | --- | --- | -| **Stable core** | synthetic / self-contained tasks (e.g. `algotune_knn`) | Arbor's regression harness — stable, cheap, deterministic | -| **Frontier shelf** | tasks anchored on representative *hot work* | track the field; the "climb real benchmarks" use case | +## What's here, what's next -**The freeze axis** — every pack declares what it holds fixed, which decides what it measures: +- ✅ **Available now:** the format, `verify`, `list`, `scaffold`, the `add` spine, and the + first example benchmark, `algotune_knn`. +- ⏳ **In progress:** making `add` smarter (research the benchmark and bring its baseline up + automatically), and adding more benchmarks. -- **Freeze the model** (edit a scaffold/prompt) → measures the *method*; cheap, clean - attribution, narrow coverage. -- **Freeze a budget** (compute/wall-clock; edit training + scaffold + data) → *any* mechanism - competes on equal footing (the MLE-bench style); broad coverage, needs the compute. - -**Task shapes that fit** — anything where you optimize an *artifact* against a held-out score: -algorithm/efficiency (kernels, AlgoTune), ML-engineering/tabular (Kaggle/MLE-bench), -coding/agents (SWE-bench), prompt/reasoning scaffolds, and training-efficiency. **Out of -scope**: tasks that only *evaluate a frozen model* (raw multimodal QA) or need hardware/sim -(embodied robotics) — there is no artifact for Arbor to optimize. - -## What's built vs. planned - -| Capability | Command / artifact | Status | -| --- | --- | --- | -| Pack format + front-matter contract (incl. `frozen:`) | `arbor-zoo//` | ✅ shipped | -| The gate that admits a pack | `arbor benchmark verify` | ✅ shipped | -| Index the packs | `arbor benchmark list` | ✅ shipped | -| Make a local dir Arbor-ready / author a pack | `arbor benchmark scaffold` (+ MCP tool, intake wiring) | ✅ shipped | -| Acquire a remote benchmark + scaffold a draft | `arbor benchmark add` (git / HF, global cache) | ✅ spine shipped | -| First verified dogfood pack | `arbor-zoo/algotune_knn` | ✅ shipped | -| **Collection intelligence** — survey a direction/work → harvest baseline → bring it up | the agent stages behind `add` | ⏳ planned | -| More curated packs across task shapes | `arbor-zoo/…` | ⏳ ongoing | -| Browsable zoo / leaderboard view | — | ⏳ deferred | -| Optimizing across many benchmarks at once | — | 🔭 far-future | - -The internal design behind the collection feature lives in the dev notes -(`docs/dev/benchmark-add.md`) along with a researched -[backlog of collectable benchmarks](https://github.com/RUC-NLPIR/Arbor/blob/main/docs/dev/benchmark-backlog.md). - -## How you use it today - -There are four entry points, from "just run it" to "add a new one": - -1. **Run Arbor on an existing pack.** Copy a pack out of the checkout (Arbor uses git - worktrees), then point Arbor at it: - ```bash - cp -r arbor-zoo/algotune_knn /tmp/algotune_knn - cd /tmp/algotune_knn && git init -q && git add -A && git commit -qm baseline - arbor # confirm the contract, then it iterates - ``` -2. **Make your own task Arbor-ready.** If you have code but no runnable eval / split, scaffold - the measurement plumbing (it never writes the solution): - ```bash - arbor benchmark scaffold ./my_task --style light # eval + split + solution stub - arbor benchmark scaffold ./my_task --style zoo # + README contract + PROVENANCE - ``` -3. **Author a zoo pack.** Scaffold with `--style zoo`, fill in the baseline + eval + - PROVENANCE, then gate it: `arbor benchmark verify arbor-zoo/` until it exits 0, and a - maintainer accepts it. Drafting may be automated; **acceptance is a human step**. -4. **Collect a benchmark** (Phase 1 today). Acquire a remote benchmark into the global cache - and scaffold a draft to complete: - ```bash - arbor benchmark add https://github.com/owner/repo --name my_bench - ``` - -## The discipline (why a green check means something) - -A pack only counts when these hold — partly machine-enforced by `verify`, partly human-reviewed: - -- **Held-out**: dev/test are provably disjoint; merges gate on the protected test split. -- **Frozen substrate**: what's fixed is declared, so an improvement is attributable and two - runs are comparable (no winning by swapping in a bigger model). -- **Provenance + human acceptance**: source, baseline, and a mandatory contamination - assessment are written down and read by a maintainer — never auto-accepted. - -That's the whole shape: the zoo marks out *locked optimization problems*, Arbor runs its -normal loop inside them, and the verifier + provenance keep the results trustworthy. - -See the [format reference](zoo.md) to author one, or the -[roadmap](roadmap.md) for where the wider effort is going. +To author one, see the [format reference](zoo.md). For where this is heading, see the +[roadmap](roadmap.md). diff --git a/docs/zoo-overview.zh.md b/docs/zoo-overview.zh.md index cf528bb..6250934 100644 --- a/docs/zoo-overview.zh.md +++ b/docs/zoo-overview.zh.md @@ -1,103 +1,48 @@ -# 基准动物园 —— 总览与路线 +# 基准库（Benchmark Zoo） -这一页是基准动物园的**全景**:它是什么、我们怎么想、做了什么、还没做什么、现在怎么用。确切的 -文件夹格式和校验器检查见[格式参考](zoo.md)。 +Arbor 的工作方式很简单:对着一个能打分的任务,反复改代码、跑评测、把让分数变好的改动留下来。 +**基准库**就是一批这样的任务的集合——每个都打包成统一格式,拿来就能直接让 Arbor 上手优化。它放在 +仓库的 [`arbor-zoo/`](https://github.com/RUC-NLPIR/Arbor/tree/main/arbor-zoo) 下,一个文件夹一个基准。 -## 它是什么 +## 它用来干什么 -**基准动物园**是一组经过筛选、可校验的优化题,可以让 Arbor 对准它们——用同一套标准格式打包, -让任何人都能复跑并核对数字。它放在 -[`arbor-zoo/`](https://github.com/RUC-NLPIR/Arbor/tree/main/arbor-zoo) 里,每个基准一个文件夹。 +- **现成的任务,直接刷。** 库里每个基准都自带评测和一份基线,`arbor` 对准它就能开始优化、刷分。 +- **把你自己的任务也变成这样。** 有代码、但还不能评测?一条命令补上评测的架子,Arbor 就能跑了。 +- **自动收集(开发中)。** 让 Arbor 自己去 GitHub / HuggingFace 上把一个 benchmark 拉下来,整理成 + 库里的格式。 -两条原则贯穿始终: +## 一个基准长什么样 -- **回归 harness 优先,展示其次。** zoo 的首要职责是用一小批人工核对、可复跑的任务,让 Arbor - 始终被诚实地测着。对外展示是次要的。 -- **质量封顶,不做战绩榜。** 它**刻意不做**我们自己结果的排行榜。我们用代表性工作来*把一道 - 可复用、baseline 公平的任务定位准*——而不是去攒"我们超过了多少篇论文"。 +就是一个文件夹,里面三样东西: -## 要内化的那一个核心观念 +- 一份 **README**:这个任务是什么、要优化哪个分数; +- 一份 **基线代码**:Arbor 的优化起点,也是它唯一能改的部分; +- 一个 **评测脚本**:跑一下打印一行 `score:`;它受保护,Arbor 改不了它、也就没法作弊。 -**一个 pack 不是"一个数据集",而是一道锁死的优化题:** +Arbor 做的事就是一个循环:改基线 → 跑评测 → 分数涨了就留下,如此反复。 -> `数据 + 冻结底座 + 编辑面 + 指标 + 参照基线` +## 几个入口 -同一个数据集,从不同角度刷(调 prompt vs 微调模型 vs 设计 scaffold),就是**不同的 pack**。 -*角度即 pack*,编码在 README front-matter 里(什么可改、什么冻结、测什么)。所以"怎么刷一个 -benchmark"是在**写 pack 那一刻**决定的,不在运行时。 +| 你想做什么 | 命令 | +| --- | --- | +| 看库里有哪些基准 | `arbor benchmark list` | +| 在某个基准上跑 Arbor 优化 | 把它拷出仓库,`git init`,在目录里跑 `arbor` | +| 检查一个基准合不合格 | `arbor benchmark verify <目录>` | +| 把你自己的代码补成 Arbor 能跑的 | `arbor benchmark scaffold <目录>` | +| 从 GitHub / HF 拉一个 benchmark 进来 | `arbor benchmark add <地址> --name <名字>` | -## 我们的分类思路(心智地图) +完整跑一个的例子: -**两层结构。** +```bash +cp -r arbor-zoo/algotune_knn /tmp/algotune_knn # 拷出 Arbor 仓库 +cd /tmp/algotune_knn && git init -q && git add -A && git commit -qm baseline +arbor # 确认任务,然后它开始迭代 +``` -| 层 | 是什么 | 用途 | -| --- | --- | --- | -| **稳定核心层** | 合成/自包含任务(如 `algotune_knn`) | Arbor 的回归 harness——稳定、便宜、确定性 | -| **前沿货架层** | 锚定代表性*热门工作*的任务 | 追踪前沿;"刷真实 benchmark"的场景 | +## 现在有什么、还在做什么 -**冻结轴** —— 每个 pack 声明它固定什么,这决定了它测的是什么: +- ✅ **已经能用:** 基准格式、校验(`verify`)、列表(`list`)、补脚手架(`scaffold`)、收集的主体 + (`add`),以及第一个示例基准 `algotune_knn`。 +- ⏳ **还在做:** 让 `add` 更聪明(自动调研 benchmark、把基线跑通),以及加入更多基准。 -- **冻模型**(改 scaffold/prompt)→ 测*方法*;便宜、归因干净、覆盖窄。 -- **冻预算**(算力/wall-clock;改训练+scaffold+数据)→ *任意*机制平等竞争(MLE-bench 式); - 覆盖广,但要算力撑得住。 - -**适合的任务形态** —— 凡是"优化一个 artifact、对着 held-out 分数刷"的都适合:算法/效率(kernel、 -AlgoTune)、ML 工程/表格(Kaggle/MLE-bench)、代码/agent(SWE-bench)、prompt/推理 scaffold、 -训练效率。**不在范围内**:只*评测一个冻结模型*的(纯多模态 QA)、或要硬件/仿真的(具身机器人) -——那里没有可供 Arbor 优化的 artifact。 - -## 做了什么 vs 还没做 - -| 能力 | 命令 / 产物 | 状态 | -| --- | --- | --- | -| pack 格式 + front-matter 契约(含 `frozen:`) | `arbor-zoo//` | ✅ 已交付 | -| 准入门 | `arbor benchmark verify` | ✅ 已交付 | -| 索引 pack | `arbor benchmark list` | ✅ 已交付 | -| 把本地目录补成 Arbor-ready / 写一个 pack | `arbor benchmark scaffold`(+ MCP 工具、intake 接线) | ✅ 已交付 | -| 获取远端 benchmark + 起草草稿 | `arbor benchmark add`(git / HF,全局缓存) | ✅ 骨架已交付 | -| 第一个过校验的 dogfood pack | `arbor-zoo/algotune_knn` | ✅ 已交付 | -| **收集智能** —— 综述一个方向/工作 → 收割 baseline → 跑通 | `add` 背后的 agent 阶段 | ⏳ 计划中 | -| 跨形态的更多策展 pack | `arbor-zoo/…` | ⏳ 进行中 | -| 可浏览的 zoo / leaderboard 视图 | — | ⏳ 暂缓 | -| 同时对多个 benchmark 优化 | — | 🔭 远期 | - -收集功能背后的内部设计在 dev 笔记(`docs/dev/benchmark-add.md`)里,并附了一份调研过的 -[可收集 benchmark 待办清单](https://github.com/RUC-NLPIR/Arbor/blob/main/docs/dev/benchmark-backlog.md)。 - -## 现在怎么用 - -四个入口,从"直接跑"到"加一个新的": - -1. **在现有 pack 上跑 Arbor。** 把 pack 拷出检出目录(Arbor 用 git worktree),再让 Arbor 对准它: - ```bash - cp -r arbor-zoo/algotune_knn /tmp/algotune_knn - cd /tmp/algotune_knn && git init -q && git add -A && git commit -qm baseline - arbor # 确认契约,然后它开始迭代 - ``` -2. **把你自己的任务补成 Arbor-ready。** 如果你有代码但没有能跑的 eval / 划分,补上测量管线 - (它绝不写解法): - ```bash - arbor benchmark scaffold ./my_task --style light # eval + 划分 + solution 占位 - arbor benchmark scaffold ./my_task --style zoo # + README 契约 + PROVENANCE - ``` -3. **写一个 zoo pack。** 用 `--style zoo` 脚手架,填好 baseline + eval + PROVENANCE,然后过门: - 反复 `arbor benchmark verify arbor-zoo/` 直到退出 0,再由维护者接受。起草可自动, - **接受是人工步骤**。 -4. **收集一个 benchmark**(目前是 Phase 1)。把远端 benchmark 拉进全局缓存并起草草稿待补全: - ```bash - arbor benchmark add https://github.com/owner/repo --name my_bench - ``` - -## 纪律(为什么一个绿勾是有分量的) - -一个 pack 只有满足这些才算数——一部分由 `verify` 机器强制,一部分人工审核: - -- **留出**:dev/test 可证明不相交;merge 以受保护的 test 划分为门。 -- **冻结底座**:固定了什么是声明出来的,所以"提升"可归因、两次 run 可比(不能靠换更大模型取胜)。 -- **来源 + 人工接受**:来源、baseline、以及一份必填的污染评估,都写下来并由维护者读过——绝不 - 自动接受。 - -整体就是这个形状:zoo 划出一道道*锁死的优化题*,Arbor 在里面跑它本来的循环,校验器 + 来源卡片 -保证结果可信。 - -写一个 pack 见[格式参考](zoo.md),整条线的去向见[路线图](roadmap.md)。 +想自己写一个基准,详细格式见[格式参考](zoo.md);整条线的规划见[路线图](roadmap.md)。 diff --git a/docs/zoo.md b/docs/zoo.md index ceb1277..672c63d 100644 --- a/docs/zoo.md +++ b/docs/zoo.md @@ -1,156 +1,85 @@ -# Benchmark Zoo +# Benchmark Zoo — format reference -The **benchmark zoo** is a curated set of benchmarks packaged in one standard, -verifiable format so that anyone can re-run them with Arbor and check the numbers. -It lives in [`arbor-zoo/`](https://github.com/RUC-NLPIR/Arbor/tree/main/arbor-zoo), -one folder per benchmark. +This page is the exact format for a benchmark folder and what `arbor benchmark verify` +checks. For the big picture, see the [overview](zoo-overview.md). -The zoo is **quality-capped, not quantity-driven**. Its first job is to be Arbor's own -regression harness — a small set of hand-checked, reproducible tasks that exercise the -Coordinator/Executor loop — and only secondarily an outward showcase. It is deliberately -**not** a leaderboard of our own results. An unverified benchmark does not enter the zoo: -eval correctness is the bedrock, and a benchmark you cannot trust is worse than none. - -The format is **documentation-first with a tiny machine contract**. A benchmark is a -well-documented repo; the few facts a verifier and an unattended harness genuinely need — -and which prose cannot be checked against — live in a small YAML **front-matter** block at -the top of the README (no separate manifest file). Everything human stays in prose. +The format is **documentation-first**: a benchmark is a well-documented folder. The README +is plain prose that Arbor reads at intake — there is no YAML manifest to fill in. ## What a benchmark folder contains -Each `arbor-zoo//` is one self-contained benchmark holding four things — a guide -for the **user**, a description for **Arbor**, a runnable **baseline**, and a protected -**eval**: +Each `arbor-zoo//` holds four things: | File / dir | Role | For | | --- | --- | --- | -| `README.md` | A YAML **front-matter** contract (metric, splits, baseline, edit surface) + prose body in four fixed sections. | machine + user + Arbor | -| baseline code | The **baseline implementation** and Arbor's edit surface — `solution.py`, *or a whole set of files / a subdirectory*. | Arbor edits | -| `eval.sh` *or* `eval.py` | Protected eval entrypoint. `bash eval.sh dev\|test` (or `python eval.py --split …`) prints one line `score: `. | protected | -| `task.py` *(if used)* | Protected ground truth: problem generator, reference solver, independent verifier. | protected | -| `data/` | Bundled data; or a `download.sh` when the data may not be redistributed. | — | -| `PROVENANCE.md` | Source, setup & environment, license, baseline implementation, baseline reproduction, contamination, caveats. Seven fixed sections. | human review | - -The **baseline can be more than one file** — the `edit:` list in the front-matter names -the editable surface; everything else (the eval harness, ground-truth files, data) is the -protected remainder. - -Folders whose name begins with `_` (e.g. -[`_template`](https://github.com/RUC-NLPIR/Arbor/tree/main/arbor-zoo/_template)) are -scaffolding and are skipped by every tool. - -### `README.md` front-matter — the machine contract - -A small YAML block fenced by `---` at the very top of the README. It carries only the -facts that are not prose and that the verifier / an unattended run need: - -```yaml ---- -name: algotune_knn -metric: - direction: maximize # maximize | minimize -eval: - cmd: "bash eval.sh" # optional; omit to use the eval.sh/eval.py convention. - # the verifier appends `dev` / `test` -splits: # how dev/test differ — lets verify prove they're disjoint - kind: seed_range # seed_range | path - dev: { base: 1000, count: 3 } - test: { base: 9000, count: 3 } -baseline: - score: 1.0 # what `eval dev` prints today (verify checks reality matches) - tolerance: 0.30 # relative margin for reproduce + determinism - kind: timing # exact | timing (timing uses a ratio tolerance) -edit: [solution.py] # editable files/globs (1+); everything else is protected -frozen: # OPTIONAL — the freeze axis (what's held fixed for comparability) - model: gpt-x # freeze the model → measures the edited artifact, not a model swap - budget: "wall-clock 1h" # or freeze only a budget → any mechanism (train/scaffold) competes ---- -``` +| `README.md` | What the task is, the metric, what Arbor may edit, how dev/test differ — in plain language. | Arbor (and humans) | +| baseline code | The editable baseline — `solution.py`, or a whole set of files. | Arbor edits | +| `eval.sh` *or* `eval.py` | Protected eval entrypoint. `bash eval.sh dev\|test` (or `python eval.py --split …`) prints one `score: ` line. | protected | +| `PROVENANCE.md` | Source, setup, how the baseline works, contamination, caveats. | humans | +| `data/`, `task.py`, … | Any data / ground-truth the eval needs (protected). | — | -For a path-based split, write `splits: {kind: path, dev: ["data/dev/**"], test: ["data/test/**"]}`. -The same field names reuse the [plugin](plugins.md) vocabulary, so a verified benchmark -can be turned into a plugin with little rework. +Folders whose name starts with `_` (e.g. `_template`) are scaffolding and are skipped. -The optional **`frozen:`** field is the *freeze axis* — what a pack holds fixed so an -improvement is attributable and two runs are comparable. Freeze the **model** (`edit:` is a -scaffold/prompt) to measure the edited method; or freeze only a **budget** (compute/wall-clock, -with `edit:` spanning training + scaffold + data) to let *any* mechanism compete on equal -footing (MLE-bench style). Omit it for self-contained artifact-optimization tasks (e.g. -`algotune_knn`) that freeze nothing but the protected eval. +### `README.md` — the task, in plain language -### `README.md` body — four fixed sections +The README is what Arbor reads to understand the task, the same way its intake reads any +repo. Write it however reads best; a benchmark usually covers four things: -In this order, so every benchmark reads the same way: +1. **The task** — what it is and what a solution looks like. +2. **The metric** — what the eval prints (one `score:` line) and whether higher or lower is + better. +3. **What Arbor may edit** — the baseline file(s); everything else (the eval harness, + ground-truth, data) is off-limits. +4. **Dev / test** — how the two differ, so the held-out split is clear (disjoint seeds, or + `data/dev/` vs `data/test/`). -1. **Task & metric** — what the task is, what a solution looks like, the edit surface, - what is off-limits. -2. **Run the baseline** — the exact `eval` commands and what they print (point to - PROVENANCE → Setup & environment for install/hardware/keys). -3. **Optimize with Arbor** — how to point Arbor at the benchmark and the suggested - research contract. -4. **Provenance** — a pointer to `PROVENANCE.md`. +There is **no fixed baseline number** in the format: the same baseline gives different scores +on different hardware/models, so it's described in PROVENANCE, not pinned as a value. -### `PROVENANCE.md` — seven fixed sections +### `PROVENANCE.md` — the human card -`PROVENANCE.md` is the line between a trustworthy benchmark and one that merely looks the -part. Required headings: **Source**, **Setup & environment**, **Data source & license**, -**Baseline implementation**, **Baseline reproduction**, **Contamination assessment**, -**Caveats**. The verifier checks the headings are present; the *content* is read and -accepted by a human — never auto-accepted. +Required sections (the verifier checks they're present): **Source**, **Setup & environment**, +**Baseline**, **Contamination assessment**, **Caveats**. This is where source, license, how the +baseline works (and how much its score varies), and the held-out reasoning are written down for +a maintainer to read. -## Run Arbor on a benchmark +## What `arbor benchmark verify` checks -Arbor runs experiments in git worktrees off the repo root, so work from a copy -**outside** the Arbor checkout: +`verify` is a light **structural** check — a completeness lint, not a correctness gate. It does +**not run the eval** (a baseline's score isn't universal). It checks: + +- a `README.md` is present and non-empty; +- a `PROVENANCE.md` is present with its required sections; +- an eval entrypoint (`eval.sh` or `eval.py`) is present. ```bash -cp -r arbor-zoo/algotune_knn /tmp/algotune_knn -cd /tmp/algotune_knn -git init -q && git add -A && git commit -qm baseline -arbor # confirm the contract in the intake chat +arbor benchmark verify arbor-zoo/ # exits non-zero if a piece is missing +arbor benchmark list arbor-zoo # index the benchmarks ``` -## Verify a benchmark +Whether dev/test are *truly* held out and what the baseline really does are stated in +PROVENANCE prose and judged by a human — not machine-enforced. -```bash -arbor benchmark verify arbor-zoo/ # exits non-zero on any failure -arbor benchmark verify arbor-zoo/ --no-eval # structural checks only -arbor benchmark list arbor-zoo # plain index of benchmarks -``` +## Run Arbor on a benchmark -The checks prove what the contract makes provable and defer judgement calls to human -review: +Arbor runs experiments in git worktrees off the repo root, so work from a copy **outside** the +Arbor checkout: -| Check | Tier | What it proves | -| --- | --- | --- | -| `contract` | strong | README front-matter is present and its required fields are valid. | -| `readme-sections` | strong | README body has the four fixed sections. | -| `provenance` | strong | PROVENANCE has all seven headings (incl. baseline implementation + contamination). | -| `splits-disjoint` | strong | Dev/test are provably disjoint for the declared split mechanism. | -| `edit-surface` | strong | The declared editable files exist; the harness/ground-truth/data are not editable. | -| `eval-dev` / `eval-test` | strong | `eval dev` and `eval test` each run and print a parseable score. | -| `baseline-reproduces` | strong | The bundled baseline reproduces `baseline.score` within `tolerance`. | -| `determinism` | strong | Two dev runs agree (ratio tolerance for `kind: timing`, exact equality otherwise). | -| `contamination` | advisory | The contamination assessment is present; its content needs human acceptance. | - -What is still **left to human review** (the contract cannot prove intent): that the -contamination assessment is honest, and that the declared baseline/split reflect a real -held-out protocol rather than a convenient fiction. +```bash +cp -r arbor-zoo/algotune_knn /tmp/algotune_knn +cd /tmp/algotune_knn +git init -q && git add -A && git commit -qm baseline +arbor # Arbor reads the README, confirms the task, then iterates +``` ## Add a benchmark -1. Copy the scaffold: `cp -r arbor-zoo/_template arbor-zoo/`. -2. Fill in the front-matter contract + baseline (one or more code files), the protected - eval (`eval.sh` / `eval.py`, `task.py`), the four-section `README.md`, and the - seven-section `PROVENANCE.md`. -3. Run `arbor benchmark verify arbor-zoo/` until it exits 0. -4. A maintainer reviews the PROVENANCE card and accepts the benchmark. **Drafting may be - automated; acceptance is not** — and the agent that implements a baseline must be - separate from the loop that later optimizes it, or the evaluation is self-certifying. - -!!! note "Coming next" - Use `arbor benchmark scaffold --style zoo` to generate a fresh pack skeleton - (front-matter contract, eval stub, dev/test split, PROVENANCE card) and verify it - structurally. A future `arbor benchmark add ""` flow will - additionally *search for and draft* a benchmark from one you name, then hand it to the - verifier and to you for acceptance. +1. Scaffold the structure: `arbor benchmark scaffold arbor-zoo/ --style zoo`. This writes + an eval stub, a `solution.py` placeholder, a natural-language `README.md`, and a + `PROVENANCE.md` — never the solution itself. +2. Fill in the baseline (`solution.py`), the eval (`eval.py`/`eval.sh`), the README (for Arbor), + and PROVENANCE (for humans). +3. Run `arbor benchmark verify arbor-zoo/` until it exits 0, then a maintainer accepts it. + Drafting may be automated; **acceptance is a human step**. + +For an end-to-end example, see [`arbor-zoo/algotune_knn`](https://github.com/RUC-NLPIR/Arbor/tree/main/arbor-zoo/algotune_knn). diff --git a/docs/zoo.zh.md b/docs/zoo.zh.md index 5722c4f..876c923 100644 --- a/docs/zoo.zh.md +++ b/docs/zoo.zh.md @@ -1,133 +1,78 @@ -# 基准动物园（Benchmark Zoo） +# 基准库 —— 格式参考 -**基准动物园**是一组经过筛选、用同一套标准且可校验的格式打包的基准，让任何人都能用 Arbor 复跑并 -核对数字。它放在 -[`arbor-zoo/`](https://github.com/RUC-NLPIR/Arbor/tree/main/arbor-zoo) 里，每个基准一个文件夹。 +这一页讲一个基准文件夹的确切格式,以及 `arbor benchmark verify` 检查什么。整体介绍见 +[总览](zoo-overview.md)。 -zoo 是**质量封顶、不是数量驱动**。它的首要用途是 Arbor 自己的回归 harness——一小批人工核对、可复跑 -的任务，用来检验 Coordinator/Executor 循环——其次才是对外展示。它**刻意不做**我们自己战绩的榜单。 -未经校验的基准不进 zoo：eval 正确性是地基，一个你无法信任的基准比没有更糟。 - -格式是**文档优先 + 一个极小的机器契约**。一个基准就是一个文档齐全的 repo；那些体检工具和无人值守 -harness 真正需要、又没法靠 prose 核对的几项事实，写在 README 顶部一小段 YAML **front-matter** 里 -（**不是单独的清单文件**）。其余给人读的全部留在 prose 中。 +格式是**文档优先**的:一个基准就是一个文档齐全的文件夹。README 是 Arbor 在 intake 时读的自然语言 +说明——**没有要填的 YAML 清单**。 ## 一个基准文件夹包含什么 -每个 `arbor-zoo//` 是一个自包含基准，装着四样东西——给 **user** 的 guide、给 **Arbor** 的 -说明、一个可跑的 **baseline**、一个受保护的 **eval**： +每个 `arbor-zoo//` 里有四样东西: | 文件 / 目录 | 作用 | 面向 | | --- | --- | --- | -| `README.md` | 一段 YAML **front-matter** 契约（指标、splits、baseline、编辑面）+ 四个固定章节的 prose 正文。 | 机器 + user + Arbor | -| baseline 代码 | **baseline 实现**，也是 Arbor 的编辑面——`solution.py`，*或一整套文件 / 一个子目录*。 | Arbor 编辑 | -| `eval.sh` *或* `eval.py` | 受保护 eval 入口。`bash eval.sh dev\|test`（或 `python eval.py --split …`）打印一行 `score: `。 | 受保护 | -| `task.py`（如用到） | 受保护 ground truth：问题生成器、参考解、独立校验器。 | 受保护 | -| `data/` | 随包数据；不可再分发时放 `download.sh`。 | — | -| `PROVENANCE.md` | 来源、环境与配置、license、baseline 实现、baseline 复现、污染评估、注意事项。七个固定章节。 | 人工审核 | - -**baseline 可以不止一个文件**——front-matter 里的 `edit:` 列出编辑面；其余一切（eval harness、 -ground-truth 文件、数据）是受保护的其余部分。 - -以 `_` 开头的文件夹（如 -[`_template`](https://github.com/RUC-NLPIR/Arbor/tree/main/arbor-zoo/_template)）是脚手架，所有 -工具都会跳过。 - -### `README.md` front-matter —— 机器契约 - -README 最顶部一段用 `---` 围起来的小 YAML 块。只放那些非 prose、且体检/无人值守跑需要的事实： - -```yaml ---- -name: algotune_knn -metric: - direction: maximize # maximize | minimize -eval: - cmd: "bash eval.sh" # 可选；省略则用 eval.sh/eval.py 约定。体检会追加 dev / test -splits: # dev/test 怎么分——让体检能证明二者不相交 - kind: seed_range # seed_range | path - dev: { base: 1000, count: 3 } - test: { base: 9000, count: 3 } -baseline: - score: 1.0 # eval dev 现在打印的分数（体检核对实测是否相符） - tolerance: 0.30 # reproduce + determinism 的相对容差 - kind: timing # exact | timing（timing 用比值容差） -edit: [solution.py] # 可编辑文件/glob（≥1）；其余一切受保护 -frozen: # 可选 —— 冻结轴（为可比性固定什么） - model: gpt-x # 冻模型 → 测被编辑的方法，而非换模型 - budget: "wall-clock 1h" # 或只冻预算 → 任意机制（训练/scaffold）平等竞争 ---- -``` +| `README.md` | 任务是什么、指标、Arbor 能改什么、dev/test 怎么分——用自然语言写。 | Arbor(和人) | +| 基线代码 | 可编辑的基线——`solution.py`,或一整套文件。 | Arbor 编辑 | +| `eval.sh` *或* `eval.py` | 受保护 eval 入口。`bash eval.sh dev\|test`(或 `python eval.py --split …`)打印一行 `score: `。 | 受保护 | +| `PROVENANCE.md` | 来源、环境、baseline 怎么实现、污染评估、注意事项。 | 人看 | +| `data/`、`task.py`… | eval 需要的数据 / ground-truth(受保护)。 | — | -基于路径的 split 写成 `splits: {kind: path, dev: ["data/dev/**"], test: ["data/test/**"]}`。 -这些字段名复用[插件](plugins.md)词汇，所以一个通过校验的基准稍加改动即可变成 plugin。 +以 `_` 开头的文件夹(如 `_template`)是脚手架,会被跳过。 -可选的 **`frozen:`** 是*冻结轴*——一个 pack 固定什么，好让"提升"可归因、两次 run 可比。**冻模型** -（`edit:` 是 scaffold/prompt）测被编辑的方法；或只**冻预算**（算力/wall-clock，`edit:` 覆盖训练+ -scaffold+数据）让任意机制平等竞争（MLE-bench 式）。自包含的 artifact 优化任务（如 `algotune_knn`， -只冻受保护的 eval）可省略它。 +### `README.md` —— 用大白话写清任务 -### `README.md` 正文 —— 四个固定章节 +README 就是 Arbor 用来理解任务的东西,跟它 intake 读任何 repo 一样。怎么读着顺就怎么写;一个基准 +通常写清四件事: -按此顺序，让每个基准读起来都一样： +1. **任务** —— 是什么、一个解长什么样。 +2. **指标** —— eval 打印什么(一行 `score:`)、越大还是越小好。 +3. **Arbor 能改什么** —— 基线文件;其余一切(eval、ground-truth、数据)不许碰。 +4. **dev / test** —— 两者怎么分,让留出集清楚(不相交的种子,或 `data/dev/` vs `data/test/`)。 -1. **Task & metric** —— 任务是什么、一个解长什么样、编辑面、什么不许碰。 -2. **Run the baseline** —— eval 的确切命令及其输出（安装/硬件/key 指向 PROVENANCE → Setup & environment）。 -3. **Optimize with Arbor** —— 怎么把 Arbor 对准这个基准，以及建议的研究契约。 -4. **Provenance** —— 指向 `PROVENANCE.md`。 +格式里**没有固定的 baseline 数字**:同一份基线在不同硬件/模型上跑出来不一样,所以它写在 PROVENANCE +里,而不是钉成一个值。 -### `PROVENANCE.md` —— 七个固定章节 +### `PROVENANCE.md` —— 给人看的卡片 -`PROVENANCE.md` 是可信基准与"看起来像样"的基准之间的分界。必含标题：**Source**、 -**Setup & environment**、**Data source & license**、**Baseline implementation**、 -**Baseline reproduction**、**Contamination assessment**、**Caveats**。体检工具检查标题在场； -*内容*由人来读、由人来接受——绝不自动接受。 +必含章节(校验器会查在不在):**Source**、**Setup & environment**、**Baseline**、 +**Contamination assessment**、**Caveats**。来源、license、baseline 怎么实现的(以及分数波动多大)、 +留出集的理由,都写在这里给维护者读。 -## 用 Arbor 跑一个基准 +## `arbor benchmark verify` 查什么 + +`verify` 是个轻量的**结构**检查——查齐全,不查正确性,也**不跑 eval**(baseline 分数本就不通用)。 +它查: -Arbor 在仓库根的 git worktree 里跑实验，所以请在 Arbor 检出**之外**的副本里工作： +- `README.md` 在、且非空; +- `PROVENANCE.md` 在、且必含章节齐全; +- eval 入口(`eval.sh` 或 `eval.py`)在。 ```bash -cp -r arbor-zoo/algotune_knn /tmp/algotune_knn -cd /tmp/algotune_knn -git init -q && git add -A && git commit -qm baseline -arbor # 在接入对话里确认契约 +arbor benchmark verify arbor-zoo/ # 缺东西就非零退出 +arbor benchmark list arbor-zoo # 列出有哪些基准 ``` -## 校验一个基准 +dev/test 是否*真的*留出、baseline 到底干了什么,写在 PROVENANCE 文字里、由人判断——不做机器强制。 -```bash -arbor benchmark verify arbor-zoo/ # 任一项不过即非零退出 -arbor benchmark verify arbor-zoo/ --no-eval # 仅结构检查 -arbor benchmark list arbor-zoo # 基准的朴素索引 -``` +## 用 Arbor 跑一个基准 -检查项证明契约能证明的，把判断题留给人工审核： +Arbor 在仓库根的 git worktree 里跑实验,所以请在 Arbor 检出**之外**的副本里工作: -| 检查 | 档位 | 证明什么 | -| --- | --- | --- | -| `contract` | strong | README front-matter 在场且必填字段有效。 | -| `readme-sections` | strong | README 正文含四个固定章节。 | -| `provenance` | strong | PROVENANCE 七个标题齐全（含 baseline 实现 + 污染评估）。 | -| `splits-disjoint` | strong | 按声明的 split 机制，dev/test 可证明不相交。 | -| `edit-surface` | strong | 声明的可编辑文件存在；harness/ground-truth/数据不可编辑。 | -| `eval-dev` / `eval-test` | strong | `eval dev` 与 `eval test` 各能跑出可解析分数。 | -| `baseline-reproduces` | strong | 随包 baseline 在 `tolerance` 内复现 `baseline.score`。 | -| `determinism` | strong | 两次 dev 跑一致（`kind: timing` 用比值容差，否则要求相等）。 | -| `contamination` | advisory | 污染评估在场；内容需人工接受。 | - -**仍交给人工**（契约证明不了意图）的有：污染评估是否诚实，以及声明的 baseline/split 是否反映真实的 -留出协议，而非方便编出来的说法。 +```bash +cp -r arbor-zoo/algotune_knn /tmp/algotune_knn +cd /tmp/algotune_knn +git init -q && git add -A && git commit -qm baseline +arbor # Arbor 读 README、确认任务,然后开始迭代 +``` ## 新增一个基准 -1. 复制脚手架：`cp -r arbor-zoo/_template arbor-zoo/`。 -2. 填好 front-matter 契约 + baseline（一个或多个代码文件）、受保护 eval（`eval.sh` / `eval.py`、 - `task.py`）、四章节 `README.md`，以及七章节 `PROVENANCE.md`。 -3. 反复跑 `arbor benchmark verify arbor-zoo/` 直到退出 0。 -4. 维护者审阅 PROVENANCE 卡片并接受这个基准。**起草可自动，接受不可**——且实现 baseline 的 agent - 必须与之后优化它的 loop 分开，否则评测是自证的。 +1. 起架子:`arbor benchmark scaffold arbor-zoo/ --style zoo`。它写一个 eval 占位、 + `solution.py` 占位、自然语言 `README.md` 和 `PROVENANCE.md`——但绝不替你写解法。 +2. 填好基线(`solution.py`)、eval(`eval.py`/`eval.sh`)、README(给 Arbor 看)、PROVENANCE(给人看)。 +3. 反复 `arbor benchmark verify arbor-zoo/` 直到退出 0,再由维护者接受。起草可以自动, + **接受是人工这一步**。 -!!! note "下一步" - `arbor benchmark add "<论文 / repo / 数据集>"` 流程将让 Arbor 搜索、下载并*起草*一个你点名的 - 基准——再交给校验器和你来接受。本页的格式与校验器就是这个流程的落点。 +端到端的例子见 +[`arbor-zoo/algotune_knn`](https://github.com/RUC-NLPIR/Arbor/tree/main/arbor-zoo/algotune_knn)。 diff --git a/mkdocs.yml b/mkdocs.yml index a393138..7ed590f 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -121,7 +121,7 @@ plugins: Interaction Modes: 交互模式 Web UI & Monitoring: Web UI 与监控 Plugins: 插件 - Benchmark Zoo: 基准动物园 + Benchmark Zoo: 基准库 Overview: 概览 Format & verifier: 格式与校验器 Skills: 技能 diff --git a/src/cli/commands/benchmark_cmd.py b/src/cli/commands/benchmark_cmd.py index 0d31a54..f9961ed 100644 --- a/src/cli/commands/benchmark_cmd.py +++ b/src/cli/commands/benchmark_cmd.py @@ -2,8 +2,8 @@ Subcommands: -* ``arbor benchmark verify `` — run the gate that decides whether a pack - is allowed into the zoo. Exits non-zero if any check fails. +* ``arbor benchmark verify `` — structurally check a benchmark folder + (README + PROVENANCE + eval entrypoint present). Does not run the eval. * ``arbor benchmark list [zoo-dir]`` — print a plain index of packs (not a ranked leaderboard). * ``arbor benchmark scaffold `` — write the measurement plumbing (light) or a @@ -17,7 +17,6 @@ import subprocess from pathlib import Path -from typing import Any import typer @@ -25,7 +24,6 @@ VerifyResult, collect, discover_packs, - find_eval_entrypoint, select_acquirer, verify_pack, ) @@ -56,44 +54,26 @@ def verify_command( ..., help="Path to the benchmark directory (with README.md + eval.sh/eval.py).", ), - no_eval: bool = typer.Option( - False, "--no-eval", - help="Skip the eval-running checks (structural validation only).", - ), - timeout: int = typer.Option( - 600, "--timeout", - help="Per-eval timeout in seconds.", - ), ) -> None: - """Verify a Task Pack against the zoo format. Exits 1 if any check fails.""" + """Structurally verify a benchmark folder (does not run the eval). Exits 1 on failure.""" target = pack_dir.resolve() if not target.is_dir(): typer.secho(f"error: not a directory: {target}", fg=typer.colors.RED, err=True) raise typer.Exit(code=2) - if not (target / "README.md").exists() or find_eval_entrypoint(target) is None: - typer.secho( - f"error: {target} is not a benchmark (needs README.md + eval.sh/eval.py)", - fg=typer.colors.RED, err=True, - ) - raise typer.Exit(code=2) typer.echo(f"verifying {target.name} …") - results = verify_pack(target, run_eval=not no_eval, timeout=timeout) + results = verify_pack(target) for r in results: _render(r) fails = [r for r in results if r.status == "fail"] - warns = [r for r in results if r.status == "warn"] if fails: typer.secho( - f"\n{len(fails)} check(s) failed — pack does NOT enter the zoo.", + f"\n{len(fails)} check(s) failed — see the hints above.", fg=typer.colors.RED, err=True, ) raise typer.Exit(code=1) - typer.secho( - f"\nall checks passed ({len(warns)} advisory warning(s) — human review still required).", - fg=typer.colors.GREEN, - ) + typer.secho("\nlooks good — all structural checks pass.", fg=typer.colors.GREEN) @benchmark_app.command("list") @@ -123,7 +103,6 @@ def scaffold_command( style: str = typer.Option("light", "--style", help="light | zoo."), split_kind: str = typer.Option("seed_range", "--split-kind", help="seed_range | path."), entrypoint: str = typer.Option("eval.py", "--entrypoint", help="eval.py | eval.sh."), - baseline: float = typer.Option(0.0, "--baseline", help="Declared baseline score (zoo)."), edit: list[str] = typer.Option(["solution.py"], "--edit", help="Editable file/glob (repeatable)."), git_init: bool = typer.Option(False, "--git-init", help="git init + baseline commit."), ) -> None: @@ -132,23 +111,10 @@ def scaffold_command( target = target_dir.resolve() pack_name = name or target.name - splits: dict[str, Any] - if split_kind == "path": - splits = {"kind": "path", "dev": ["data/dev/**"], "test": ["data/test/**"]} - elif split_kind == "seed_range": - splits = {"kind": "seed_range", "dev": {"base": 1000, "count": 3}, - "test": {"base": 9000, "count": 3}} - else: - typer.secho( - f"error: --split-kind must be 'seed_range' or 'path', got {split_kind!r}", - fg=typer.colors.RED, err=True, - ) - raise typer.Exit(code=2) try: res = scaffold_benchmark( - target, name=pack_name, metric_direction=direction, splits=splits, - baseline={"score": baseline, "tolerance": 0.0, "kind": "exact"}, - edit=edit, style=style, eval_entrypoint=entrypoint, + target, name=pack_name, metric_direction=direction, style=style, + split_kind=split_kind, eval_entrypoint=entrypoint, edit=edit, ) except ValueError as exc: typer.secho(f"error: {exc}", fg=typer.colors.RED, err=True) diff --git a/src/mcp/server.py b/src/mcp/server.py index b2a9f81..cc5c4ba 100644 --- a/src/mcp/server.py +++ b/src/mcp/server.py @@ -140,28 +140,26 @@ def tree_set_meta( @server.tool() def scaffold_benchmark( name: str, - metric_direction: str, - splits: dict[str, Any], - baseline: dict[str, Any] | None = None, - edit: list[str] | None = None, - eval_cmd: str | None = None, + metric_direction: str = "maximize", style: str = "light", + split_kind: str = "seed_range", eval_entrypoint: str = "eval.py", + edit: list[str] | None = None, git_init: bool = False, cwd: str | None = None, ) -> dict[str, Any]: """Create the Arbor-ready reference folder in cwd. style: light (runnable eval + dev/test split + editable solution.py) | - zoo (adds the README front-matter contract + PROVENANCE.md and runs the - structural verifier). Writes measurement plumbing only — never the - solution. Idempotent: existing files are reported as skipped. Takes no - run_name: scaffolding precedes a session, so it operates on a directory. + zoo (adds a natural-language README + PROVENANCE.md and runs the structural + verifier). Writes measurement plumbing only — never the solution. + Idempotent: existing files are reported as skipped. Takes no run_name: + scaffolding precedes a session, so it operates on a directory. """ return ops.scaffold_benchmark( - _cwd(cwd), name=name, metric_direction=metric_direction, splits=splits, - baseline=baseline, edit=edit, eval_cmd=eval_cmd, style=style, - eval_entrypoint=eval_entrypoint, git_init=git_init, + _cwd(cwd), name=name, metric_direction=metric_direction, style=style, + split_kind=split_kind, eval_entrypoint=eval_entrypoint, edit=edit, + git_init=git_init, ) # ── Evaluation ─────────────────────────────────────────────────────────── diff --git a/src/mcp/session_ops.py b/src/mcp/session_ops.py index ef48da2..7942629 100644 --- a/src/mcp/session_ops.py +++ b/src/mcp/session_ops.py @@ -331,13 +331,11 @@ def scaffold_benchmark( cwd: str | Path, *, name: str, - metric_direction: str, - splits: dict[str, Any], - baseline: dict[str, Any] | None = None, - edit: list[str] | None = None, - eval_cmd: str | None = None, + metric_direction: str = "maximize", style: str = "light", + split_kind: str = "seed_range", eval_entrypoint: str = "eval.py", + edit: list[str] | None = None, git_init: bool = False, ) -> dict[str, Any]: """Create the Arbor reference folder under *cwd*; optionally git-init it. @@ -350,9 +348,8 @@ def scaffold_benchmark( target = Path(cwd) res = _scaffold( - target, name=name, metric_direction=metric_direction, splits=splits, - baseline=baseline, edit=edit, eval_cmd=eval_cmd, style=style, - eval_entrypoint=eval_entrypoint, + target, name=name, metric_direction=metric_direction, style=style, + split_kind=split_kind, eval_entrypoint=eval_entrypoint, edit=edit, ) committed = False if git_init and not (target / ".git").exists(): diff --git a/src/zoo/__init__.py b/src/zoo/__init__.py index 2c07bfe..4196ac9 100644 --- a/src/zoo/__init__.py +++ b/src/zoo/__init__.py @@ -1,10 +1,10 @@ """``arbor.zoo`` — the benchmark format, its verifier, scaffolder, and collection spine. -A benchmark is a self-contained directory under ``arbor-zoo/``: a README whose YAML -front-matter is a tiny machine contract and whose body is prose, a PROVENANCE card, a -runnable baseline (one or more code files), and a protected eval entrypoint. This package -provides pack discovery + the contract (:mod:`~arbor.zoo.pack`), the -``arbor benchmark verify`` gate (:mod:`~arbor.zoo.verify`), the scaffolder +A benchmark is a self-contained directory under ``arbor-zoo/``: a natural-language README +(the task description Arbor reads), a PROVENANCE card for humans, a runnable baseline (one +or more code files), and a protected eval entrypoint. The format is documentation-first — +no machine manifest. This package provides pack discovery (:mod:`~arbor.zoo.pack`), the +``arbor benchmark verify`` lint (:mod:`~arbor.zoo.verify`), the scaffolder (:mod:`~arbor.zoo.scaffold`), and the ``arbor benchmark add`` collection spine (:mod:`~arbor.zoo.acquire` / :mod:`~arbor.zoo.cache` / :mod:`~arbor.zoo.collect`). See ``docs/zoo.md``. @@ -17,13 +17,10 @@ from .collect import CollectResult, collect from .pack import ( EVAL_ENTRYPOINTS, - Contract, PackSummary, discover_packs, find_eval_entrypoint, is_pack_dir, - load_contract, - read_front_matter, ) from .scaffold import ScaffoldResult, scaffold_benchmark from .verify import VerifyResult, verify_pack @@ -33,7 +30,6 @@ "Acquired", "Acquirer", "CollectResult", - "Contract", "GitRepoAcquirer", "Manifest", "PackSummary", @@ -46,8 +42,6 @@ "discover_packs", "find_eval_entrypoint", "is_pack_dir", - "load_contract", - "read_front_matter", "scaffold_benchmark", "select_acquirer", "verify_pack", diff --git a/src/zoo/collect.py b/src/zoo/collect.py index 64a1d5a..8e2ccad 100644 --- a/src/zoo/collect.py +++ b/src/zoo/collect.py @@ -37,7 +37,7 @@ class CollectResult: def pending(self) -> list[str]: """Human/agent steps still required before the pack can be accepted.""" return [ - "Stage 0 survey: confirm canonical source, general baseline, angle + frozen substrate", + "Stage 0 survey: confirm the canonical source, the general baseline, and what Arbor edits", "Stage 2 bring-up: make the harvested baseline run; real eval.sh prints score: on dev/test", "Fill PROVENANCE (baseline implementation + contamination) and README sections", "Re-run `arbor benchmark verify` until green, then human-accept into arbor-zoo/", @@ -79,18 +79,12 @@ def collect( # ── Stage 3: scaffold the draft pack (reuses the zoo scaffolder) ───────── draft_dir = dest_root / name - scaffold_res = scaffold_benchmark( - draft_dir, name=name, metric_direction="maximize", - splits={"kind": "seed_range", - "dev": {"base": 1000, "count": 3}, - "test": {"base": 9000, "count": 3}}, - style="zoo", - ) + scaffold_res = scaffold_benchmark(draft_dir, name=name, style="zoo") result.draft_pack_dir = draft_dir result.notes.append( f"scaffolded draft pack at {draft_dir} ({len(scaffold_res.created)} files)") - # ── Stage 4 (structural): the zoo scaffolder already ran verify(run_eval=False) ── + # ── Stage 4 (structural): the zoo scaffolder already ran the verify lint ── result.verify_results = scaffold_res.verify result.ok = True return result diff --git a/src/zoo/pack.py b/src/zoo/pack.py index 6e293d4..500c7cf 100644 --- a/src/zoo/pack.py +++ b/src/zoo/pack.py @@ -1,33 +1,26 @@ -"""Pack discovery + the README front-matter *contract* for the ``arbor-zoo`` format. +"""Pack discovery for the ``arbor-zoo`` benchmark format. A *benchmark* is a directory under ``arbor-zoo/`` holding a self-contained task: -a ``README.md`` whose YAML **front-matter** is a tiny machine-readable contract and -whose body is human/agent prose, a ``PROVENANCE.md`` card, a runnable **baseline** -(one or more code files), and a protected eval entrypoint (``eval.sh``/``eval.py``) -that prints one ``score: `` line for ``dev`` and ``test``. - -There is no separate manifest file. The few facts a verifier and an unattended -harness genuinely need — and which prose cannot be checked against — live in the -README front-matter (metric direction, dev/test split, expected baseline, editable -surface). Everything human (setup, license, baseline write-up, contamination) lives -in prose in the README body and ``PROVENANCE.md``. See ``docs/zoo.md``. +a natural-language ``README.md`` (what the task is, which score to optimize, what +Arbor may edit — read by Arbor at intake), a ``PROVENANCE.md`` card for humans, a +runnable **baseline** (one or more code files), and a protected eval entrypoint +(``eval.sh``/``eval.py``) that prints one ``score: `` line. + +The format is **documentation-first**: there is no machine manifest. Discovery and +verification work by convention — see :mod:`arbor.zoo.verify` and ``docs/zoo.md``. """ from __future__ import annotations import logging -import re -from dataclasses import dataclass, field +from dataclasses import dataclass from pathlib import Path -from typing import Any log = logging.getLogger(__name__) # Eval entrypoints recognised by convention, in preference order. EVAL_ENTRYPOINTS = ("eval.sh", "eval.py") -_FRONT_MATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n?(.*)$", re.DOTALL) - @dataclass(frozen=True) class PackSummary: @@ -38,60 +31,6 @@ class PackSummary: path: str -@dataclass -class Contract: - """The README front-matter contract. Every field defaults empty so a partial - contract still loads; the verifier decides whether an omission is fatal.""" - - name: str = "" - metric: dict[str, Any] = field(default_factory=dict) # {direction} - eval: dict[str, Any] = field(default_factory=dict) # {cmd} (optional; convention otherwise) - splits: dict[str, Any] = field(default_factory=dict) # {kind, dev, test} - baseline: dict[str, Any] = field(default_factory=dict) # {score, tolerance, kind} - edit: list[str] = field(default_factory=list) # editable globs (1+); rest is protected - frozen: dict[str, Any] = field(default_factory=dict) # {model, budget} — the freeze axis (optional) - present: bool = False # was there front-matter at all? - - -def read_front_matter(md_path: Path) -> tuple[dict[str, Any] | None, str]: - """Split *md_path* into (front-matter dict | None, body). - - Front-matter is a leading ``---``-fenced YAML block. Returns ``(None, full_text)`` - when there is none. - """ - if not md_path.exists(): - return None, "" - text = md_path.read_text(encoding="utf-8") - m = _FRONT_MATTER_RE.match(text) - if not m: - return None, text - try: - import yaml - data = yaml.safe_load(m.group(1)) - except ImportError: - raise ImportError("PyYAML is required to read pack front-matter") - if not isinstance(data, dict): - return None, text - return data, m.group(2) - - -def load_contract(pack_dir: Path) -> Contract: - """Parse the README front-matter contract for *pack_dir* (empty if absent).""" - data, _ = read_front_matter(pack_dir / "README.md") - if data is None: - return Contract() - return Contract( - name=data.get("name", pack_dir.name), - metric=data.get("metric", {}) or {}, - eval=data.get("eval", {}) or {}, - splits=data.get("splits", {}) or {}, - baseline=data.get("baseline", {}) or {}, - edit=data.get("edit", []) or [], - frozen=data.get("frozen", {}) or {}, - present=True, - ) - - def find_eval_entrypoint(pack_dir: Path) -> str | None: """Return the eval entrypoint filename in *pack_dir*, or None if absent.""" for name in EVAL_ENTRYPOINTS: @@ -109,9 +48,21 @@ def is_pack_dir(path: Path) -> bool: def _readme_description(pack_dir: Path) -> str: - """First non-heading, non-blank line of the README body — a one-line description.""" - _, body = read_front_matter(pack_dir / "README.md") - for line in body.splitlines(): + """First non-heading, non-blank line of the README — a one-line description. + + Tolerates (and skips) a legacy leading ``---``-fenced block if one is present. + """ + readme = pack_dir / "README.md" + if not readme.exists(): + return "(no description)" + lines = readme.read_text(encoding="utf-8").splitlines() + i = 0 + if lines and lines[0].strip() == "---": + for j in range(1, len(lines)): + if lines[j].strip() == "---": + i = j + 1 + break + for line in lines[i:]: stripped = line.strip() if stripped and not stripped.startswith("#"): return stripped diff --git a/src/zoo/scaffold.py b/src/zoo/scaffold.py index 2c13204..99aed58 100644 --- a/src/zoo/scaffold.py +++ b/src/zoo/scaffold.py @@ -1,24 +1,19 @@ """``arbor.zoo.scaffold`` — create the reference benchmark folder structure. -Deterministic, keyless file writer. Given the contract facts the host model -decided during intake, it writes the *measurement plumbing* (eval entrypoint, -dev/test split layout, an editable ``solution.py`` placeholder) and, for the -``zoo`` style, the README front-matter contract + PROVENANCE card. It never -writes the solution logic itself. - -Idempotent and non-destructive: an existing file is recorded under ``skipped`` -and never overwritten. Templates are rendered in-package (not copied from -``arbor-zoo/_template``, which is not shipped in the wheel) so the front-matter -is guaranteed to round-trip through :func:`arbor.zoo.pack.read_front_matter`. +Deterministic, keyless file writer. It writes the *measurement plumbing* (an eval +entrypoint, a dev/test split layout, an editable ``solution.py`` placeholder) and, for +the ``zoo`` style, a natural-language ``README.md`` (the task description Arbor reads at +intake) + a ``PROVENANCE.md`` card for humans. It never writes the solution logic. + +The format is documentation-first: the README is plain prose (no YAML manifest), so the +scaffolded files are starting points for a human/agent to fill in. Idempotent and +non-destructive: an existing file is recorded under ``skipped`` and never overwritten. """ from __future__ import annotations from dataclasses import dataclass, field from pathlib import Path -from typing import Any - -import yaml from .pack import find_eval_entrypoint from .verify import VerifyResult, verify_pack @@ -26,6 +21,7 @@ _STYLES = ("light", "zoo") _ENTRYPOINTS = ("eval.py", "eval.sh") _DIRECTIONS = ("maximize", "minimize") +_SPLIT_KINDS = ("seed_range", "path") @dataclass @@ -40,7 +36,7 @@ class ScaffoldResult: # ── templates ────────────────────────────────────────────────────────────── -_SOLUTION = '''"""solution.py — the ONLY editable artifact (Arbor's edit surface). +_SOLUTION = '''"""solution.py — the editable baseline (Arbor's edit surface). Replace this with a working baseline. It must be correct first; Arbor then optimizes it to improve the score. Keep the entry-point signature stable — @@ -70,37 +66,30 @@ def solve(problem): _PROVENANCE = """# Provenance -All seven headings below are required. The verifier checks they are present; a -maintainer reads and accepts the content before the benchmark ships. +This card is for humans. Fill in every section; a maintainer reads it before the +benchmark is accepted. ## Source -Where the benchmark comes from — paper, repo, or competition, with a link, and -how the data was collected or generated. +Where the benchmark comes from — paper, repo, or competition, with a link, and how the +data was collected or generated. ## Setup & environment -Hardware (CPU / GPU), Python version, install command, env vars, and any API -keys, downloads, or services the user must provision. State whether it is offline. - -## Data source & license - -Where the data comes from, its license, and whether it may be redistributed. - -## Baseline implementation +Hardware (CPU / GPU), Python version, install command, env vars, and any API keys, +downloads, or services the user must provision. State whether it is offline. License of +the code and data, and whether the data may be redistributed. -How the shipped baseline works — the approach, why it scores what it does, and -what headroom it leaves for Arbor. +## Baseline -## Baseline reproduction - -The number `eval dev` prints today (must match `baseline.score` in the README -front-matter) and any gap from a published number. +How the shipped baseline works, and what score it tends to produce. **Results vary** by +user, hardware, and (for API tasks) model — note the range you saw rather than a single +fixed number. ## Contamination assessment -**Mandatory.** Could the test split be in a model's pre-training data? Is the -held-out split truly disjoint from dev? +**Mandatory.** Could the test split be in a model's pre-training data? Is the held-out +split truly disjoint from dev? Explain why a high score reflects real capability. ## Caveats @@ -108,38 +97,30 @@ def solve(problem): """ -def _eval_py(splits: dict[str, Any]) -> str: +def _eval_py(split_kind: str) -> str: """Render a protected eval.py stub for the declared split kind. - The stub catches NotImplementedError and prints ``score: 0.0`` so the metric - is always parseable before the baseline is filled in. + The stub catches NotImplementedError and prints ``score: 0.0`` so the metric is + parseable before the baseline is filled in. """ - if splits.get("kind") == "path": + if split_kind == "path": + head = "from pathlib import Path\n" body = ( " data_dir = Path(__file__).parent / \"data\" / split\n" " _instances = sorted(p for p in data_dir.glob(\"*\") if p.is_file())\n" " raise NotImplementedError(\"score solution.solve over the split's instances\")\n" ) - head = "from pathlib import Path\n" else: - dev = splits.get("dev", {}) or {} - test = splits.get("test", {}) or {} - head = ( - f"DEV_SEED_BASE = {int(dev.get('base', 1000))}\n" - f"TEST_SEED_BASE = {int(test.get('base', 9000))}\n" - f"DEV_COUNT = {int(dev.get('count', 3))}\n" - f"TEST_COUNT = {int(test.get('count', 3))}\n" - ) + head = "DEV_SEED_BASE = 1000\nTEST_SEED_BASE = 9000\nN_INSTANCES = 3\n" body = ( " base = DEV_SEED_BASE if split == \"dev\" else TEST_SEED_BASE\n" - " count = DEV_COUNT if split == \"dev\" else TEST_COUNT\n" - " _seeds = [base + i for i in range(count)]\n" + " _seeds = [base + i for i in range(N_INSTANCES)]\n" " raise NotImplementedError(\"compute the score from solution.solve\")\n" ) return ( '"""eval.py — PROTECTED evaluation harness. Do not edit during a research run.\n\n' - "Prints exactly one line ``score: `` for ``--split dev|test``. dev/test use\n" - "disjoint data; keep any constants in sync with the README front-matter ``splits:``.\n" + "Prints exactly one line ``score: `` for ``--split dev|test``. dev and test\n" + "must use disjoint data; describe the split in README / PROVENANCE.\n" '"""\n\n' "from __future__ import annotations\n\n" "import argparse\n" @@ -160,48 +141,47 @@ def _eval_py(splits: dict[str, Any]) -> str: ) -def _readme(name: str, direction: str, splits: dict, baseline: dict, - edit: list[str], eval_cmd: str | None, eval_entrypoint: str = "eval.py") -> str: - fm: dict[str, Any] = {"name": name, "metric": {"direction": direction}} - if eval_cmd: - fm["eval"] = {"cmd": eval_cmd} - fm["splits"] = splits - fm["baseline"] = baseline - fm["edit"] = edit - front = yaml.safe_dump(fm, sort_keys=False, default_flow_style=False) - if eval_entrypoint == "eval.sh": - run = ("bash eval.sh dev # iterate here\n" - "bash eval.sh test # held-out gate\n") - else: - run = ("python eval.py --split dev # iterate here\n" - "python eval.py --split test # held-out gate\n") - body = ( +def _readme(name: str, direction: str, edit: list[str], eval_entrypoint: str, + split_kind: str) -> str: + """Render a natural-language README — the task description Arbor reads at intake.""" + better = "higher is better" if direction == "maximize" else "lower is better" + editable = ", ".join(f"`{e}`" for e in edit) + run = ("bash eval.sh dev\nbash eval.sh test" + if eval_entrypoint == "eval.sh" + else "python eval.py --split dev\npython eval.py --split test") + split_note = ( + "dev and test use **disjoint seed ranges** so the held-out split is never the data " + "you tune on." if split_kind == "seed_range" else + "dev and test live in **separate folders** (`data/dev/`, `data/test/`); the test " + "split is held out." + ) + return ( f"# {name}\n\n" "One-line summary of the benchmark.\n\n" - "## Task & metric\n" - "What the task is, what a solution looks like, the edit surface, and what is " - "off-limits (the eval harness and any ground-truth files).\n\n" - "## Run the baseline\n" + "## The task\n" + "TODO — what the task is and what a solution looks like.\n\n" + "## Metric\n" + f"Running the eval prints one `score:` line; **{better}**.\n\n" + "## What Arbor may edit\n" + f"{editable} is the editable baseline. The eval harness and any ground-truth / data " + "are off-limits.\n\n" + "## Dev / test\n" + f"{split_note}\n\n" + "## Run it\n" "```bash\n" - f"{run}" - "```\n" - "Each prints one `score: ` line.\n\n" - "## Optimize with Arbor\n" - "Copy this folder out of the Arbor checkout (it uses git worktrees), then run " - "`arbor` and confirm the contract.\n\n" - "## Provenance\n" - "See [`PROVENANCE.md`](PROVENANCE.md).\n" + f"{run}\n" + "```\n\n" + "See [`PROVENANCE.md`](PROVENANCE.md) for source, setup, and the baseline write-up.\n" ) - return f"---\n{front}---\n\n{body}" -def _next_steps(style: str, res: ScaffoldResult) -> list[str]: +def _next_steps(style: str) -> list[str]: steps = [ "Fill in `solution.py` with the simplest correct baseline.", "Implement `evaluate()` in `eval.py` to score `solution.solve`.", ] if style == "zoo": - steps.append("Complete `PROVENANCE.md` (all seven sections) before submitting.") + steps.append("Complete `README.md` (the task for Arbor) and `PROVENANCE.md` (for humans).") steps.append("Run `arbor benchmark verify ` until it exits 0.") return steps @@ -210,13 +190,11 @@ def scaffold_benchmark( target: Path, *, name: str, - metric_direction: str, - splits: dict, - baseline: dict | None = None, - edit: list[str] | None = None, - eval_cmd: str | None = None, + metric_direction: str = "maximize", style: str = "light", + split_kind: str = "seed_range", eval_entrypoint: str = "eval.py", + edit: list[str] | None = None, ) -> ScaffoldResult: """Scaffold the Arbor reference folder under *target*. See module docstring.""" if style not in _STYLES: @@ -225,11 +203,12 @@ def scaffold_benchmark( raise ValueError(f"metric_direction must be one of {_DIRECTIONS}") if eval_entrypoint not in _ENTRYPOINTS: raise ValueError(f"eval_entrypoint must be one of {_ENTRYPOINTS}") + if split_kind not in _SPLIT_KINDS: + raise ValueError(f"split_kind must be one of {_SPLIT_KINDS}") target = Path(target) target.mkdir(parents=True, exist_ok=True) edit = edit or ["solution.py"] - baseline = baseline or {"score": 0.0, "tolerance": 0.0, "kind": "exact"} res = ScaffoldResult() def write(rel: str, content: str) -> None: @@ -238,35 +217,30 @@ def write(rel: str, content: str) -> None: res.skipped.append(rel) return p.parent.mkdir(parents=True, exist_ok=True) - # Force LF: a CRLF-translated eval.sh shebang ("…bash\r") is a broken - # interpreter on Unix, and the rest of the pack should stay LF too. + # Force LF: a CRLF-translated eval.sh shebang is a broken interpreter on Unix. p.write_text(content, encoding="utf-8", newline="\n") res.created.append(rel) existing = find_eval_entrypoint(target) if existing is None: - write("eval.py", _eval_py(splits)) + write("eval.py", _eval_py(split_kind)) if eval_entrypoint == "eval.sh": write("eval.sh", _EVAL_SH) else: res.skipped.append(existing) - if splits.get("kind") == "path": - # Visible placeholder instances (not hidden .gitkeep): globs like - # `data/dev/**` skip dotfiles, so empty .gitkeep dirs would fail the - # splits-disjoint check. One example per split keeps the pack - # structurally verifiable; the user replaces them with real data. + if split_kind == "path": + # Visible placeholder instances (globs skip dotfiles, so .gitkeep wouldn't show). write("data/dev/example_001.txt", "# replace with a real dev instance\n") write("data/test/example_001.txt", "# replace with a real held-out test instance\n") write("solution.py", _SOLUTION) if style == "zoo": - write("README.md", _readme(name, metric_direction, splits, baseline, edit, eval_cmd, - eval_entrypoint)) + write("README.md", _readme(name, metric_direction, edit, eval_entrypoint, split_kind)) write("PROVENANCE.md", _PROVENANCE) write("requirements.txt", "# add runtime dependencies here\n") - res.verify = verify_pack(target, run_eval=False) + res.verify = verify_pack(target) - res.next_steps = _next_steps(style, res) + res.next_steps = _next_steps(style) return res diff --git a/src/zoo/verify.py b/src/zoo/verify.py index 2a476be..320a813 100644 --- a/src/zoo/verify.py +++ b/src/zoo/verify.py @@ -1,57 +1,34 @@ -"""``arbor benchmark verify`` — the gate that decides whether a benchmark enters the zoo. +"""``arbor benchmark verify`` — a light structural lint for a benchmark folder. -The machine-checkable contract lives in the README **front-matter** (no separate -manifest file): metric direction, the dev/test split mechanism, the expected baseline, -and the editable surface. That restores the checks that matter most — that dev/test are -provably disjoint, that the baseline number is honest, and that the harness is protected -— while all human prose (setup, license, baseline write-up, contamination) stays in the -README body and ``PROVENANCE.md``. +The zoo format is documentation-first and natural-language: the README describes the +task for Arbor in plain prose (no machine manifest), and PROVENANCE is the human card. +So ``verify`` is a completeness check, **not** a correctness gate — it does not run the +eval (a baseline's score is not universal: it varies by user, hardware, and model). It +checks the pieces are present: a README, a PROVENANCE card with its sections, and an eval +entrypoint. Whether dev/test are truly held out and what the baseline really does are +stated in PROVENANCE prose and judged by a human. :func:`verify_pack` returns one :class:`VerifyResult` per check; the CLI exits non-zero if any has ``status == "fail"``. - -The score parser and shell runner mirror :func:`arbor.mcp.session_ops.parse_score` / -``_run_shell`` rather than importing them, to keep ``arbor.zoo`` dependency-light -(stdlib + PyYAML) and free of import cycles with the CLI. """ from __future__ import annotations -import fnmatch -import glob -import os -import re -import subprocess from dataclasses import dataclass from pathlib import Path -from typing import Any, Callable - -from .pack import Contract, find_eval_entrypoint, load_contract - -DEFAULT_EVAL_TIMEOUT = 600 +from typing import Callable -# Fixed README body sections (keyword → label), verified by heading presence. -README_SECTIONS = ( - ("task", "Task & metric"), - ("run", "Run the baseline"), - ("optimize", "Optimize with Arbor"), - ("provenance", "Provenance"), -) +from .pack import find_eval_entrypoint -# Required PROVENANCE.md headings. +# Required PROVENANCE.md headings (keyword → human label), verified by presence. PROVENANCE_HEADINGS = ( ("source", "Source"), ("setup", "Setup & environment"), - ("license", "Data source & license"), - ("implementation", "Baseline implementation"), - ("reproduction", "Baseline reproduction"), + ("baseline", "Baseline"), ("contamination", "Contamination assessment"), ("caveat", "Caveats"), ) -_SCORE_RE = re.compile(r"\bscore\s*[:=]\s*([-+]?\d+(?:\.\d+)?)", re.I) -_DIRECTIONS = {"maximize", "minimize"} - @dataclass class VerifyResult: @@ -64,324 +41,65 @@ class VerifyResult: hint: str | None = None -# ── small reused helpers (mirrors of session_ops) ───────────────────────────── - -def _parse_score(text: str) -> float | None: - matches = _SCORE_RE.findall(text) - return float(matches[-1]) if matches else None - - -def _run_shell(cmd: str, cwd: Path, timeout: int) -> tuple[int, str, bool]: - proc = subprocess.Popen( - cmd, cwd=str(cwd), shell=True, stdin=subprocess.DEVNULL, - stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, - env={**os.environ, "TERM": "dumb"}, - ) - try: - out, _ = proc.communicate(timeout=timeout) - return proc.returncode or 0, out, False - except subprocess.TimeoutExpired: - proc.kill() - out, _ = proc.communicate() - return -1, out + f"\n[timed out after {timeout}s]", True - - -def _eval_command(ctx: "_Ctx", split: str) -> str | None: - """Eval command for *split*: contract ``eval.cmd`` if set, else by convention.""" - cmd = ctx.contract.eval.get("cmd") - if cmd: - return f"{cmd.replace('{cwd}', str(ctx.pack_dir))} {split}" - entry = find_eval_entrypoint(ctx.pack_dir) - if entry == "eval.sh": - return f'bash "{ctx.pack_dir / "eval.sh"}" {split}' - if entry == "eval.py": - py = os.environ.get("PYTHON", "python3") - return f'{py} "{ctx.pack_dir / "eval.py"}" --split {split}' - return None - - -def _matches_any(path: str, patterns: list[str]) -> bool: - return any(fnmatch.fnmatch(path, pat) for pat in patterns) - - def _headings(md_text: str) -> list[str]: return [ln.lstrip("#").strip().lower() for ln in md_text.splitlines() if ln.lstrip().startswith("#")] -def _within_tol(a: float, b: float, tol: float) -> bool: - return abs(a - b) <= tol * max(abs(a), abs(b), 1e-9) - - -# ── verification context + checks ───────────────────────────────────────────── - -@dataclass -class _Ctx: - pack_dir: Path - contract: Contract - run_eval: bool - timeout: int - dev_score: float | None = None - dev_score2: float | None = None - test_score: float | None = None - dev_rc: int = 0 - test_rc: int = 0 - eval_ran: bool = False - - -def _check_contract(ctx: _Ctx) -> VerifyResult: - """Front-matter contract is present and its required fields are valid.""" - c = ctx.contract - if not c.present: - return VerifyResult( - "contract", "fail", "README has no front-matter contract", - "add a leading ---...--- YAML block — see docs/zoo.md", - ) - missing: list[str] = [] - if not c.name: - missing.append("name") - if c.metric.get("direction") not in _DIRECTIONS: - missing.append("metric.direction (maximize|minimize)") - if not c.splits.get("kind"): - missing.append("splits.kind") - if not c.splits.get("dev") or not c.splits.get("test"): - missing.append("splits.dev/test") - if not isinstance(c.baseline.get("score"), (int, float)): - missing.append("baseline.score") - if not c.edit: - missing.append("edit (editable files)") +def _check_files_present(pack_dir: Path) -> VerifyResult: + """README, PROVENANCE, and an eval entrypoint exist (the eval is not run).""" + missing = [f for f in ("README.md", "PROVENANCE.md") if not (pack_dir / f).exists()] + if find_eval_entrypoint(pack_dir) is None: + missing.append("eval.sh or eval.py") if missing: return VerifyResult( - "contract", "fail", - f"front-matter missing/invalid: {', '.join(missing)}", - "see the front-matter schema in docs/zoo.md", - ) - # `frozen:` is optional (the freeze axis); if present it must be a mapping. - if c.frozen and not isinstance(c.frozen, dict): - return VerifyResult( - "contract", "fail", "frozen: must be a mapping (e.g. {model: …, budget: …})", - "declare what is held fixed so improvement is attributable", + "files-present", "fail", + f"missing: {', '.join(missing)}", + "a benchmark needs a README, a PROVENANCE card, and an eval entrypoint", ) - note = f"; freeze: {', '.join(sorted(c.frozen))}" if isinstance(c.frozen, dict) and c.frozen else "" - return VerifyResult("contract", "pass", f"front-matter contract complete{note}") + return VerifyResult("files-present", "pass", "README, PROVENANCE, and eval entrypoint present") -def _check_readme_sections(ctx: _Ctx) -> VerifyResult: - _, body = _split_readme(ctx) - heads = _headings(body) - missing = [label for kw, label in README_SECTIONS if not any(kw in h for h in heads)] - if missing: - return VerifyResult("readme-sections", "fail", - f"README missing sections: {', '.join(missing)}", - "use the fixed section order from docs/zoo.md") - return VerifyResult("readme-sections", "pass", "all fixed README sections present") +def _check_readme(pack_dir: Path) -> VerifyResult: + """The README (Arbor's task description) exists and has some content.""" + readme = pack_dir / "README.md" + if not readme.exists(): + return VerifyResult("readme", "fail", "README.md missing") + if len(readme.read_text(encoding="utf-8").strip()) < 20: + return VerifyResult("readme", "fail", "README.md is essentially empty", + "describe the task, the metric, and what Arbor may edit") + return VerifyResult("readme", "pass", "README present") -def _check_provenance(ctx: _Ctx) -> VerifyResult: - prov = ctx.pack_dir / "PROVENANCE.md" +def _check_provenance(pack_dir: Path) -> VerifyResult: + """PROVENANCE.md exists and has its required headings (the human card).""" + prov = pack_dir / "PROVENANCE.md" if not prov.exists(): - return VerifyResult("provenance", "fail", "PROVENANCE.md missing", - "a benchmark without a provenance card is not trustworthy") + return VerifyResult("provenance", "fail", "PROVENANCE.md missing") heads = _headings(prov.read_text(encoding="utf-8")) missing = [label for kw, label in PROVENANCE_HEADINGS if not any(kw in h for h in heads)] if missing: return VerifyResult("provenance", "fail", f"PROVENANCE missing sections: {', '.join(missing)}", - "baseline implementation + contamination assessment are mandatory") + "the contamination assessment in particular is required") return VerifyResult("provenance", "pass", "all required PROVENANCE sections present") -def _check_splits_disjoint(ctx: _Ctx) -> VerifyResult: - """dev/test provably disjoint — reasons about the declared split mechanism.""" - splits = ctx.contract.splits - kind = splits.get("kind") - dev, test = splits.get("dev"), splits.get("test") - if kind == "seed_range": - if not isinstance(dev, dict) or not isinstance(test, dict): - return VerifyResult("splits-disjoint", "fail", - "seed_range dev/test must be mappings with base+count") - try: - dev_set = set(range(int(dev["base"]), int(dev["base"]) + int(dev["count"]))) - test_set = set(range(int(test["base"]), int(test["base"]) + int(test["count"]))) - except (KeyError, TypeError, ValueError) as exc: - return VerifyResult("splits-disjoint", "fail", f"seed_range malformed: {exc}") - overlap = dev_set & test_set - if overlap: - return VerifyResult("splits-disjoint", "fail", - f"dev/test seed ranges overlap on {sorted(overlap)}", - "held-out test must not share seeds with dev") - return VerifyResult("splits-disjoint", "pass", - f"seed ranges disjoint (dev {len(dev_set)}, test {len(test_set)})") - if kind == "path": - def expand(globs: Any) -> set[str]: - out: set[str] = set() - for g in globs or []: - for m in glob.glob(g, root_dir=str(ctx.pack_dir), recursive=True): - if (ctx.pack_dir / m).is_file(): - out.add(m) - return out - dev_files, test_files = expand(dev), expand(test) - path_overlap = dev_files & test_files - if path_overlap: - return VerifyResult("splits-disjoint", "fail", - f"dev/test path globs share files: {sorted(path_overlap)[:5]}") - if not dev_files or not test_files: - return VerifyResult("splits-disjoint", "fail", - "path splits matched no files on one side") - return VerifyResult("splits-disjoint", "pass", - f"path splits disjoint (dev {len(dev_files)}, test {len(test_files)})") - return VerifyResult("splits-disjoint", "fail", f"unknown splits.kind: {kind!r}", - "splits.kind must be 'seed_range' or 'path'") - - -def _check_edit_surface(ctx: _Ctx) -> VerifyResult: - """The declared editable files exist; the harness/data are not editable.""" - patterns = ctx.contract.edit - # Every edit glob must match at least one existing file. - unmatched = [ - g for g in patterns - if not glob.glob(g, root_dir=str(ctx.pack_dir), recursive=True) - ] - if unmatched: - return VerifyResult("edit-surface", "fail", - f"edit patterns match no files: {', '.join(unmatched)}", - "list the baseline file(s) Arbor may change") - # The harness and data must NOT be editable. - must_protect = [f for f in ("eval.sh", "eval.py", "task.py") if (ctx.pack_dir / f).exists()] - if (ctx.pack_dir / "data").is_dir(): - must_protect.append("data/_probe") - leaked = [f for f in must_protect if _matches_any(f, patterns)] - if leaked: - return VerifyResult("edit-surface", "fail", - f"editable surface includes protected files: {', '.join(leaked)}", - "the eval harness, ground truth, and data must stay protected") - return VerifyResult("edit-surface", "pass", "editable surface declared; harness protected") - - -def _run_evals(ctx: _Ctx) -> None: - if not ctx.run_eval: - return - dev_cmd, test_cmd = _eval_command(ctx, "dev"), _eval_command(ctx, "test") - if not dev_cmd or not test_cmd: - return - ctx.eval_ran = True - ctx.dev_rc, dev_out, _ = _run_shell(dev_cmd, ctx.pack_dir, ctx.timeout) - ctx.dev_score = _parse_score(dev_out) - _, out2, _ = _run_shell(dev_cmd, ctx.pack_dir, ctx.timeout) - ctx.dev_score2 = _parse_score(out2) - ctx.test_rc, test_out, _ = _run_shell(test_cmd, ctx.pack_dir, ctx.timeout) - ctx.test_score = _parse_score(test_out) - - -def _check_eval_dev(ctx: _Ctx) -> VerifyResult: - if not ctx.eval_ran: - return VerifyResult("eval-dev", "warn", "eval not run (run_eval=False)") - if ctx.dev_rc != 0 or ctx.dev_score is None: - return VerifyResult("eval-dev", "fail", f"eval dev rc={ctx.dev_rc}, score={ctx.dev_score}", - "eval must exit 0 and print one `score: ` line") - return VerifyResult("eval-dev", "pass", f"dev score parsed: {ctx.dev_score}") - - -def _check_eval_test(ctx: _Ctx) -> VerifyResult: - if not ctx.eval_ran: - return VerifyResult("eval-test", "warn", "eval not run (run_eval=False)") - if ctx.test_rc != 0 or ctx.test_score is None: - return VerifyResult("eval-test", "fail", f"eval test rc={ctx.test_rc}, score={ctx.test_score}", - "the held-out test split must also print a parseable score") - return VerifyResult("eval-test", "pass", f"test score parsed: {ctx.test_score}") - - -def _check_baseline(ctx: _Ctx) -> VerifyResult: - """The bundled baseline reproduces the declared score within tolerance.""" - if not ctx.eval_ran: - return VerifyResult("baseline-reproduces", "warn", "eval not run (run_eval=False)") - if ctx.dev_score is None: - return VerifyResult("baseline-reproduces", "fail", "no dev score to compare") - declared = ctx.contract.baseline.get("score") - if not isinstance(declared, (int, float)): - return VerifyResult("baseline-reproduces", "fail", f"baseline.score not numeric: {declared!r}") - tol = float(ctx.contract.baseline.get("tolerance", 0.0)) - if not _within_tol(ctx.dev_score, float(declared), tol): - return VerifyResult("baseline-reproduces", "fail", - f"dev score {ctx.dev_score} not within {tol:.0%} of declared {declared}", - "the declared baseline must match what the bundled solution prints") - return VerifyResult("baseline-reproduces", "pass", - f"dev score {ctx.dev_score} matches declared baseline {declared} (±{tol:.0%})") - - -def _check_determinism(ctx: _Ctx) -> VerifyResult: - """Two dev runs agree. Timing metrics use tolerance + a >0 correctness invariant; - other metrics require exact equality.""" - if not ctx.eval_ran: - return VerifyResult("determinism", "warn", "eval not run (run_eval=False)") - s1, s2 = ctx.dev_score, ctx.dev_score2 - if s1 is None or s2 is None: - return VerifyResult("determinism", "fail", f"a dev run produced no score ({s1}, {s2})") - if ctx.contract.baseline.get("kind") == "timing": - if s1 <= 0 or s2 <= 0: - return VerifyResult("determinism", "fail", - f"a dev run failed the correctness gate (scores {s1}, {s2})") - tol = float(ctx.contract.baseline.get("tolerance", 0.0)) - if not _within_tol(s1, s2, tol): - return VerifyResult("determinism", "fail", - f"dev scores diverge beyond {tol:.0%}: {s1} vs {s2}", - "widen baseline.tolerance or pin threads if this is timing noise") - return VerifyResult("determinism", "pass", f"two timing runs agree ({s1} ≈ {s2})") - if s1 != s2: - return VerifyResult("determinism", "fail", f"non-timing metric not reproducible: {s1} vs {s2}", - "a deterministic eval must return the same score every run") - return VerifyResult("determinism", "pass", f"two runs identical ({s1})") - - -def _check_contamination(ctx: _Ctx) -> VerifyResult: - return VerifyResult("contamination", "warn", - "contamination assessment present — requires human acceptance, never auto-accepted", - "a maintainer must confirm the test set isn't pre-trained on") - - -# README body is parsed once and cached on the context via a module helper. -def _split_readme(ctx: _Ctx) -> tuple[dict[str, Any] | None, str]: - from .pack import read_front_matter - return read_front_matter(ctx.pack_dir / "README.md") - - -_STRUCTURAL_CHECKS: tuple[Callable[[_Ctx], VerifyResult], ...] = ( - _check_contract, - _check_readme_sections, +_CHECKS: tuple[Callable[[Path], VerifyResult], ...] = ( + _check_files_present, + _check_readme, _check_provenance, - _check_splits_disjoint, - _check_edit_surface, - _check_contamination, ) -_EVAL_CHECKS: tuple[Callable[[_Ctx], VerifyResult], ...] = ( - _check_eval_dev, - _check_eval_test, - _check_baseline, - _check_determinism, -) - -def verify_pack( - pack_dir: Path, - *, - run_eval: bool = True, - timeout: int = DEFAULT_EVAL_TIMEOUT, -) -> list[VerifyResult]: - """Verify the benchmark in *pack_dir*. Returns one :class:`VerifyResult` per check. - Set ``run_eval=False`` to skip the four eval-running checks (they become ``warn``). - """ +def verify_pack(pack_dir: Path) -> list[VerifyResult]: + """Structurally verify the benchmark in *pack_dir* (does not run the eval).""" pack_dir = pack_dir.resolve() - ctx = _Ctx(pack_dir=pack_dir, contract=load_contract(pack_dir), - run_eval=run_eval, timeout=timeout) - def _safe(check: Callable[[_Ctx], VerifyResult]) -> VerifyResult: + def _safe(check: Callable[[Path], VerifyResult]) -> VerifyResult: try: - return check(ctx) + return check(pack_dir) except Exception as exc: # noqa: BLE001 — a check bug must not crash verify return VerifyResult(check.__name__, "fail", f"check raised: {exc}") - results = [_safe(c) for c in _STRUCTURAL_CHECKS] - _run_evals(ctx) - results += [_safe(c) for c in _EVAL_CHECKS] - return results + return [_safe(c) for c in _CHECKS] diff --git a/tests/test_benchmark_cli.py b/tests/test_benchmark_cli.py index 2a4ca9e..1c532dd 100644 --- a/tests/test_benchmark_cli.py +++ b/tests/test_benchmark_cli.py @@ -29,7 +29,7 @@ def test_scaffold_zoo_renders_verify_rows(tmp_path: Path) -> None: assert result.exit_code == 0, result.output assert (dest / "README.md").exists() assert (dest / "PROVENANCE.md").exists() - assert "contract" in result.output # a verify row name was rendered + assert "provenance" in result.output # a verify row name was rendered def test_scaffold_rejects_invalid_split_kind(tmp_path: Path) -> None: diff --git a/tests/test_mcp_session_ops.py b/tests/test_mcp_session_ops.py index de2d552..872e84d 100644 --- a/tests/test_mcp_session_ops.py +++ b/tests/test_mcp_session_ops.py @@ -307,18 +307,9 @@ def test_worktree_remove_refuses_paths_outside_scratch(tmp_path: Path) -> None: # ── scaffold_benchmark ──────────────────────────────────────────────────────── -_SCAFFOLD_SPLITS = { - "kind": "seed_range", - "dev": {"base": 1000, "count": 3}, - "test": {"base": 9000, "count": 3}, -} - def test_scaffold_benchmark_light(tmp_path): - out = ops.scaffold_benchmark( - tmp_path, name="demo", metric_direction="maximize", - splits=_SCAFFOLD_SPLITS, style="light", - ) + out = ops.scaffold_benchmark(tmp_path, name="demo", style="light") assert "eval.py" in out["created"] assert out["verify"] == [] assert out["git_committed"] is False @@ -326,10 +317,7 @@ def test_scaffold_benchmark_light(tmp_path): def test_scaffold_benchmark_zoo_with_git_init(tmp_path): - out = ops.scaffold_benchmark( - tmp_path, name="demo", metric_direction="maximize", - splits=_SCAFFOLD_SPLITS, style="zoo", git_init=True, - ) + out = ops.scaffold_benchmark(tmp_path, name="demo", style="zoo", git_init=True) assert out["git_committed"] is True assert (tmp_path / ".git").exists() assert all(r["status"] != "fail" for r in out["verify"]) diff --git a/tests/test_zoo_algotune_knn.py b/tests/test_zoo_algotune_knn.py index 89512d0..39d1287 100644 --- a/tests/test_zoo_algotune_knn.py +++ b/tests/test_zoo_algotune_knn.py @@ -1,14 +1,10 @@ -"""End-to-end: the shipped ``arbor-zoo`` packs verify clean. +"""End-to-end: the shipped ``arbor-zoo`` packs pass the structural verify lint. -Runs the real verifier (which executes each pack's ``eval.sh``) against the packs -checked into the repo, plus the CLI exit-code contract. Skipped where numpy is -unavailable, since the algotune_knn eval needs it. +``verify`` does not run the eval, so this needs no numpy and is not timing-sensitive. """ from __future__ import annotations -import shutil -import subprocess from pathlib import Path import pytest @@ -20,58 +16,31 @@ _REPO_ROOT = Path(__file__).resolve().parent.parent _ZOO = _REPO_ROOT / "arbor-zoo" - -def _eval_python_has_numpy() -> bool: - """The pack's eval.sh runs under ``python3`` (not the test runner), so probe - that interpreter — numpy may be present there even when the runner lacks it.""" - py = shutil.which("python3") or shutil.which("python") - if not py: - return False - return subprocess.run([py, "-c", "import numpy"], capture_output=True).returncode == 0 - - -requires_numpy = pytest.mark.skipif( - not _eval_python_has_numpy(), - reason="zoo eval needs numpy in python3", -) requires_zoo = pytest.mark.skipif(not _ZOO.is_dir(), reason="arbor-zoo not present") -@pytest.fixture(autouse=True) -def _stable_knn_timing(monkeypatch): - """Raise the median-of-N timing reps + instances so the speedup ratio is stable - under CI load (the timing metric is otherwise noisy run-to-run).""" - monkeypatch.setenv("KNN_TRIALS", "9") - monkeypatch.setenv("KNN_INSTANCES", "5") - - @requires_zoo -@requires_numpy def test_algotune_knn_verifies_clean() -> None: - results = verify_pack(_ZOO / "algotune_knn") - fails = [(r.name, r.message) for r in results if r.status == "fail"] - assert fails == [], f"algotune_knn should verify clean, got fails: {fails}" + fails = [(r.name, r.message) for r in verify_pack(_ZOO / "algotune_knn") if r.status == "fail"] + assert fails == [], f"algotune_knn should verify clean, got: {fails}" @requires_zoo -@requires_numpy def test_every_shipped_pack_verifies() -> None: - """The CI gate: no unverified pack enters the zoo.""" packs = discover_packs(_ZOO) assert packs, "expected at least one pack in arbor-zoo" for summary in packs: - results = verify_pack(Path(summary.path)) - fails = [(r.name, r.message) for r in results if r.status == "fail"] + fails = [(r.name, r.message) for r in verify_pack(Path(summary.path)) if r.status == "fail"] assert fails == [], f"pack {summary.name} has failures: {fails}" @requires_zoo -@requires_numpy def test_cli_verify_exit_code_zero() -> None: result = CliRunner().invoke(app, ["benchmark", "verify", str(_ZOO / "algotune_knn")]) assert result.exit_code == 0, result.output -def test_cli_verify_missing_pack_errors(tmp_path: Path) -> None: +def test_cli_verify_incomplete_pack_fails(tmp_path: Path) -> None: + # an empty dir: verify runs (no eval) and reports the missing pieces → exit 1 result = CliRunner().invoke(app, ["benchmark", "verify", str(tmp_path)]) - assert result.exit_code == 2 # not a benchmark dir (no README/eval here) + assert result.exit_code == 1 diff --git a/tests/test_zoo_pack.py b/tests/test_zoo_pack.py index 290dbcf..6047a3e 100644 --- a/tests/test_zoo_pack.py +++ b/tests/test_zoo_pack.py @@ -1,32 +1,14 @@ -"""Tests for zoo pack discovery + front-matter contract (``arbor.zoo.pack``).""" +"""Tests for zoo pack discovery (``arbor.zoo.pack``).""" from __future__ import annotations from pathlib import Path -from arbor.zoo import ( - discover_packs, - find_eval_entrypoint, - is_pack_dir, - load_contract, - read_front_matter, -) - -_FM = """\ ---- -name: demo -metric: {direction: maximize} -splits: {kind: seed_range, dev: {base: 1000, count: 3}, test: {base: 9000, count: 3}} -baseline: {score: 1.0, tolerance: 0.1, kind: timing} -edit: [solution.py] ---- -# title - -A demo benchmark. -""" - - -def _make_pack(zoo: Path, name: str, *, readme: str = _FM, entry: str = "eval.sh") -> Path: +from arbor.zoo import discover_packs, find_eval_entrypoint, is_pack_dir + + +def _make_pack(zoo: Path, name: str, *, readme: str = "# title\n\nA demo benchmark.\n", + entry: str = "eval.sh") -> Path: pack = zoo / name pack.mkdir(parents=True, exist_ok=True) (pack / "README.md").write_text(readme, encoding="utf-8") @@ -55,57 +37,23 @@ def test_find_eval_entrypoint_prefers_sh(tmp_path: Path) -> None: assert find_eval_entrypoint(pack) == "eval.sh" -def test_read_front_matter(tmp_path: Path) -> None: - md = tmp_path / "README.md" - md.write_text(_FM) - fm, body = read_front_matter(md) - assert fm is not None and fm["name"] == "demo" - assert "A demo benchmark." in body - assert "---" not in body # front-matter stripped from the body - - -def test_read_front_matter_absent(tmp_path: Path) -> None: - md = tmp_path / "README.md" - md.write_text("# no front matter\n\nbody") - fm, body = read_front_matter(md) - assert fm is None and "body" in body - - -def test_load_contract(tmp_path: Path) -> None: - pack = _make_pack(tmp_path, "demo") - c = load_contract(pack) - assert c.present - assert c.metric["direction"] == "maximize" - assert c.splits["kind"] == "seed_range" - assert c.baseline["score"] == 1.0 - assert c.edit == ["solution.py"] - - -def test_load_contract_absent_is_not_present(tmp_path: Path) -> None: - pack = _make_pack(tmp_path, "demo", readme="# no front matter\n\nbody\n") - assert load_contract(pack).present is False - - -def test_load_contract_frozen_optional(tmp_path: Path) -> None: - # absent → empty dict - assert load_contract(_make_pack(tmp_path, "a")).frozen == {} - # present → loaded - fm = _FM.replace("edit: [solution.py]\n", "edit: [solution.py]\nfrozen: {model: gpt-x}\n") - pack = _make_pack(tmp_path, "b", readme=fm) - assert load_contract(pack).frozen == {"model": "gpt-x"} - - def test_discover_skips_underscore_and_non_packs(tmp_path: Path) -> None: _make_pack(tmp_path, "real") - _make_pack(tmp_path, "_template") - (tmp_path / "not_a_pack").mkdir() + _make_pack(tmp_path, "_template") # scaffold — skipped + (tmp_path / "not_a_pack").mkdir() # no README/eval — skipped (tmp_path / "README.md").write_text("index", encoding="utf-8") assert [p.name for p in discover_packs(tmp_path)] == ["real"] -def test_discover_description_from_body(tmp_path: Path) -> None: - _make_pack(tmp_path, "demo") - assert discover_packs(tmp_path)[0].description == "A demo benchmark." +def test_discover_description_from_readme(tmp_path: Path) -> None: + _make_pack(tmp_path, "demo", readme="# Heading\n\nThe one-line summary.\n") + assert discover_packs(tmp_path)[0].description == "The one-line summary." + + +def test_discover_description_tolerates_legacy_front_matter(tmp_path: Path) -> None: + readme = "---\nname: demo\n---\n\n# Heading\n\nReal summary line.\n" + _make_pack(tmp_path, "demo", readme=readme) + assert discover_packs(tmp_path)[0].description == "Real summary line." def test_discover_empty_dir(tmp_path: Path) -> None: diff --git a/tests/test_zoo_scaffold.py b/tests/test_zoo_scaffold.py index 742e743..52cd255 100644 --- a/tests/test_zoo_scaffold.py +++ b/tests/test_zoo_scaffold.py @@ -10,105 +10,70 @@ from arbor.zoo import ScaffoldResult, scaffold_benchmark, verify_pack -_SEED_SPLITS = { - "kind": "seed_range", - "dev": {"base": 1000, "count": 3}, - "test": {"base": 9000, "count": 3}, -} -_PATH_SPLITS = {"kind": "path", "dev": ["data/dev/**"], "test": ["data/test/**"]} - def test_light_scaffold_creates_eval_split_and_solution(tmp_path: Path) -> None: - res = scaffold_benchmark( - tmp_path, name="demo", metric_direction="maximize", - splits=_SEED_SPLITS, style="light", - ) + res = scaffold_benchmark(tmp_path, name="demo", style="light") assert isinstance(res, ScaffoldResult) - assert "eval.py" in res.created - assert "solution.py" in res.created - assert (tmp_path / "eval.py").exists() - assert (tmp_path / "solution.py").exists() + assert "eval.py" in res.created and "solution.py" in res.created + assert (tmp_path / "eval.py").exists() and (tmp_path / "solution.py").exists() assert res.verify == [] # light style does not verify def test_light_eval_template_prints_parseable_score(tmp_path: Path) -> None: - scaffold_benchmark(tmp_path, name="demo", metric_direction="maximize", - splits=_SEED_SPLITS, style="light") + scaffold_benchmark(tmp_path, name="demo", style="light") proc = subprocess.run( [sys.executable, str(tmp_path / "eval.py"), "--split", "dev"], capture_output=True, text=True, ) - assert proc.returncode == 0 - assert "score:" in proc.stdout + assert proc.returncode == 0 and "score:" in proc.stdout def test_zoo_readme_run_commands_match_entrypoint(tmp_path: Path) -> None: - # eval.sh entrypoint → the README "Run the baseline" block uses bash eval.sh, - # not python eval.py. - scaffold_benchmark(tmp_path / "sh", name="demo", metric_direction="maximize", - splits=_SEED_SPLITS, style="zoo", eval_entrypoint="eval.sh") + scaffold_benchmark(tmp_path / "sh", name="demo", style="zoo", eval_entrypoint="eval.sh") readme = (tmp_path / "sh" / "README.md").read_text() assert "bash eval.sh dev" in readme and "python eval.py" not in readme - # eval.py entrypoint → python eval.py - scaffold_benchmark(tmp_path / "py", name="demo", metric_direction="maximize", - splits=_SEED_SPLITS, style="zoo", eval_entrypoint="eval.py") + scaffold_benchmark(tmp_path / "py", name="demo", style="zoo", eval_entrypoint="eval.py") assert "python eval.py --split dev" in (tmp_path / "py" / "README.md").read_text() +def test_zoo_readme_is_natural_language_no_front_matter(tmp_path: Path) -> None: + scaffold_benchmark(tmp_path, name="demo", metric_direction="minimize", style="zoo") + readme = (tmp_path / "README.md").read_text() + assert not readme.startswith("---") # no YAML front-matter + assert "lower is better" in readme # direction rendered in prose + + def test_path_split_creates_data_dirs(tmp_path: Path) -> None: - res = scaffold_benchmark(tmp_path, name="demo", metric_direction="minimize", - splits=_PATH_SPLITS, style="light") - # Visible placeholders (not hidden .gitkeep) so path globs can match them. + res = scaffold_benchmark(tmp_path, name="demo", split_kind="path", style="light") assert "data/dev/example_001.txt" in res.created assert "data/test/example_001.txt" in res.created -def test_zoo_path_split_passes_structural_verify(tmp_path: Path) -> None: - # Regression: path-kind zoo packs must also pass splits-disjoint, which - # requires glob-matchable (non-dotfile) data instances on both sides. - scaffold_benchmark(tmp_path, name="demo", metric_direction="maximize", - splits=_PATH_SPLITS, style="zoo") - results = verify_pack(tmp_path, run_eval=False) - fails = [r for r in results if r.status == "fail"] - assert not fails, f"path-kind structural verify failed: {[(r.name, r.message) for r in fails]}" - - def test_generated_eval_sh_uses_lf_line_endings(tmp_path: Path) -> None: # Regression: a CRLF shebang ("…bash\r") is a broken interpreter on Unix. - scaffold_benchmark(tmp_path, name="demo", metric_direction="maximize", - splits=_SEED_SPLITS, style="light", eval_entrypoint="eval.sh") + scaffold_benchmark(tmp_path, name="demo", style="light", eval_entrypoint="eval.sh") raw = (tmp_path / "eval.sh").read_bytes() assert b"\r\n" not in raw assert raw.split(b"\n", 1)[0] == b"#!/usr/bin/env bash" def test_zoo_scaffold_passes_structural_verify(tmp_path: Path) -> None: - res = scaffold_benchmark( - tmp_path, name="demo", metric_direction="maximize", - splits=_SEED_SPLITS, baseline={"score": 0.0, "tolerance": 0.0, "kind": "exact"}, - edit=["solution.py"], style="zoo", - ) - assert "README.md" in res.created - assert "PROVENANCE.md" in res.created - # Re-verify directly to prove the round-trip is real (not just trusting res.verify). - results = verify_pack(tmp_path, run_eval=False) - fails = [r for r in results if r.status == "fail"] + res = scaffold_benchmark(tmp_path, name="demo", style="zoo") + assert "README.md" in res.created and "PROVENANCE.md" in res.created + fails = [r for r in verify_pack(tmp_path) if r.status == "fail"] assert not fails, f"structural verify failed: {[(r.name, r.message) for r in fails]}" def test_scaffold_is_idempotent(tmp_path: Path) -> None: - scaffold_benchmark(tmp_path, name="demo", metric_direction="maximize", - splits=_SEED_SPLITS, style="zoo") - res2 = scaffold_benchmark(tmp_path, name="demo", metric_direction="maximize", - splits=_SEED_SPLITS, style="zoo") - assert res2.created == [] - assert "solution.py" in res2.skipped + scaffold_benchmark(tmp_path, name="demo", style="zoo") + res2 = scaffold_benchmark(tmp_path, name="demo", style="zoo") + assert res2.created == [] and "solution.py" in res2.skipped -def test_invalid_style_and_direction_raise(tmp_path: Path) -> None: +def test_invalid_args_raise(tmp_path: Path) -> None: + with pytest.raises(ValueError): + scaffold_benchmark(tmp_path, name="x", style="bogus") with pytest.raises(ValueError): - scaffold_benchmark(tmp_path, name="x", metric_direction="maximize", - splits=_SEED_SPLITS, style="bogus") + scaffold_benchmark(tmp_path, name="x", metric_direction="up") with pytest.raises(ValueError): - scaffold_benchmark(tmp_path, name="x", metric_direction="up", - splits=_SEED_SPLITS, style="light") + scaffold_benchmark(tmp_path, name="x", split_kind="weird") diff --git a/tests/test_zoo_verify.py b/tests/test_zoo_verify.py index 46215b2..b3e39b5 100644 --- a/tests/test_zoo_verify.py +++ b/tests/test_zoo_verify.py @@ -1,57 +1,28 @@ -"""Tests for the zoo verifier (``arbor.zoo.verify``). - -Each test builds a minimal valid benchmark in ``tmp_path`` and toggles a single defect, -asserting the relevant check flips to ``fail`` (or ``warn``). The default ``eval.sh`` -prints a constant ``score: 1.0`` so eval-running checks are fast and deterministic. -""" +"""Tests for the zoo verifier (``arbor.zoo.verify``) — a light structural lint.""" from __future__ import annotations -import copy from pathlib import Path -import yaml - from arbor.zoo import verify_pack -_CONTRACT: dict = { - "name": "demo", - "metric": {"direction": "maximize"}, - "splits": {"kind": "seed_range", "dev": {"base": 1000, "count": 3}, - "test": {"base": 9000, "count": 3}}, - "baseline": {"score": 1.0, "tolerance": 0.0, "kind": "exact"}, - "edit": ["solution.py"], -} -_BODY = ( - "# demo\n\nA demo benchmark.\n\n## Task & metric\nx\n## Run the baseline\nx\n" - "## Optimize with Arbor\nx\n## Provenance\nx\n" -) _PROVENANCE = ( - "# Provenance\n\n## Source\nx\n## Setup & environment\nx\n## Data source & license\nx\n" - "## Baseline implementation\nx\n## Baseline reproduction\nx\n" + "# Provenance\n\n## Source\nx\n## Setup & environment\nx\n## Baseline\nx\n" "## Contamination assessment\nx\n## Caveats\nx\n" ) -_EVAL_OK = '#!/usr/bin/env bash\necho "score: 1.0"\n' +_README = "# demo\n\nA demo benchmark.\n\n## The task\nx\n## Metric\nx\n" -def _build(tmp_path: Path, *, contract: dict | None = "default", body: str = _BODY, - provenance: str = _PROVENANCE, eval_sh: str | None = _EVAL_OK, - eval_py: str | None = None, files: dict | None = None) -> Path: +def _build(tmp_path: Path, *, readme: str | None = _README, + provenance: str | None = _PROVENANCE, eval_entry: str | None = "eval.sh") -> Path: pack = tmp_path / "demo" pack.mkdir(parents=True, exist_ok=True) - c = copy.deepcopy(_CONTRACT) if contract == "default" else contract - readme = _BODY if c is None else f"---\n{yaml.safe_dump(c)}---\n{body}" - (pack / "README.md").write_text(readme) - (pack / "PROVENANCE.md").write_text(provenance) - (pack / "solution.py").write_text("# baseline\n") - if eval_sh is not None: - (pack / "eval.sh").write_text(eval_sh) - if eval_py is not None: - (pack / "eval.py").write_text(eval_py) - for rel, content in (files or {}).items(): - f = pack / rel - f.parent.mkdir(parents=True, exist_ok=True) - f.write_text(content) + if readme is not None: + (pack / "README.md").write_text(readme) + if provenance is not None: + (pack / "PROVENANCE.md").write_text(provenance) + if eval_entry is not None: + (pack / eval_entry).write_text("echo 'score: 1.0'\n") return pack @@ -59,131 +30,34 @@ def _by_name(results) -> dict: return {r.name: r for r in results} -# ── happy path ──────────────────────────────────────────────────────────────── - -def test_valid_pack_has_no_fails(tmp_path: Path) -> None: +def test_valid_pack_passes(tmp_path: Path) -> None: results = verify_pack(_build(tmp_path)) - fails = [(r.name, r.message) for r in results if r.status == "fail"] - assert fails == [], f"unexpected fails: {fails}" - assert {r.name for r in results if r.status == "warn"} == {"contamination"} - - -# ── contract (front-matter) ─────────────────────────────────────────────────── - -def test_missing_front_matter_fails(tmp_path: Path) -> None: - r = _by_name(verify_pack(_build(tmp_path, contract=None)))["contract"] - assert r.status == "fail" and "front-matter" in r.message - - -def test_missing_contract_field_fails(tmp_path: Path) -> None: - c = copy.deepcopy(_CONTRACT) - del c["baseline"]["score"] - r = _by_name(verify_pack(_build(tmp_path, contract=c)))["contract"] - assert r.status == "fail" and "baseline.score" in r.message + assert [r for r in results if r.status == "fail"] == [] -def test_bad_metric_direction_fails(tmp_path: Path) -> None: - c = copy.deepcopy(_CONTRACT) - c["metric"]["direction"] = "highest" - r = _by_name(verify_pack(_build(tmp_path, contract=c)))["contract"] - assert r.status == "fail" and "metric.direction" in r.message +def test_missing_eval_entrypoint_fails(tmp_path: Path) -> None: + r = _by_name(verify_pack(_build(tmp_path, eval_entry=None)))["files-present"] + assert r.status == "fail" and "eval" in r.message -def test_frozen_valid_passes_and_surfaces(tmp_path: Path) -> None: - c = copy.deepcopy(_CONTRACT) - c["frozen"] = {"model": "gpt-x", "budget": "1h"} - r = _by_name(verify_pack(_build(tmp_path, contract=c)))["contract"] - assert r.status == "pass" and "freeze" in r.message +def test_missing_provenance_fails(tmp_path: Path) -> None: + r = _by_name(verify_pack(_build(tmp_path, provenance=None))) + assert r["files-present"].status == "fail" -def test_frozen_malformed_fails(tmp_path: Path) -> None: - c = copy.deepcopy(_CONTRACT) - c["frozen"] = "gpt-x" # must be a mapping - r = _by_name(verify_pack(_build(tmp_path, contract=c)))["contract"] - assert r.status == "fail" and "frozen" in r.message - - -# ── sections ────────────────────────────────────────────────────────────────── - -def test_missing_readme_section_fails(tmp_path: Path) -> None: - body = _BODY.replace("## Optimize with Arbor\nx\n", "") - r = _by_name(verify_pack(_build(tmp_path, body=body)))["readme-sections"] - assert r.status == "fail" and "Optimize" in r.message - - -def test_missing_provenance_heading_fails(tmp_path: Path) -> None: - prov = _PROVENANCE.replace("## Baseline implementation\nx\n", "") +def test_missing_provenance_section_fails(tmp_path: Path) -> None: + prov = _PROVENANCE.replace("## Contamination assessment\nx\n", "") r = _by_name(verify_pack(_build(tmp_path, provenance=prov)))["provenance"] - assert r.status == "fail" and "Baseline implementation" in r.message - - -# ── splits disjoint ─────────────────────────────────────────────────────────── - -def test_seed_ranges_overlap_fails(tmp_path: Path) -> None: - c = copy.deepcopy(_CONTRACT) - c["splits"]["test"] = {"base": 1001, "count": 3} - r = _by_name(verify_pack(_build(tmp_path, contract=c)))["splits-disjoint"] - assert r.status == "fail" and "overlap" in r.message - + assert r.status == "fail" and "Contamination" in r.message -def test_path_splits_disjoint_pass(tmp_path: Path) -> None: - c = copy.deepcopy(_CONTRACT) - c["splits"] = {"kind": "path", "dev": ["data/dev/**"], "test": ["data/test/**"]} - pack = _build(tmp_path, contract=c, files={"data/dev/a.txt": "1", "data/test/b.txt": "2"}) - assert _by_name(verify_pack(pack))["splits-disjoint"].status == "pass" - -# ── edit surface ────────────────────────────────────────────────────────────── - -def test_edit_pattern_matches_nothing_fails(tmp_path: Path) -> None: - c = copy.deepcopy(_CONTRACT) - c["edit"] = ["does_not_exist.py"] - r = _by_name(verify_pack(_build(tmp_path, contract=c)))["edit-surface"] +def test_empty_readme_fails(tmp_path: Path) -> None: + r = _by_name(verify_pack(_build(tmp_path, readme="# x\n")))["readme"] assert r.status == "fail" -def test_editable_harness_fails(tmp_path: Path) -> None: - c = copy.deepcopy(_CONTRACT) - c["edit"] = ["solution.py", "eval.sh"] # eval.sh must stay protected - r = _by_name(verify_pack(_build(tmp_path, contract=c)))["edit-surface"] - assert r.status == "fail" and "eval.sh" in r.message - - -# ── eval running ────────────────────────────────────────────────────────────── - -def test_eval_without_score_fails(tmp_path: Path) -> None: - r = _by_name(verify_pack(_build(tmp_path, eval_sh='#!/usr/bin/env bash\necho "nope"\n'))) - assert r["eval-dev"].status == "fail" - - -def test_eval_py_entrypoint_works(tmp_path: Path) -> None: - r = _by_name(verify_pack(_build(tmp_path, eval_sh=None, eval_py='print("score: 1.0")\n'))) - assert r["eval-dev"].status == "pass" and r["eval-test"].status == "pass" - - -# ── baseline reproduction ───────────────────────────────────────────────────── - -def test_baseline_out_of_tolerance_fails(tmp_path: Path) -> None: - r = _by_name(verify_pack(_build(tmp_path, eval_sh='#!/usr/bin/env bash\necho "score: 2.0"\n'))) - assert r["baseline-reproduces"].status == "fail" - - -# ── determinism ─────────────────────────────────────────────────────────────── - -def test_nondeterministic_exact_fails(tmp_path: Path) -> None: - r = _by_name(verify_pack(_build(tmp_path, eval_sh='#!/usr/bin/env bash\necho "score: ${RANDOM}.0"\n'))) - assert r["determinism"].status == "fail" - - -def test_timing_metric_within_tolerance_pass(tmp_path: Path) -> None: - c = copy.deepcopy(_CONTRACT) - c["baseline"] = {"score": 1.0, "tolerance": 0.15, "kind": "timing"} - assert _by_name(verify_pack(_build(tmp_path, contract=c)))["determinism"].status == "pass" - - -# ── run_eval=False ──────────────────────────────────────────────────────────── - -def test_no_eval_flag_skips_eval_checks(tmp_path: Path) -> None: - r = _by_name(verify_pack(_build(tmp_path), run_eval=False)) - assert r["eval-dev"].status == "warn" and r["baseline-reproduces"].status == "warn" - assert r["splits-disjoint"].status == "pass" # structural still runs +def test_verify_does_not_run_eval(tmp_path: Path) -> None: + # An eval that would exit 1 if run must not affect verify (it's never executed). + pack = _build(tmp_path, eval_entry=None) + (pack / "eval.sh").write_text("#!/usr/bin/env bash\nexit 1\n") + assert [r for r in verify_pack(pack) if r.status == "fail"] == [] From f2195558990bafe7b14b5406db77db7d9b292069 Mon Sep 17 00:00:00 2001 From: ignorejjj Date: Wed, 24 Jun 2026 15:57:03 +0800 Subject: [PATCH 3/3] docs(zoo): professional register for the zoo docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Revise the overview and format-reference pages (EN + zh) from a colloquial tone to a professional technical-documentation register — e.g. '任务说明' not '用大白话写清任务', '可直接运行的优化任务' not '现成的任务,直接刷', 'no fixed template' not 'however reads best'. Content and structure unchanged; mkdocs --strict passes. --- docs/zoo-overview.md | 64 ++++++++++++++++--------------- docs/zoo-overview.zh.md | 55 +++++++++++++------------- docs/zoo.md | 15 ++++---- docs/zoo.zh.md | 85 +++++++++++++++++++++-------------------- 4 files changed, 111 insertions(+), 108 deletions(-) diff --git a/docs/zoo-overview.md b/docs/zoo-overview.md index deb3e22..f056d01 100644 --- a/docs/zoo-overview.md +++ b/docs/zoo-overview.md @@ -1,56 +1,58 @@ # Benchmark Zoo -Arbor works by pointing at a task that prints a score, then repeatedly editing the code, -running the eval, and keeping the changes that improve the score. The **benchmark zoo** is a -collection of such tasks — each packaged in one standard format, so you can grab one and let -Arbor start optimizing it right away. It lives in -[`arbor-zoo/`](https://github.com/RUC-NLPIR/Arbor/tree/main/arbor-zoo), one folder per benchmark. +Arbor's workflow is to take a task that prints a score and iteratively edit the code, run +the eval, and keep the changes that improve the score. The **benchmark zoo** is a collection +of such tasks, each packaged in one standard format so it can be handed directly to Arbor to +optimize. It lives in [`arbor-zoo/`](https://github.com/RUC-NLPIR/Arbor/tree/main/arbor-zoo), +one folder per benchmark. ## What it's for -- **Ready-made tasks to optimize.** Each benchmark ships an eval and a baseline — point - `arbor` at it and it starts improving the score. -- **Turn your own task into one.** Have code but no runnable eval yet? One command adds the - eval scaffolding so Arbor can run it. +- **Ready-to-run optimization tasks.** Each benchmark ships an eval script and a baseline; + point Arbor at it and it begins optimizing. +- **Onboard your own task.** If you have code but no runnable eval, one command adds the eval + scaffolding. - **Collect new ones (in progress).** Have Arbor fetch a benchmark from GitHub / HuggingFace - and shape it into the zoo format. + and shape it into this format. -## What a benchmark looks like +## What a benchmark contains -One folder with three things: +Each benchmark is a directory with three parts: -- a **README** — what the task is and which score to optimize; -- **baseline code** — Arbor's starting point, and the only part it's allowed to edit; -- an **eval script** — run it and it prints one `score:` line; it's protected, so Arbor can't - game it. +- a **README** — the task description: what the task is, the metric, and what Arbor may edit; + read by Arbor during intake; +- **baseline code** — the starting point for optimization, and the only part Arbor may edit + (e.g. `solution.py`); +- an **eval script** — prints one `score:` line when run; it is protected, so Arbor cannot + modify it. -Arbor's loop is just: edit the baseline → run the eval → keep the change if the score went up, +Arbor's loop is: edit the baseline → run the eval → keep the change if the score improved, and repeat. ## Entry points -| What you want to do | Command | +| Purpose | Command | | --- | --- | -| See what's in the zoo | `arbor benchmark list` | -| Optimize a benchmark with Arbor | copy it out, `git init`, then run `arbor` in it | -| Check a benchmark is valid | `arbor benchmark verify ` | -| Make your own code Arbor-ready | `arbor benchmark scaffold ` | -| Pull a benchmark from GitHub / HF | `arbor benchmark add --name ` | +| List the benchmarks | `arbor benchmark list` | +| Run Arbor on a benchmark | copy it out of the repo, `git init`, run `arbor` inside it | +| Verify a benchmark's structure | `arbor benchmark verify ` | +| Make your code a runnable benchmark | `arbor benchmark scaffold ` | +| Fetch a benchmark from GitHub / HF | `arbor benchmark add --name ` | -Running one, start to finish: +Running one: ```bash -cp -r arbor-zoo/algotune_knn /tmp/algotune_knn # copy out of the Arbor checkout +cp -r arbor-zoo/algotune_knn /tmp/algotune_knn # copy out of the Arbor repo cd /tmp/algotune_knn && git init -q && git add -A && git commit -qm baseline arbor # confirm the task, then it iterates ``` -## What's here, what's next +## Status -- ✅ **Available now:** the format, `verify`, `list`, `scaffold`, the `add` spine, and the - first example benchmark, `algotune_knn`. -- ⏳ **In progress:** making `add` smarter (research the benchmark and bring its baseline up - automatically), and adding more benchmarks. +- **Available:** the format, `verify`, `list`, `scaffold`, the `add` spine, and the first + example benchmark, `algotune_knn`. +- **In progress:** strengthening `add` (research the benchmark and bring its baseline up), and + adding more benchmarks. -To author one, see the [format reference](zoo.md). For where this is heading, see the +For the exact format, see the [format reference](zoo.md); for the wider plan, see the [roadmap](roadmap.md). diff --git a/docs/zoo-overview.zh.md b/docs/zoo-overview.zh.md index 6250934..fac6871 100644 --- a/docs/zoo-overview.zh.md +++ b/docs/zoo-overview.zh.md @@ -1,48 +1,47 @@ # 基准库（Benchmark Zoo） -Arbor 的工作方式很简单:对着一个能打分的任务,反复改代码、跑评测、把让分数变好的改动留下来。 -**基准库**就是一批这样的任务的集合——每个都打包成统一格式,拿来就能直接让 Arbor 上手优化。它放在 -仓库的 [`arbor-zoo/`](https://github.com/RUC-NLPIR/Arbor/tree/main/arbor-zoo) 下,一个文件夹一个基准。 +Arbor 的工作流程是:针对一个可评分的任务,迭代地修改代码、运行评测,并保留能提升分数的改动。 +**基准库**是一组此类任务的集合,每个任务以统一格式打包,可直接交由 Arbor 优化。它位于仓库的 +[`arbor-zoo/`](https://github.com/RUC-NLPIR/Arbor/tree/main/arbor-zoo) 目录下,每个基准一个文件夹。 -## 它用来干什么 +## 主要用途 -- **现成的任务,直接刷。** 库里每个基准都自带评测和一份基线,`arbor` 对准它就能开始优化、刷分。 -- **把你自己的任务也变成这样。** 有代码、但还不能评测?一条命令补上评测的架子,Arbor 就能跑了。 -- **自动收集(开发中)。** 让 Arbor 自己去 GitHub / HuggingFace 上把一个 benchmark 拉下来,整理成 - 库里的格式。 +- **可直接运行的优化任务。** 每个基准自带评测脚本与基线实现,将 Arbor 指向它即可开始优化。 +- **接入你自己的任务。** 若你已有代码但缺少可运行的评测,一条命令即可补全评测脚手架。 +- **自动收集(开发中)。** 由 Arbor 从 GitHub / HuggingFace 获取一个 benchmark 并整理为本格式。 -## 一个基准长什么样 +## 基准的组成 -就是一个文件夹,里面三样东西: +每个基准是一个目录,包含以下部分: -- 一份 **README**:这个任务是什么、要优化哪个分数; -- 一份 **基线代码**:Arbor 的优化起点,也是它唯一能改的部分; -- 一个 **评测脚本**:跑一下打印一行 `score:`;它受保护,Arbor 改不了它、也就没法作弊。 +- **README** —— 任务说明:任务内容、优化的指标、Arbor 可修改的范围。供 Arbor 在接入阶段读取。 +- **基线实现** —— 优化的起点,也是 Arbor 唯一可修改的部分(如 `solution.py`)。 +- **评测脚本** —— 运行后打印一行 `score:`;该脚本受保护,Arbor 不可修改。 -Arbor 做的事就是一个循环:改基线 → 跑评测 → 分数涨了就留下,如此反复。 +Arbor 的迭代循环为:修改基线 → 运行评测 → 若分数提升则保留,循环往复。 -## 几个入口 +## 使用入口 -| 你想做什么 | 命令 | +| 用途 | 命令 | | --- | --- | -| 看库里有哪些基准 | `arbor benchmark list` | -| 在某个基准上跑 Arbor 优化 | 把它拷出仓库,`git init`,在目录里跑 `arbor` | -| 检查一个基准合不合格 | `arbor benchmark verify <目录>` | -| 把你自己的代码补成 Arbor 能跑的 | `arbor benchmark scaffold <目录>` | -| 从 GitHub / HF 拉一个 benchmark 进来 | `arbor benchmark add <地址> --name <名字>` | +| 列出库中的基准 | `arbor benchmark list` | +| 在某个基准上运行 Arbor | 将其拷出仓库,`git init` 后在目录内运行 `arbor` | +| 校验一个基准的结构 | `arbor benchmark verify <目录>` | +| 将你的代码补全为可运行的基准 | `arbor benchmark scaffold <目录>` | +| 从 GitHub / HF 获取一个 benchmark | `arbor benchmark add <地址> --name <名称>` | -完整跑一个的例子: +运行示例: ```bash cp -r arbor-zoo/algotune_knn /tmp/algotune_knn # 拷出 Arbor 仓库 cd /tmp/algotune_knn && git init -q && git add -A && git commit -qm baseline -arbor # 确认任务,然后它开始迭代 +arbor # 确认任务后开始迭代 ``` -## 现在有什么、还在做什么 +## 进展 -- ✅ **已经能用:** 基准格式、校验(`verify`)、列表(`list`)、补脚手架(`scaffold`)、收集的主体 - (`add`),以及第一个示例基准 `algotune_knn`。 -- ⏳ **还在做:** 让 `add` 更聪明(自动调研 benchmark、把基线跑通),以及加入更多基准。 +- **已支持:** 基准格式、校验(`verify`)、列表(`list`)、脚手架(`scaffold`)、收集主体(`add`), + 以及首个示例基准 `algotune_knn`。 +- **进行中:** 增强 `add`(自动调研 benchmark 并跑通基线),以及补充更多基准。 -想自己写一个基准,详细格式见[格式参考](zoo.md);整条线的规划见[路线图](roadmap.md)。 +撰写一个基准的详细格式见[格式参考](zoo.md);整体规划见[路线图](roadmap.md)。 diff --git a/docs/zoo.md b/docs/zoo.md index 672c63d..fe5f7cd 100644 --- a/docs/zoo.md +++ b/docs/zoo.md @@ -20,21 +20,22 @@ Each `arbor-zoo//` holds four things: Folders whose name starts with `_` (e.g. `_template`) are scaffolding and are skipped. -### `README.md` — the task, in plain language +### `README.md` — the task description The README is what Arbor reads to understand the task, the same way its intake reads any -repo. Write it however reads best; a benchmark usually covers four things: +repository. There is no fixed template; a description usually covers four things: 1. **The task** — what it is and what a solution looks like. 2. **The metric** — what the eval prints (one `score:` line) and whether higher or lower is better. -3. **What Arbor may edit** — the baseline file(s); everything else (the eval harness, - ground-truth, data) is off-limits. -4. **Dev / test** — how the two differ, so the held-out split is clear (disjoint seeds, or +3. **What Arbor may edit** — the baseline file(s); everything else (the eval script, + reference implementation, data) is protected. +4. **Dev / test** — how the two are split, so the held-out set is clear (disjoint seeds, or `data/dev/` vs `data/test/`). -There is **no fixed baseline number** in the format: the same baseline gives different scores -on different hardware/models, so it's described in PROVENANCE, not pinned as a value. +The format contains **no fixed baseline number**: the same baseline gives different scores on +different hardware/models, so the baseline is described in PROVENANCE rather than pinned as a +value. ### `PROVENANCE.md` — the human card diff --git a/docs/zoo.zh.md b/docs/zoo.zh.md index 876c923..f0561c9 100644 --- a/docs/zoo.zh.md +++ b/docs/zoo.zh.md @@ -1,78 +1,79 @@ # 基准库 —— 格式参考 -这一页讲一个基准文件夹的确切格式,以及 `arbor benchmark verify` 检查什么。整体介绍见 +本页说明一个基准目录的格式,以及 `arbor benchmark verify` 的检查项。整体介绍见 [总览](zoo-overview.md)。 -格式是**文档优先**的:一个基准就是一个文档齐全的文件夹。README 是 Arbor 在 intake 时读的自然语言 -说明——**没有要填的 YAML 清单**。 +本格式以文档为中心:一个基准即一个文档完备的目录。README 为 Arbor 在接入阶段读取的自然语言说明, +无需填写 YAML 清单。 -## 一个基准文件夹包含什么 +## 目录组成 -每个 `arbor-zoo//` 里有四样东西: +每个 `arbor-zoo//` 包含以下内容: | 文件 / 目录 | 作用 | 面向 | | --- | --- | --- | -| `README.md` | 任务是什么、指标、Arbor 能改什么、dev/test 怎么分——用自然语言写。 | Arbor(和人) | -| 基线代码 | 可编辑的基线——`solution.py`,或一整套文件。 | Arbor 编辑 | -| `eval.sh` *或* `eval.py` | 受保护 eval 入口。`bash eval.sh dev\|test`(或 `python eval.py --split …`)打印一行 `score: `。 | 受保护 | -| `PROVENANCE.md` | 来源、环境、baseline 怎么实现、污染评估、注意事项。 | 人看 | -| `data/`、`task.py`… | eval 需要的数据 / ground-truth(受保护)。 | — | +| `README.md` | 任务说明:任务内容、指标、Arbor 可修改的范围、dev/test 的划分方式(自然语言)。 | Arbor(及人) | +| 基线实现 | 可修改的基线 —— `solution.py`,或一组文件。 | Arbor 修改 | +| `eval.sh` *或* `eval.py` | 受保护的评测入口。`bash eval.sh dev\|test`(或 `python eval.py --split …`)打印一行 `score: `。 | 受保护 | +| `PROVENANCE.md` | 来源、运行环境、基线说明、污染评估、注意事项。 | 人工查阅 | +| `data/`、`task.py` 等 | 评测所需的数据 / 参考实现(受保护)。 | — | -以 `_` 开头的文件夹(如 `_template`)是脚手架,会被跳过。 +以 `_` 开头的目录(如 `_template`)为脚手架,工具会自动跳过。 -### `README.md` —— 用大白话写清任务 +### `README.md` —— 任务说明 -README 就是 Arbor 用来理解任务的东西,跟它 intake 读任何 repo 一样。怎么读着顺就怎么写;一个基准 -通常写清四件事: +README 是 Arbor 理解任务的依据,其读取方式与接入任意代码仓库时一致。无固定模板,一份说明通常包含 +以下四部分: -1. **任务** —— 是什么、一个解长什么样。 -2. **指标** —— eval 打印什么(一行 `score:`)、越大还是越小好。 -3. **Arbor 能改什么** —— 基线文件;其余一切(eval、ground-truth、数据)不许碰。 -4. **dev / test** —— 两者怎么分,让留出集清楚(不相交的种子,或 `data/dev/` vs `data/test/`)。 +1. **任务** —— 任务内容,以及一个解的形态。 +2. **指标** —— 评测打印的内容(一行 `score:`),以及越大或越小为优。 +3. **可修改范围** —— 作为基线的文件;其余内容(评测脚本、参考实现、数据)均不可修改。 +4. **dev / test** —— 两者的划分方式,以明确留出集(不相交的随机种子,或 `data/dev/`、`data/test/` + 两个目录)。 -格式里**没有固定的 baseline 数字**:同一份基线在不同硬件/模型上跑出来不一样,所以它写在 PROVENANCE -里,而不是钉成一个值。 +格式中**不包含固定的基线分数**:同一基线在不同硬件 / 模型下分数不同,因此基线写入 PROVENANCE,而非 +固化为一个数值。 -### `PROVENANCE.md` —— 给人看的卡片 +### `PROVENANCE.md` —— 来源说明 -必含章节(校验器会查在不在):**Source**、**Setup & environment**、**Baseline**、 -**Contamination assessment**、**Caveats**。来源、license、baseline 怎么实现的(以及分数波动多大)、 -留出集的理由,都写在这里给维护者读。 +必含章节(校验器检查其是否存在):**Source**、**Setup & environment**、**Baseline**、 +**Contamination assessment**、**Caveats**。来源、许可证、基线的实现方式(及分数的波动范围)、留出集的 +说明等,均在此记录,供维护者审阅。 -## `arbor benchmark verify` 查什么 +## `arbor benchmark verify` 的检查项 -`verify` 是个轻量的**结构**检查——查齐全,不查正确性,也**不跑 eval**(baseline 分数本就不通用)。 -它查: +`verify` 为轻量的**结构**检查,用于确认组成完整,而非正确性门禁,且**不运行评测**(基线分数并非通用 +数值)。它检查: -- `README.md` 在、且非空; -- `PROVENANCE.md` 在、且必含章节齐全; -- eval 入口(`eval.sh` 或 `eval.py`)在。 +- `README.md` 存在且非空; +- `PROVENANCE.md` 存在且必含章节齐全; +- 评测入口(`eval.sh` 或 `eval.py`)存在。 ```bash -arbor benchmark verify arbor-zoo/ # 缺东西就非零退出 -arbor benchmark list arbor-zoo # 列出有哪些基准 +arbor benchmark verify arbor-zoo/ # 缺少任一项即以非零码退出 +arbor benchmark list arbor-zoo # 列出基准 ``` -dev/test 是否*真的*留出、baseline 到底干了什么,写在 PROVENANCE 文字里、由人判断——不做机器强制。 +dev/test 是否真正留出、基线的实际行为等,记录于 PROVENANCE 并由人工判断,不做机器强制。 -## 用 Arbor 跑一个基准 +## 在基准上运行 Arbor -Arbor 在仓库根的 git worktree 里跑实验,所以请在 Arbor 检出**之外**的副本里工作: +Arbor 在仓库根目录的 git worktree 中运行实验,因此请在 Arbor 仓库**之外**的副本中操作: ```bash cp -r arbor-zoo/algotune_knn /tmp/algotune_knn cd /tmp/algotune_knn git init -q && git add -A && git commit -qm baseline -arbor # Arbor 读 README、确认任务,然后开始迭代 +arbor # Arbor 读取 README、确认任务后开始迭代 ``` ## 新增一个基准 -1. 起架子:`arbor benchmark scaffold arbor-zoo/ --style zoo`。它写一个 eval 占位、 - `solution.py` 占位、自然语言 `README.md` 和 `PROVENANCE.md`——但绝不替你写解法。 -2. 填好基线(`solution.py`)、eval(`eval.py`/`eval.sh`)、README(给 Arbor 看)、PROVENANCE(给人看)。 -3. 反复 `arbor benchmark verify arbor-zoo/` 直到退出 0,再由维护者接受。起草可以自动, - **接受是人工这一步**。 +1. 生成脚手架:`arbor benchmark scaffold arbor-zoo/ --style zoo`。该命令写入评测占位、 + `solution.py` 占位、自然语言 `README.md` 与 `PROVENANCE.md`,但不会生成解法本身。 +2. 补全基线(`solution.py`)、评测(`eval.py` / `eval.sh`)、README(供 Arbor)与 PROVENANCE(供人工)。 +3. 反复运行 `arbor benchmark verify arbor-zoo/` 直至以 0 退出,再由维护者审核接受。起草可自动化, + **接受为人工环节**。 -端到端的例子见 +完整示例见 [`arbor-zoo/algotune_knn`](https://github.com/RUC-NLPIR/Arbor/tree/main/arbor-zoo/algotune_knn)。