refactor(zoo): natural-language format (drop front-matter) + overview page#34
Merged
Conversation
A user-facing big-picture page for the zoo: the vision (regression-harness-first, not a trophy board), the core idea (a pack is a locked optimization problem), the categorization (two layers + the freeze axis + task shapes), what's shipped vs planned, the four usage entry points (run / scaffold / author / add), and the discipline. Bilingual (EN + zh); nav groups it with the existing format reference.
The README front-matter contract (metric / splits / baseline / tolerance / edit /
frozen) proved too rigid, and a fixed baseline number isn't universal — the same
baseline gives different scores by user, hardware, and model. So the format goes
documentation-first:
- **README is natural language, for Arbor.** It describes the task, the metric, what
Arbor may edit, and how dev/test differ — in plain prose Arbor reads at intake. No
YAML manifest.
- **PROVENANCE is for humans.** Source, setup, how the baseline works (and that results
vary), contamination, caveats.
- **`verify` is a light structural lint** — README + PROVENANCE(+sections) + an eval
entrypoint present. It no longer runs the eval (a score isn't a universal value), so
the held-out/baseline/determinism machine checks are gone; those are stated in
PROVENANCE prose and judged by a human.
Changes: pack.py (drop Contract/front-matter parsing), verify.py (structural-only),
scaffold.py (emit NL README + PROVENANCE; signature now name/metric_direction/style/
split_kind/eval_entrypoint/edit), collect.py + benchmark_cmd.py + mcp session_ops/server
(drop front-matter/eval-run args), the algotune_knn pack + _template (NL README,
PROVENANCE without a baseline number), and docs/zoo.{md,zh} + arbor-zoo/README.
Also: a simpler, user-focused zoo overview page and "基准库" (not "基准动物园") in zh.
Tests rewritten for the light format. ruff + mypy + full suite + mkdocs --strict green.
Note: the internal dev design notes (docs/dev/benchmark-add.md, benchmark-backlog.md)
still describe the old front-matter contract and need a later reconciliation pass when
the collection agent stages are built.
ignorejjj
changed the title
Revise the overview and format-reference pages (EN + zh) from a colloquial tone to a professional technical-documentation register — e.g. '任务说明' not '用大白话写清任务', '可直接运行的优化任务' not '现成的任务,直接刷', 'no fixed template' not 'however reads best'. Content and structure unchanged; mkdocs --strict passes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The README front-matter contract (metric / splits / baseline / tolerance / edit / frozen) was too rigid, and a fixed
baseline.scoreisn't universal — the same baseline gives different numbers by user, hardware, and model. So the zoo format goes documentation-first.The new format
verify= a light structural lint. README + PROVENANCE(+sections) + an eval entrypoint present. It no longer runs the eval (a score isn't a universal value), so the held-out / baseline / determinism machine checks are gone — those live in PROVENANCE prose and are judged by a human.Changes
src/zoo/:pack.py(drop theContract/ front-matter parsing),verify.py(structural-only),scaffold.py(emit an NL README + PROVENANCE; simpler signature),collect.py+__init__.benchmark verifydrops--no-eval/--timeout;scaffolddrops the front-matter flags;session_ops/serverscaffold tool updated.algotune_knnpack and_template→ NL README + PROVENANCE (no baseline number).docs/zoo.{md,zh}rewritten to the NL format; a simpler, user-focused overview page (zoo-overview.md, EN + zh); "基准库" instead of "基准动物园" in zh.ruff + mypy + full suite +
mkdocs build --strictall green.Follow-up (not in this PR)
The internal dev design notes (
docs/dev/benchmark-add.md,benchmark-backlog.md, excluded from the published site) still describe the old front-matter contract; they need a reconciliation pass when the collection agent stages are built.