feat(zoo): query-driven collection — discovery + bring-up agents

README.md

@@ -290,6 +290,29 @@ In one 6-cycle run this drove the dev speedup from **1.01x → 7.77x** (held-out
 test **1.00x → 7.22x**). See [`examples/algotune_knn/README.md`](examples/algotune_knn/README.md)
 for the research contract and tuning knobs.
 
+### Collect a benchmark from a request (experimental)
+
+You don't have to assemble a benchmark by hand. `arbor benchmark add` turns a
+one-line request into a runnable **draft task**: it finds the dataset/benchmark and,
+on an interactive terminal, asks you **which dataset** to use and **where the
+baseline comes from** — harvest an existing implementation, implement the method you
+described, or find one online — then acquires the data and brings up a draft
+(baseline + eval + `README` + provenance). It does *not* force-run the eval; a real
+run may need your served model / API key.
+
+```bash
+# name a work, or give a goal + a method
+arbor benchmark add "get me the datasets WebThinker uses"
+arbor benchmark add "I want to climb GPQA with a self-consistency baseline"
+
+# or point straight at a repo / HF dataset (add --bringup to also build the baseline)
+arbor benchmark add https://github.com/owner/repo --name my-bench --bringup
+```
+
+Drafting is automated; acceptance stays human. `arbor benchmark verify <dir>`
+structurally checks a pack, and `arbor benchmark list` indexes the zoo. See the
+[benchmark zoo overview](docs/zoo-overview.md) for the format and the full flow.
+
 ---
 
 ## 🧠 How It Works

docs/roadmap.md

@@ -117,11 +117,14 @@ Still open:
 
 - **Grow the collection** to 3–5 high-quality, human-checked packs across distinct
   task shapes, using `algotune_knn` as the reference. Cap on quality, not count.
-- **`arbor benchmark add`** — semi-automatic conversion: the intake agent *drafts*
-  a Task Pack from a raw benchmark, gated behind the verifier and a human accept
-  step (draft-automatic, accept-verified — never auto-accepted). The
-  baseline-implementing agent stays separate from the loop that later optimizes it,
-  so evaluation isn't self-certifying. *(Designed; not yet built.)*
+- **`arbor benchmark add`** — semi-automatic conversion: from a one-line request the
+  agent finds the dataset, asks (on an interactive terminal) which dataset to use and
+  where the baseline comes from (harvest an existing one / implement the method you
+  described / find one online), and brings up a runnable draft — gated behind the
+  verifier and a human accept step (draft-automatic, accept-verified — never
+  auto-accepted). The baseline-implementing agent stays separate from the loop that
+  later optimizes it, so evaluation isn't self-certifying. *(Built: discovery +
+  interactive bring-up; bring-up reasoning still maturing.)*
 - **Lower a pack into a [plugin](plugins.md)** for one-line retargeting — the
   front-matter contract reuses the `plugin` vocabulary (`eval_contract` /
   `protected_paths`), so it should fall out with little rework (pairs with 2.2).

docs/roadmap.zh.md

@@ -97,9 +97,11 @@
 
 - **扩充集合**到 3–5 个高质量、人工核对的 pack，覆盖不同任务形态，以 `algotune_knn` 为
   参考。质量封顶，不是数量封顶。
-- **`arbor benchmark add`**——半自动转换：intake agent 从原始基准*起草*一个 Task Pack，
-  再由校验器和人工接受这一步把关（起草自动、接受需校验——绝不自动接受）。*实现* baseline 的
-  agent 与之后优化它的 loop 分开，使评测不自证。*（已设计；尚未实现。）*
+- **`arbor benchmark add`**——半自动转换:从一句话需求出发,agent 找到数据集,在交互终端里
+  询问用户**用哪个数据集、baseline 从哪来**(收割现成的 / 按你描述的方法实现 / 上网找),并产出
+  一个可运行草稿,再由校验器和人工接受这一步把关(起草自动、接受需校验——绝不自动接受)。*实现*
+  baseline 的 agent 与之后优化它的 loop 分开,使评测不自证。*(已实现:discovery + 交互式
+  bring-up;bring-up 的推理仍在打磨。)*
 - **把一个 pack 降级成 [plugin](plugins.md)** 以实现一行改写重定向——front-matter 契约
   复用 `plugin` 词汇（`eval_contract` / `protected_paths`），应能几乎无返工地导出（与 2.2
   配套）。

docs/zoo-overview.md

@@ -12,8 +12,12 @@ one folder per benchmark.
   point Arbor at it and it begins optimizing.
 - **Onboard your own task.** If you have code but no runnable eval, one command adds the eval
   scaffolding.
-- **Collect new ones (in progress).** Have Arbor fetch a benchmark from GitHub / HuggingFace
-  and shape it into this format.
+- **Collect new ones (in progress).** Describe what you want in one line — name a work
+  ("get me the datasets WebThinker uses") or a goal ("I want to climb GPQA with a
+  self-consistency method"). An agent finds the dataset/benchmark, asks you which dataset to
+  use and where the baseline should come from (harvest an existing one, implement the method
+  you described, or find one online), acquires the data, and brings up a runnable draft. You
+  can also point it straight at a repo URL.
 
 ## What a benchmark contains
 
@@ -37,7 +41,7 @@ and repeat.
 | Run Arbor on a benchmark | copy it out of the repo, `git init`, run `arbor` inside it |
 | Verify a benchmark's structure | `arbor benchmark verify <dir>` |
 | Make your code a runnable benchmark | `arbor benchmark scaffold <dir>` |
-| Fetch a benchmark from GitHub / HF | `arbor benchmark add <url> --name <name>` |
+| Find & build a benchmark from a request | `arbor benchmark add "<request>"` (or a repo URL) |
 
 Running one:
 
@@ -51,8 +55,9 @@ arbor                                             # confirm the task, then it it
 
 - **Available:** the format, `verify`, `list`, `scaffold`, the `add` spine, and the first
   example benchmark, `algotune_knn`.
-- **In progress:** strengthening `add` (research the benchmark and bring its baseline up), and
-  adding more benchmarks.
+- **In progress:** strengthening `add` — from a one-line request, find the dataset, ask which
+  one and where the baseline comes from, and bring up a runnable draft — and adding more
+  benchmarks.
 
 For the exact format, see the [format reference](zoo.md); for the wider plan, see the
 [roadmap](roadmap.md).

docs/zoo-overview.zh.md

@@ -8,7 +8,7 @@ Arbor 的工作流程是:针对一个可评分的任务,迭代地修改代码、
 
 - **可直接运行的优化任务。** 每个基准自带评测脚本与基线实现,将 Arbor 指向它即可开始优化。
 - **接入你自己的任务。** 若你已有代码但缺少可运行的评测,一条命令即可补全评测脚手架。
-- **自动收集(开发中)。** 由 Arbor 从 GitHub / HuggingFace 获取一个 benchmark 并整理为本格式。
+- **自动收集(开发中)。** 用一句话描述需求——点名一个工作("帮我获取 WebThinker 用到的数据集"),或一个目标("我想用自一致性方法刷 GPQA")。由 agent 找到数据集/benchmark,**询问你要哪个数据集、baseline 从哪来**(收割现成的、按你描述的方法实现、或上网找现成实现),获取数据并产出一个可运行草稿;也可直接给一个 repo 地址。
 
 ## 基准的组成
 
@@ -28,7 +28,7 @@ Arbor 的迭代循环为:修改基线 → 运行评测 → 若分数提升则保
 | 在某个基准上运行 Arbor | 将其拷出仓库,`git init` 后在目录内运行 `arbor` |
 | 校验一个基准的结构 | `arbor benchmark verify <目录>` |
 | 将你的代码补全为可运行的基准 | `arbor benchmark scaffold <目录>` |
-| 从 GitHub / HF 获取一个 benchmark | `arbor benchmark add <地址> --name <名称>` |
+| 查找并构建一个 benchmark | `arbor benchmark add "<需求>"`(或 repo 地址) |
 
 运行示例:
 
@@ -42,6 +42,6 @@ arbor                                             # 确认任务后开始迭代
 
 - **已支持:** 基准格式、校验(`verify`)、列表(`list`)、脚手架(`scaffold`)、收集主体(`add`),
   以及首个示例基准 `algotune_knn`。
-- **进行中:** 增强 `add`(自动调研 benchmark 并跑通基线),以及补充更多基准。
+- **进行中:** 增强 `add`(一句话需求 → 找到数据集、问你要哪个/baseline 从哪来、产出可运行草稿),以及补充更多基准。
 
 撰写一个基准的详细格式见[格式参考](zoo.md);整体规划见[路线图](roadmap.md)。

src/cli/commands/benchmark_cmd.py

@@ -8,9 +8,13 @@
   leaderboard).
 * ``arbor benchmark scaffold <dir>`` — write the measurement plumbing (light) or a
   full zoo benchmark (zoo) into an existing local directory.
-* ``arbor benchmark add <spec> --name <name>`` — acquire a benchmark (git repo / HF
-  dataset) into the global cache and scaffold a draft pack (Phase 1: the deterministic
-  spine; the agent-driven survey + bring-up are the next sub-phase).
+* ``arbor benchmark add "<request>" | <url>`` — turn a one-line request into a runnable
+  draft task. A natural-language request is handled end-to-end by an agent: it finds the
+  dataset/benchmark, asks (on a TTY) which dataset to use and where the baseline comes from
+  (harvest an existing one / implement the method you described / find one online), acquires
+  the data, and brings up a runnable draft (baseline + eval + README + PROVENANCE) — without
+  force-running the eval. A bare URL/HF spec skips discovery and just acquires + scaffolds;
+  ``--bringup`` also brings up its baseline.
 """
 
 from __future__ import annotations
@@ -22,7 +26,9 @@
 
 from ...zoo import (
     VerifyResult,
+    bringup,
     collect,
+    discover,
     discover_packs,
     select_acquirer,
     verify_pack,
@@ -48,6 +54,24 @@ def _render(r: VerifyResult) -> None:
             typer.secho(f"         hint: {r.hint}", fg=typer.colors.RED, err=True)
 
 
+def _user_runner(*, with_search: bool, ask_user: bool = False):
+    """Build an agent runner that uses the user's configured provider (~/.arbor/config.yaml),
+    so the collection agents inherit the same LLM as `arbor run` (e.g. openai-oauth/gpt-5.5).
+
+    ``ask_user`` adds a console AskUser tool so the agent can put a genuinely human decision
+    (e.g. which implementation is the baseline) to the user; enable it only on an interactive
+    terminal."""
+    from ...zoo import real_agent_runner
+    from ..user_config import llm_defaults
+
+    llm = llm_defaults()
+    return real_agent_runner(
+        with_search=with_search, ask_user=ask_user,
+        provider=llm.get("provider"), model=llm.get("model"),
+        api_key=llm.get("api_key"), base_url=llm.get("base_url"),
+    )
+
+
 @benchmark_app.command("verify")
 def verify_command(
     pack_dir: Path = typer.Argument(
@@ -147,29 +171,86 @@ def scaffold_command(
 def add_command(
     spec: str = typer.Argument(
         ...,
-        help="A git repo URL (optionally `url@commit`) or a HF dataset (`hf:<id>`).",
+        help="A natural-language query, a git repo URL (optionally `url@commit`), or a HF "
+             "dataset (`hf:<id>`). A query is resolved by the discovery agent.",
     ),
-    name: str = typer.Option(
-        ..., "--name", "-n",
-        help="Pack name (the arbor-zoo/<name> folder).",
+    name: str | None = typer.Option(
+        None, "--name", "-n",
+        help="Pack name (the arbor-zoo/<name> folder). Optional for a query (discovery suggests one).",
     ),
     dest: Path = typer.Option(
         Path("arbor-zoo"), "--dest",
         help="Where to write the draft pack (default: ./arbor-zoo).",
     ),
+    assume_yes: bool = typer.Option(
+        False, "--yes", "-y", help="Skip the confirmation prompt for a discovered benchmark.",
+    ),
+    do_bringup: bool = typer.Option(
+        False, "--bringup",
+        help="Force the bring-up agent (implement/wire the baseline + write the eval). "
+             "A natural-language query runs bring-up by default; use this to also trigger it "
+             "for a bare URL/HF spec. Needs a configured LLM provider / API key.",
+    ),
+    max_turns: int = typer.Option(
+        100, "--max-turns", help="Agent turn budget (discovery / bring-up).",
+    ),
 ) -> None:
-    """Acquire a benchmark and scaffold a draft pack (deterministic spine).
+    """Turn a one-line request into a runnable draft task.
 
-    Phase 1: selects an acquirer, clones/downloads into the global cache, scaffolds a
-    draft pack, and structurally verifies it. The agent-driven survey + baseline bring-up
-    are the next sub-phase — this leaves a draft for a human (or a later agent pass) to
-    complete, then `arbor benchmark verify` and accept.
+    A **natural-language request** is handled by an agent end-to-end: it searches GitHub /
+    HuggingFace / arXiv, and — on an interactive terminal — asks you which dataset to use and
+    where the baseline should come from (harvest an existing one, implement the method you
+    described, or find one online). It then acquires the data and brings up a *runnable draft*
+    (baseline + eval + README + PROVENANCE). It does NOT force-run the eval — a real run may
+    need your served model / API key. A bare URL/HF spec skips discovery and just acquires +
+    scaffolds; add ``--bringup`` to also bring up its baseline. Acceptance stays human.
     """
-    if select_acquirer(spec) is None:
-        typer.secho(
-            f"error: no acquirer matched {spec!r} — expected a git URL or `hf:<dataset-id>`",
-            fg=typer.colors.RED, err=True,
-        )
+    import asyncio
+    import sys
+
+    interactive = sys.stdin.isatty()
+    request = spec                          # the user's original words (shapes an implemented baseline)
+    from_query = select_acquirer(spec) is None
+    baseline_plan: dict = {}
+
+    # ── Natural-language request → discovery agent → a chosen source ──────────
+    if from_query:
+        import tempfile
+
+        typer.secho(f"searching for a benchmark matching: {spec!r} …", fg=typer.colors.CYAN)
+        try:
+            disc = asyncio.run(discover(
+                spec, run_agent=_user_runner(with_search=True, ask_user=interactive),
+                work_dir=Path(tempfile.mkdtemp(prefix="arbor-discover-")),
+                max_turns=max_turns,
+            ))
+        except Exception as exc:  # noqa: BLE001
+            typer.secho(f"  discovery could not start: {exc}", fg=typer.colors.RED, err=True)
+            typer.echo("  (configure a provider with `arbor setup` / set your API key)")
+            raise typer.Exit(code=1) from exc
+        for note in disc.notes:
+            typer.echo(f"  • {note}")
+        if not disc.ok or not disc.url:
+            typer.secho("no suitable benchmark found — give a specific repo URL instead.",
+                        fg=typer.colors.RED, err=True)
+            raise typer.Exit(code=1)
+        choice = disc.choice or {}
+        baseline_plan = disc.baseline_plan
+        typer.secho(f"\nchosen: {disc.name}  →  {disc.url}", fg=typer.colors.GREEN)
+        typer.echo(f"  metric:   {choice.get('metric', '?')}")
+        typer.echo(f"  baseline: {choice.get('baseline', '?')}")
+        if baseline_plan:
+            typer.echo(f"  plan:     baseline via {baseline_plan.get('source', '?')} — "
+                       f"{baseline_plan.get('detail', '')}")
+        typer.echo(f"  why:      {choice.get('why', '?')}")
+        if not assume_yes and not typer.confirm("\nacquire this benchmark?", default=True):
+            raise typer.Exit(code=0)
+        spec = disc.url
+        name = name or disc.name
+
+    if not name:
+        typer.secho("error: --name is required (could not infer one).",
+                    fg=typer.colors.RED, err=True)
         raise typer.Exit(code=2)
 
     typer.echo(f"collecting {name} from {spec} …")
@@ -188,6 +269,35 @@ def add_command(
     typer.echo(f"structural verify: {len(fails)} fail(s) "
                f"(eval not run — the draft eval is a stub)")
 
+    # A query runs bring-up by default (producing a runnable draft is the whole point); a bare
+    # URL/HF spec only brings up when asked with --bringup.
+    if (from_query or do_bringup) and result.draft_pack_dir:
+        materials = result.acquired.materials_dir if result.acquired else None
+        typer.secho("\nbringing up the baseline (agent) …", fg=typer.colors.CYAN)
+        try:
+            br = asyncio.run(bringup(
+                result.draft_pack_dir,
+                run_agent=_user_runner(with_search=True, ask_user=interactive),
+                materials_dir=materials,
+                instruction=request if from_query else "",
+                baseline_plan=baseline_plan,
+                max_turns=max_turns,
+            ))
+        except Exception as exc:  # noqa: BLE001 — surface provider/setup errors clearly
+            typer.secho(f"  bring-up could not start: {exc}", fg=typer.colors.RED, err=True)
+            typer.echo("  (configure a provider with `arbor setup` / set your API key, then retry)")
+            raise typer.Exit(code=1) from exc
+        for note in br.notes:
+            typer.echo(f"  • {note}")
+        if br.ran:
+            typer.secho(f"  bring-up {'ok' if br.ok else 'incomplete'} — eval ran "
+                        f"(dev score: {br.dev_score})",
+                        fg=typer.colors.GREEN if br.ok else typer.colors.YELLOW)
+        else:
+            typer.secho(f"  {'runnable draft ready' if br.ok else 'bring-up incomplete'} — "
+                        f"eval not run here (needs your model / API key)",
+                        fg=typer.colors.GREEN if br.ok else typer.colors.YELLOW)
+
     typer.secho("\nstill to do (drafting is automated, acceptance is not):",
                 fg=typer.colors.YELLOW)
     for step in result.pending:

src/zoo/__init__.py

@@ -13,6 +13,8 @@
 from __future__ import annotations
 
 from .acquire import Acquired, Acquirer, GitRepoAcquirer, Sources, select_acquirer
+from .agent_stages import BringupResult, DiscoveryResult, bringup, discover, real_agent_runner
+from .ask_tool import ConsoleAskUserTool
 from .cache import Manifest, benchmark_cache_dir, cache_root
 from .collect import CollectResult, collect
 from .pack import (
@@ -29,19 +31,25 @@
     "EVAL_ENTRYPOINTS",
     "Acquired",
     "Acquirer",
+    "BringupResult",
     "CollectResult",
+    "ConsoleAskUserTool",
+    "DiscoveryResult",
     "GitRepoAcquirer",
     "Manifest",
     "PackSummary",
     "ScaffoldResult",
     "Sources",
     "VerifyResult",
     "benchmark_cache_dir",
+    "bringup",
     "cache_root",
     "collect",
+    "discover",
     "discover_packs",
     "find_eval_entrypoint",
     "is_pack_dir",
+    "real_agent_runner",
     "scaffold_benchmark",
     "select_acquirer",
     "verify_pack",