diff --git a/README.md b/README.md index 9f017cf..69e0ba7 100644 --- a/README.md +++ b/README.md @@ -290,6 +290,29 @@ In one 6-cycle run this drove the dev speedup from **1.01x → 7.77x** (held-out test **1.00x → 7.22x**). See [`examples/algotune_knn/README.md`](examples/algotune_knn/README.md) for the research contract and tuning knobs. +### Collect a benchmark from a request (experimental) + +You don't have to assemble a benchmark by hand. `arbor benchmark add` turns a +one-line request into a runnable **draft task**: it finds the dataset/benchmark and, +on an interactive terminal, asks you **which dataset** to use and **where the +baseline comes from** — harvest an existing implementation, implement the method you +described, or find one online — then acquires the data and brings up a draft +(baseline + eval + `README` + provenance). It does *not* force-run the eval; a real +run may need your served model / API key. + +```bash +# name a work, or give a goal + a method +arbor benchmark add "get me the datasets WebThinker uses" +arbor benchmark add "I want to climb GPQA with a self-consistency baseline" + +# or point straight at a repo / HF dataset (add --bringup to also build the baseline) +arbor benchmark add https://github.com/owner/repo --name my-bench --bringup +``` + +Drafting is automated; acceptance stays human. `arbor benchmark verify ` +structurally checks a pack, and `arbor benchmark list` indexes the zoo. See the +[benchmark zoo overview](docs/zoo-overview.md) for the format and the full flow. + --- ## 🧠 How It Works diff --git a/docs/roadmap.md b/docs/roadmap.md index 2ad5e49..23a57b2 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -117,11 +117,14 @@ Still open: - **Grow the collection** to 3–5 high-quality, human-checked packs across distinct task shapes, using `algotune_knn` as the reference. Cap on quality, not count. -- **`arbor benchmark add`** — semi-automatic conversion: the intake agent *drafts* - a Task Pack from a raw benchmark, gated behind the verifier and a human accept - step (draft-automatic, accept-verified — never auto-accepted). The - baseline-implementing agent stays separate from the loop that later optimizes it, - so evaluation isn't self-certifying. *(Designed; not yet built.)* +- **`arbor benchmark add`** — semi-automatic conversion: from a one-line request the + agent finds the dataset, asks (on an interactive terminal) which dataset to use and + where the baseline comes from (harvest an existing one / implement the method you + described / find one online), and brings up a runnable draft — gated behind the + verifier and a human accept step (draft-automatic, accept-verified — never + auto-accepted). The baseline-implementing agent stays separate from the loop that + later optimizes it, so evaluation isn't self-certifying. *(Built: discovery + + interactive bring-up; bring-up reasoning still maturing.)* - **Lower a pack into a [plugin](plugins.md)** for one-line retargeting — the front-matter contract reuses the `plugin` vocabulary (`eval_contract` / `protected_paths`), so it should fall out with little rework (pairs with 2.2). diff --git a/docs/roadmap.zh.md b/docs/roadmap.zh.md index 8c45a8b..be3cfdc 100644 --- a/docs/roadmap.zh.md +++ b/docs/roadmap.zh.md @@ -97,9 +97,11 @@ - **扩充集合**到 3–5 个高质量、人工核对的 pack，覆盖不同任务形态，以 `algotune_knn` 为 参考。质量封顶，不是数量封顶。 -- **`arbor benchmark add`**——半自动转换：intake agent 从原始基准*起草*一个 Task Pack， - 再由校验器和人工接受这一步把关（起草自动、接受需校验——绝不自动接受）。*实现* baseline 的 - agent 与之后优化它的 loop 分开，使评测不自证。*（已设计；尚未实现。）* +- **`arbor benchmark add`**——半自动转换:从一句话需求出发,agent 找到数据集,在交互终端里 + 询问用户**用哪个数据集、baseline 从哪来**(收割现成的 / 按你描述的方法实现 / 上网找),并产出 + 一个可运行草稿,再由校验器和人工接受这一步把关(起草自动、接受需校验——绝不自动接受)。*实现* + baseline 的 agent 与之后优化它的 loop 分开,使评测不自证。*(已实现:discovery + 交互式 + bring-up;bring-up 的推理仍在打磨。)* - **把一个 pack 降级成 [plugin](plugins.md)** 以实现一行改写重定向——front-matter 契约 复用 `plugin` 词汇（`eval_contract` / `protected_paths`），应能几乎无返工地导出（与 2.2 配套）。 diff --git a/docs/zoo-overview.md b/docs/zoo-overview.md index f056d01..804fbc2 100644 --- a/docs/zoo-overview.md +++ b/docs/zoo-overview.md @@ -12,8 +12,12 @@ one folder per benchmark. point Arbor at it and it begins optimizing. - **Onboard your own task.** If you have code but no runnable eval, one command adds the eval scaffolding. -- **Collect new ones (in progress).** Have Arbor fetch a benchmark from GitHub / HuggingFace - and shape it into this format. +- **Collect new ones (in progress).** Describe what you want in one line — name a work + ("get me the datasets WebThinker uses") or a goal ("I want to climb GPQA with a + self-consistency method"). An agent finds the dataset/benchmark, asks you which dataset to + use and where the baseline should come from (harvest an existing one, implement the method + you described, or find one online), acquires the data, and brings up a runnable draft. You + can also point it straight at a repo URL. ## What a benchmark contains @@ -37,7 +41,7 @@ and repeat. | Run Arbor on a benchmark | copy it out of the repo, `git init`, run `arbor` inside it | | Verify a benchmark's structure | `arbor benchmark verify ` | | Make your code a runnable benchmark | `arbor benchmark scaffold ` | -| Fetch a benchmark from GitHub / HF | `arbor benchmark add --name ` | +| Find & build a benchmark from a request | `arbor benchmark add ""` (or a repo URL) | Running one: @@ -51,8 +55,9 @@ arbor # confirm the task, then it it - **Available:** the format, `verify`, `list`, `scaffold`, the `add` spine, and the first example benchmark, `algotune_knn`. -- **In progress:** strengthening `add` (research the benchmark and bring its baseline up), and - adding more benchmarks. +- **In progress:** strengthening `add` — from a one-line request, find the dataset, ask which + one and where the baseline comes from, and bring up a runnable draft — and adding more + benchmarks. For the exact format, see the [format reference](zoo.md); for the wider plan, see the [roadmap](roadmap.md). diff --git a/docs/zoo-overview.zh.md b/docs/zoo-overview.zh.md index fac6871..943f043 100644 --- a/docs/zoo-overview.zh.md +++ b/docs/zoo-overview.zh.md @@ -8,7 +8,7 @@ Arbor 的工作流程是:针对一个可评分的任务,迭代地修改代码、 - **可直接运行的优化任务。** 每个基准自带评测脚本与基线实现,将 Arbor 指向它即可开始优化。 - **接入你自己的任务。** 若你已有代码但缺少可运行的评测,一条命令即可补全评测脚手架。 -- **自动收集(开发中)。** 由 Arbor 从 GitHub / HuggingFace 获取一个 benchmark 并整理为本格式。 +- **自动收集(开发中)。** 用一句话描述需求——点名一个工作("帮我获取 WebThinker 用到的数据集"),或一个目标("我想用自一致性方法刷 GPQA")。由 agent 找到数据集/benchmark,**询问你要哪个数据集、baseline 从哪来**(收割现成的、按你描述的方法实现、或上网找现成实现),获取数据并产出一个可运行草稿;也可直接给一个 repo 地址。 ## 基准的组成 @@ -28,7 +28,7 @@ Arbor 的迭代循环为:修改基线 → 运行评测 → 若分数提升则保 | 在某个基准上运行 Arbor | 将其拷出仓库,`git init` 后在目录内运行 `arbor` | | 校验一个基准的结构 | `arbor benchmark verify <目录>` | | 将你的代码补全为可运行的基准 | `arbor benchmark scaffold <目录>` | -| 从 GitHub / HF 获取一个 benchmark | `arbor benchmark add <地址> --name <名称>` | +| 查找并构建一个 benchmark | `arbor benchmark add "<需求>"`(或 repo 地址) | 运行示例: @@ -42,6 +42,6 @@ arbor # 确认任务后开始迭代 - **已支持:** 基准格式、校验(`verify`)、列表(`list`)、脚手架(`scaffold`)、收集主体(`add`), 以及首个示例基准 `algotune_knn`。 -- **进行中:** 增强 `add`(自动调研 benchmark 并跑通基线),以及补充更多基准。 +- **进行中:** 增强 `add`(一句话需求 → 找到数据集、问你要哪个/baseline 从哪来、产出可运行草稿),以及补充更多基准。 撰写一个基准的详细格式见[格式参考](zoo.md);整体规划见[路线图](roadmap.md)。 diff --git a/src/cli/commands/benchmark_cmd.py b/src/cli/commands/benchmark_cmd.py index f9961ed..ff64d6b 100644 --- a/src/cli/commands/benchmark_cmd.py +++ b/src/cli/commands/benchmark_cmd.py @@ -8,9 +8,13 @@ leaderboard). * ``arbor benchmark scaffold `` — write the measurement plumbing (light) or a full zoo benchmark (zoo) into an existing local directory. -* ``arbor benchmark add --name `` — acquire a benchmark (git repo / HF - dataset) into the global cache and scaffold a draft pack (Phase 1: the deterministic - spine; the agent-driven survey + bring-up are the next sub-phase). +* ``arbor benchmark add "" | `` — turn a one-line request into a runnable + draft task. A natural-language request is handled end-to-end by an agent: it finds the + dataset/benchmark, asks (on a TTY) which dataset to use and where the baseline comes from + (harvest an existing one / implement the method you described / find one online), acquires + the data, and brings up a runnable draft (baseline + eval + README + PROVENANCE) — without + force-running the eval. A bare URL/HF spec skips discovery and just acquires + scaffolds; + ``--bringup`` also brings up its baseline. """ from __future__ import annotations @@ -22,7 +26,9 @@ from ...zoo import ( VerifyResult, + bringup, collect, + discover, discover_packs, select_acquirer, verify_pack, @@ -48,6 +54,24 @@ def _render(r: VerifyResult) -> None: typer.secho(f" hint: {r.hint}", fg=typer.colors.RED, err=True) +def _user_runner(*, with_search: bool, ask_user: bool = False): + """Build an agent runner that uses the user's configured provider (~/.arbor/config.yaml), + so the collection agents inherit the same LLM as `arbor run` (e.g. openai-oauth/gpt-5.5). + + ``ask_user`` adds a console AskUser tool so the agent can put a genuinely human decision + (e.g. which implementation is the baseline) to the user; enable it only on an interactive + terminal.""" + from ...zoo import real_agent_runner + from ..user_config import llm_defaults + + llm = llm_defaults() + return real_agent_runner( + with_search=with_search, ask_user=ask_user, + provider=llm.get("provider"), model=llm.get("model"), + api_key=llm.get("api_key"), base_url=llm.get("base_url"), + ) + + @benchmark_app.command("verify") def verify_command( pack_dir: Path = typer.Argument( @@ -147,29 +171,86 @@ def scaffold_command( def add_command( spec: str = typer.Argument( ..., - help="A git repo URL (optionally `url@commit`) or a HF dataset (`hf:`).", + help="A natural-language query, a git repo URL (optionally `url@commit`), or a HF " + "dataset (`hf:`). A query is resolved by the discovery agent.", ), - name: str = typer.Option( - ..., "--name", "-n", - help="Pack name (the arbor-zoo/ folder).", + name: str | None = typer.Option( + None, "--name", "-n", + help="Pack name (the arbor-zoo/ folder). Optional for a query (discovery suggests one).", ), dest: Path = typer.Option( Path("arbor-zoo"), "--dest", help="Where to write the draft pack (default: ./arbor-zoo).", ), + assume_yes: bool = typer.Option( + False, "--yes", "-y", help="Skip the confirmation prompt for a discovered benchmark.", + ), + do_bringup: bool = typer.Option( + False, "--bringup", + help="Force the bring-up agent (implement/wire the baseline + write the eval). " + "A natural-language query runs bring-up by default; use this to also trigger it " + "for a bare URL/HF spec. Needs a configured LLM provider / API key.", + ), + max_turns: int = typer.Option( + 100, "--max-turns", help="Agent turn budget (discovery / bring-up).", + ), ) -> None: - """Acquire a benchmark and scaffold a draft pack (deterministic spine). + """Turn a one-line request into a runnable draft task. - Phase 1: selects an acquirer, clones/downloads into the global cache, scaffolds a - draft pack, and structurally verifies it. The agent-driven survey + baseline bring-up - are the next sub-phase — this leaves a draft for a human (or a later agent pass) to - complete, then `arbor benchmark verify` and accept. + A **natural-language request** is handled by an agent end-to-end: it searches GitHub / + HuggingFace / arXiv, and — on an interactive terminal — asks you which dataset to use and + where the baseline should come from (harvest an existing one, implement the method you + described, or find one online). It then acquires the data and brings up a *runnable draft* + (baseline + eval + README + PROVENANCE). It does NOT force-run the eval — a real run may + need your served model / API key. A bare URL/HF spec skips discovery and just acquires + + scaffolds; add ``--bringup`` to also bring up its baseline. Acceptance stays human. """ - if select_acquirer(spec) is None: - typer.secho( - f"error: no acquirer matched {spec!r} — expected a git URL or `hf:`", - fg=typer.colors.RED, err=True, - ) + import asyncio + import sys + + interactive = sys.stdin.isatty() + request = spec # the user's original words (shapes an implemented baseline) + from_query = select_acquirer(spec) is None + baseline_plan: dict = {} + + # ── Natural-language request → discovery agent → a chosen source ────────── + if from_query: + import tempfile + + typer.secho(f"searching for a benchmark matching: {spec!r} …", fg=typer.colors.CYAN) + try: + disc = asyncio.run(discover( + spec, run_agent=_user_runner(with_search=True, ask_user=interactive), + work_dir=Path(tempfile.mkdtemp(prefix="arbor-discover-")), + max_turns=max_turns, + )) + except Exception as exc: # noqa: BLE001 + typer.secho(f" discovery could not start: {exc}", fg=typer.colors.RED, err=True) + typer.echo(" (configure a provider with `arbor setup` / set your API key)") + raise typer.Exit(code=1) from exc + for note in disc.notes: + typer.echo(f" • {note}") + if not disc.ok or not disc.url: + typer.secho("no suitable benchmark found — give a specific repo URL instead.", + fg=typer.colors.RED, err=True) + raise typer.Exit(code=1) + choice = disc.choice or {} + baseline_plan = disc.baseline_plan + typer.secho(f"\nchosen: {disc.name} → {disc.url}", fg=typer.colors.GREEN) + typer.echo(f" metric: {choice.get('metric', '?')}") + typer.echo(f" baseline: {choice.get('baseline', '?')}") + if baseline_plan: + typer.echo(f" plan: baseline via {baseline_plan.get('source', '?')} — " + f"{baseline_plan.get('detail', '')}") + typer.echo(f" why: {choice.get('why', '?')}") + if not assume_yes and not typer.confirm("\nacquire this benchmark?", default=True): + raise typer.Exit(code=0) + spec = disc.url + name = name or disc.name + + if not name: + typer.secho("error: --name is required (could not infer one).", + fg=typer.colors.RED, err=True) raise typer.Exit(code=2) typer.echo(f"collecting {name} from {spec} …") @@ -188,6 +269,35 @@ def add_command( typer.echo(f"structural verify: {len(fails)} fail(s) " f"(eval not run — the draft eval is a stub)") + # A query runs bring-up by default (producing a runnable draft is the whole point); a bare + # URL/HF spec only brings up when asked with --bringup. + if (from_query or do_bringup) and result.draft_pack_dir: + materials = result.acquired.materials_dir if result.acquired else None + typer.secho("\nbringing up the baseline (agent) …", fg=typer.colors.CYAN) + try: + br = asyncio.run(bringup( + result.draft_pack_dir, + run_agent=_user_runner(with_search=True, ask_user=interactive), + materials_dir=materials, + instruction=request if from_query else "", + baseline_plan=baseline_plan, + max_turns=max_turns, + )) + except Exception as exc: # noqa: BLE001 — surface provider/setup errors clearly + typer.secho(f" bring-up could not start: {exc}", fg=typer.colors.RED, err=True) + typer.echo(" (configure a provider with `arbor setup` / set your API key, then retry)") + raise typer.Exit(code=1) from exc + for note in br.notes: + typer.echo(f" • {note}") + if br.ran: + typer.secho(f" bring-up {'ok' if br.ok else 'incomplete'} — eval ran " + f"(dev score: {br.dev_score})", + fg=typer.colors.GREEN if br.ok else typer.colors.YELLOW) + else: + typer.secho(f" {'runnable draft ready' if br.ok else 'bring-up incomplete'} — " + f"eval not run here (needs your model / API key)", + fg=typer.colors.GREEN if br.ok else typer.colors.YELLOW) + typer.secho("\nstill to do (drafting is automated, acceptance is not):", fg=typer.colors.YELLOW) for step in result.pending: diff --git a/src/zoo/__init__.py b/src/zoo/__init__.py index 4196ac9..e12a958 100644 --- a/src/zoo/__init__.py +++ b/src/zoo/__init__.py @@ -13,6 +13,8 @@ from __future__ import annotations from .acquire import Acquired, Acquirer, GitRepoAcquirer, Sources, select_acquirer +from .agent_stages import BringupResult, DiscoveryResult, bringup, discover, real_agent_runner +from .ask_tool import ConsoleAskUserTool from .cache import Manifest, benchmark_cache_dir, cache_root from .collect import CollectResult, collect from .pack import ( @@ -29,7 +31,10 @@ "EVAL_ENTRYPOINTS", "Acquired", "Acquirer", + "BringupResult", "CollectResult", + "ConsoleAskUserTool", + "DiscoveryResult", "GitRepoAcquirer", "Manifest", "PackSummary", @@ -37,11 +42,14 @@ "Sources", "VerifyResult", "benchmark_cache_dir", + "bringup", "cache_root", "collect", + "discover", "discover_packs", "find_eval_entrypoint", "is_pack_dir", + "real_agent_runner", "scaffold_benchmark", "select_acquirer", "verify_pack", diff --git a/src/zoo/agent_stages.py b/src/zoo/agent_stages.py new file mode 100644 index 0000000..4e84bd0 --- /dev/null +++ b/src/zoo/agent_stages.py @@ -0,0 +1,349 @@ +"""Agent-driven collection stages (Stage 2: baseline bring-up). + +The collection *spine* (:mod:`arbor.zoo.collect`) is deterministic — it acquires +materials and scaffolds a draft. This module is the agent-driven part: given a live +LLM provider, :func:`bringup` spawns one agent in the draft folder to make a baseline +actually run — install deps, get the reference working, wrap a clean ``eval`` that +prints ``score:``, and write the README + PROVENANCE — then checks its work by running +the eval and the structural verifier. + +It reuses the core :class:`~arbor.core.agent.Agent` runtime (the same one the executor +uses) but stays a standalone flow: it never wires into the Coordinator/Executor research +loop (a §2.1 correctness requirement). + +The agent run is behind an injected ``run_agent`` callable so the orchestration is +testable without a live LLM (a fake runner writes the files a real agent would). The real +runner — :func:`real_agent_runner` — constructs the ``Agent`` with bash + file tools and +needs a configured provider (API key); validating its *reasoning* needs live iteration. +""" + +from __future__ import annotations + +import json +import os +import re +import subprocess +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Awaitable, Protocol + +from .pack import find_eval_entrypoint +from .verify import VerifyResult, verify_pack + + +class AgentRunner(Protocol): + """Runs an agent to completion in *cwd* and returns its final transcript text.""" + + def __call__(self, *, cwd: Path, system_prompt: str, task: str, + max_turns: int) -> Awaitable[str]: ... + + +BRINGUP_SYSTEM_PROMPT = """\ +You are a benchmark bring-up assistant. You turn acquired materials (a research repo, a +dataset) into a runnable Arbor benchmark in the current directory. You write the *measurement +plumbing and a working baseline* — never an optimized solution. + +Produce, in the current directory: + * a runnable eval: `bash eval.sh dev|test` (or `python eval.py --split dev|test`) prints + exactly one line `score: `, after a correctness check. dev and test must use + DISJOINT data (the held-out split). + * the editable baseline (e.g. `solution.py`) — the simplest correct reference, the thing + Arbor will later optimize. Do NOT optimize it. + * `README.md` — plain language: the task, the metric (and whether higher/lower is better), + which file(s) Arbor may edit, and how dev/test differ. + * `PROVENANCE.md` — for humans: Source, Setup & environment, Baseline, Contamination + assessment, Caveats. + +The "baseline" is the *starting point Arbor will optimize*, NOT a SOTA method. It may come +from three places — follow the baseline plan you are given, and use `AskUser` if it is unclear: + * harvest — take a simple runnable baseline already in the repo (direct generation, + naive RAG, an earlier system) rather than the repo's headline method; + * implement — write a baseline to the user's described method/instruction (you are given it + below as the user's request); + * web — find an existing baseline implementation online and adapt it. + +Use the acquired source materials at the path you are told. Install dependencies as needed. +You are DONE when the four artifacts exist, `arbor benchmark verify .` would pass, and the +eval is *runnable*. Do NOT block on actually running it to completion: a real run may need a +served model, an API key, or a search key the user has not set up. If you can run the eval +cheaply (e.g. CPU-only), do so to sanity-check it; otherwise make it runnable, document the +exact setup needed in README + PROVENANCE, and stop — leaving a runnable draft is success. +If you are blocked on something you cannot resolve, write what you have and explain it clearly. +""" + + +@dataclass +class BringupResult: + """Outcome of a bring-up run.""" + + transcript: str = "" + dev_score: float | None = None + ran: bool = False # the eval actually ran and produced a score + verify: list[VerifyResult] = field(default_factory=list) + ok: bool = False + notes: list[str] = field(default_factory=list) + + +def _parse_score(text: str) -> float | None: + # Mirrors arbor.mcp.session_ops.parse_score for the documented `score: ` line, + # kept local so arbor.zoo stays dependency-light. + import re + matches = re.findall(r"\bscore\s*[:=]\s*([-+]?\d+(?:\.\d+)?)", text, re.I) + return float(matches[-1]) if matches else None + + +def _run_eval_dev(pack_dir: Path, timeout: int) -> tuple[float | None, str]: + """Run the eval on the dev split and parse a score (the bring-up success check).""" + entry = find_eval_entrypoint(pack_dir) + if entry == "eval.sh": + cmd = ["bash", str(pack_dir / "eval.sh"), "dev"] + elif entry == "eval.py": + cmd = [os.environ.get("PYTHON", "python3"), str(pack_dir / "eval.py"), "--split", "dev"] + else: + return None, "no eval entrypoint" + try: + proc = subprocess.run(cmd, cwd=str(pack_dir), capture_output=True, text=True, + timeout=timeout) + except subprocess.TimeoutExpired: + return None, f"eval timed out after {timeout}s" + out = proc.stdout + proc.stderr + return _parse_score(out), out[-2000:] + + +async def bringup( + pack_dir: Path, + *, + run_agent: AgentRunner, + materials_dir: Path | None = None, + instruction: str = "", + baseline_plan: dict[str, Any] | None = None, + max_turns: int = 40, + eval_timeout: int = 600, +) -> BringupResult: + """Run the bring-up agent in *pack_dir*, then check its work. + + *run_agent* does the actual agent work (real: an :class:`Agent`; in tests: a fake that + writes the files). *materials_dir* is the acquired source the agent should draw from. + *instruction* is the user's original natural-language request (so a baseline can be + *implemented* to their described method), and *baseline_plan* records where the baseline + should come from (harvest / implement / web). + + Success is a **runnable draft**: the artifacts are present and the structural verify + passes. Actually running the eval is best-effort — a real run may need a served model / + API key the user hasn't set up — so a non-running eval is noted, not a failure. + """ + result = BringupResult() + parts = [f"Bring up the benchmark in this directory ({pack_dir}). Produce a runnable " + f"baseline and an eval that prints a `score:` line on dev and test, plus " + f"README.md + PROVENANCE.md."] + if materials_dir: + parts.append(f"The acquired source materials are at: {materials_dir}") + if instruction: + parts.append(f"The user's original request (use it to shape/implement the baseline):\n" + f"{instruction}") + if baseline_plan: + src = baseline_plan.get("source", "?") + detail = baseline_plan.get("detail", "") + parts.append(f"Baseline plan — source={src}: {detail}") + task = "\n\n".join(parts) + try: + result.transcript = await run_agent( + cwd=pack_dir, system_prompt=BRINGUP_SYSTEM_PROMPT, task=task, max_turns=max_turns) + except Exception as exc: # noqa: BLE001 — surface agent/provider errors as a blocker + result.notes.append(f"agent run failed: {exc}") + return result + + # ── success check: the pack verifies (a runnable draft); running the eval is best-effort ── + result.dev_score, eval_out = _run_eval_dev(pack_dir, eval_timeout) + result.ran = result.dev_score is not None + if not result.ran: + result.notes.append( + "eval did not produce a score here — left as a runnable draft (a real run may " + f"need a served model / API key):\n{eval_out}") + result.verify = verify_pack(pack_dir) + verify_ok = not any(r.status == "fail" for r in result.verify) + result.ok = verify_ok + if not verify_ok: + result.notes.append("structural verify still has failures") + return result + + +# ── Stage 0: discovery (natural-language query → a chosen benchmark source) ── + +DISCOVERY_SYSTEM_PROMPT = """\ +You are a benchmark discovery assistant. Given a natural-language request, you search the +web for a benchmark that fits, judge the candidates, and pick the single best one. + +Use the search and page-fetch tools to look across GitHub, HuggingFace, arXiv / +PapersWithCode, and leaderboards. For each candidate, judge: + * does it ship a runnable eval and a baseline (not just a dataset)? + * does the task have headroom to optimize (an artifact Arbor can edit), not just measure + a frozen model? + * compute fit and license — can it be cloned and run, and is it redistributable? + * is it a representative / actively-used benchmark for the request? + +Prefer a GitHub repo that already contains an eval + baseline. **Be efficient: as soon as +you have identified one suitable repo and can name its benchmark(s) and baseline(s) — usually +after reading the paper and the repo README — STOP and output your choice. Do not +exhaustively clone and grep.** + +The "baseline" is the *starting point Arbor will optimize*, NOT the repo's headline method. +A repo that proposes a method usually also ships simpler baselines (direct generation, naive +RAG, an earlier system); for Arbor those simpler baselines are the harvestable ones, because +they leave headroom to optimize. Name a concrete baseline implementation (a runnable script), +not a published number. When which one to treat as the baseline is genuinely the user's call +and an `AskUser` tool is available, ask them rather than guessing. + +The request may name a *work* that uses several datasets/benchmarks (e.g. "get me the +datasets in WebThinker"). In that case enumerate the datasets it uses and, if an `AskUser` +tool is available, ask the user **which single dataset** they want — then resolve that one. +Also decide where the baseline will come from and record it as `baseline_plan.source`: + * "harvest" — a runnable baseline script already in the repo, + * "implement" — write one to the user's described method/instruction (e.g. "design xxx"), + * "web" — find an existing implementation online. +Ask the user (AskUser) when the baseline source is genuinely their call. + +End your reply with a single fenced JSON block describing your choice (and nothing after it): + +```json +{ + "name": "short_kebab_name", + "source": {"kind": "git", "url": "https://github.com/owner/repo"}, + "metric": "what is optimized, and whether higher/lower is better", + "baseline": "where/what the harvestable baseline is", + "baseline_plan": {"source": "harvest", "detail": "which script / method / search to use"}, + "why": "one or two sentences on why this fits the request" +} +``` + +If nothing suitable is found, set "source" to null and explain in "why". +""" + + +@dataclass +class DiscoveryResult: + """Outcome of a discovery run: the chosen benchmark source (or none).""" + + choice: dict[str, Any] | None = None # {name, source:{kind,url}, metric, baseline, why} + transcript: str = "" + ok: bool = False + notes: list[str] = field(default_factory=list) + + @property + def url(self) -> str | None: + src = (self.choice or {}).get("source") or {} + return src.get("url") if isinstance(src, dict) else None + + @property + def name(self) -> str | None: + return (self.choice or {}).get("name") + + @property + def baseline_plan(self) -> dict[str, Any]: + """How/where the baseline should come from: {source: harvest|implement|web, detail}.""" + plan = (self.choice or {}).get("baseline_plan") + return plan if isinstance(plan, dict) else {} + + +def _extract_json(text: str) -> dict[str, Any] | None: + """Pull the last JSON object out of the agent's reply (a ```json fenced block, or a + bare top-level object).""" + blocks = re.findall(r"```(?:json)?\s*(\{.*?\})\s*```", text, re.DOTALL) + if not blocks: + blocks = re.findall(r"(\{(?:[^{}]|\{[^{}]*\})*\})", text, re.DOTALL) + for block in reversed(blocks): + try: + obj = json.loads(block) + except json.JSONDecodeError: + continue + if isinstance(obj, dict) and "source" in obj: + return obj + return None + + +async def discover( + query: str, + *, + run_agent: AgentRunner, + work_dir: Path, + max_turns: int = 100, +) -> DiscoveryResult: + """Run the discovery agent on a natural-language *query*; return the chosen source. + + *run_agent* should be a search-enabled runner (``real_agent_runner(with_search=True)``); + *work_dir* is a scratch directory for the agent's tools. + """ + result = DiscoveryResult() + work_dir.mkdir(parents=True, exist_ok=True) + task = ( + f"Find a benchmark for this request and pick the single best one:\n\n{query}\n\n" + "Search across GitHub / HuggingFace / arXiv, judge the candidates, and end with the " + "JSON block described in your instructions." + ) + try: + result.transcript = await run_agent( + cwd=work_dir, system_prompt=DISCOVERY_SYSTEM_PROMPT, task=task, max_turns=max_turns) + except Exception as exc: # noqa: BLE001 — surface agent/provider errors + result.notes.append(f"agent run failed: {exc}") + return result + + choice = _extract_json(result.transcript) + if choice is None: + result.notes.append("no JSON choice found in the agent's reply") + return result + result.choice = choice + if not result.url: + result.notes.append(f"no source url chosen: {choice.get('why', '(no reason given)')}") + return result + result.ok = True + return result + + +def real_agent_runner( + *, + with_search: bool = False, + ask_user: bool = False, + provider: str | None = None, + model: str | None = None, + api_key: str | None = None, + base_url: str | None = None, +) -> AgentRunner: + """Build the real :class:`Agent`-backed runner. Needs a configured provider / API key. + + With ``with_search=True`` the agent also gets keyless web search + fetch tools + (alphaXiv + Jina) so it can browse for benchmarks. With ``ask_user=True`` it gets a + console-backed :class:`~arbor.zoo.ask_tool.ConsoleAskUserTool` so it can put a genuinely + human decision (e.g. which implementation is the baseline) to the user at the terminal — + only enable this when stdin is interactive. Heavy imports are deferred so importing + :mod:`arbor.zoo` stays light. + """ + async def _run(*, cwd: Path, system_prompt: str, task: str, max_turns: int) -> str: + from arbor.core import Agent, AgentConfig, create_provider + from arbor.core.tools import get_all_tools + + # Only pass provider fields that were set, so AgentConfig's defaults (which read + # the env / user config) apply when they're omitted. + llm_kw = {k: v for k, v in + {"provider": provider, "model": model, "api_key": api_key, + "base_url": base_url}.items() if v is not None} + cfg = AgentConfig(cwd=str(cwd), max_turns=max_turns, auto_git=False, **llm_kw) + prov = create_provider(cfg) + tools = list(get_all_tools(cwd=str(cwd), config=cfg)) + if ask_user: + from .ask_tool import ConsoleAskUserTool + tools.append(ConsoleAskUserTool(cwd=str(cwd))) + if with_search: + from arbor.coordinator.config import SearchConfig + from arbor.core.tools.web.factory import ( + build_web_search_tool, + build_web_visit_tool, + ) + sc = SearchConfig(builtin_backend="alphaxiv", visit_backend="jina") + for t in (build_web_search_tool(sc, cwd=str(cwd)), + build_web_visit_tool(sc, cwd=str(cwd))): + if t is not None: + tools.append(t) + agent = Agent(provider=prov, tools=tools, system_prompt=system_prompt, config=cfg) + return await agent.run(task) + + return _run diff --git a/src/zoo/ask_tool.py b/src/zoo/ask_tool.py new file mode 100644 index 0000000..54c80fb --- /dev/null +++ b/src/zoo/ask_tool.py @@ -0,0 +1,104 @@ +"""A console-backed AskUser tool for the collection agents. + +The discovery and bring-up agents (:mod:`arbor.zoo.agent_stages`) sometimes hit a +choice that is genuinely the *user's* to make — most often **which implementation to +treat as the baseline** when a repo ships both its own headline method and simpler +references (direct generation, naive RAG, an earlier system). Rather than guess, the +agent calls this tool and the question is put to the human at the terminal. + +This is deliberately *not* the Coordinator's :class:`~arbor.coordinator.tools.AskUserTool`: +that one talks to a live UI through an event bus (``IdeaTree``/``await_user_decision``) for +the async research loop. The collection flow runs in a plain CLI (``asyncio.run`` from a +Typer command, the user sitting at the prompt), so a direct console round-trip is the right +fit. The console read is injectable (``ask=``) so the tool is testable without real stdin, +and it is only added to the toolset when stdin is a TTY — in non-interactive runs the agent +never has it and so never stalls. +""" + +from __future__ import annotations + +import asyncio +from typing import Any, Callable + +from ..core.tools.base import Tool + +# A console reader: (question, options) -> the user's answer, or None if they declined / +# no input is available. Injectable so tests don't touch real stdin. +AskFn = Callable[[str, list[str]], "str | None"] + + +def _console_ask(question: str, options: list[str]) -> str | None: + """Default reader: print the question (and any options) and read one line of stdin.""" + import typer + + typer.secho("\n┃ the collection agent needs your input:", fg=typer.colors.MAGENTA, bold=True) + typer.secho(f"┃ {question}", fg=typer.colors.MAGENTA) + if options: + for i, opt in enumerate(options, 1): + typer.echo(f" {i}. {opt}") + typer.echo(" (type a number to pick one, or write your own answer)") + try: + raw = input("your answer> ").strip() + except (EOFError, KeyboardInterrupt): + return None + if not raw: + return None + if options and raw.isdigit(): + idx = int(raw) + if 1 <= idx <= len(options): + return options[idx - 1] + return raw + + +class ConsoleAskUserTool(Tool): + """Ask the human at the terminal for a decision, then return their answer.""" + + name = "AskUser" + description = ( + "Ask the human operator a question and wait for their answer at the terminal.\n\n" + "Use this ONLY for a choice that is genuinely the user's to make and that you " + "cannot settle from the repo, the paper, or your tools. The most important case: " + "when a repo ships both its own proposed method (the SOTA system) and simpler " + "baselines (direct generation, naive RAG, an earlier system), ask which one to " + "treat as the baseline — the baseline is the starting point Arbor will optimize, " + "not necessarily the repo's headline method. Do NOT use this for routine progress " + "updates or decisions you can make yourself.\n\n" + "If no answer comes back, you are told to proceed on your best assumption — never " + "block waiting on a reply, and never ask the same thing twice." + ) + input_schema: dict[str, Any] = { + "type": "object", + "properties": { + "question": { + "type": "string", + "description": "The question to ask (be specific and self-contained).", + }, + "options": { + "type": "array", + "items": {"type": "string"}, + "description": "Optional suggested choices. Omit for a free-form answer.", + }, + }, + "required": ["question"], + } + # Not read-only: serializing this in the agent loop keeps two questions from being + # put to the human at once. The collection agents run with auto_git off, so no commit. + is_read_only = False + + def __init__(self, *, cwd: str, workspace_dir: str | None = None, ask: AskFn | None = None): + super().__init__(cwd=cwd, workspace_dir=workspace_dir) + self._ask: AskFn = ask or _console_ask + + async def execute(self, **kwargs: Any) -> str: + question = (kwargs.get("question") or "").strip() + if not question: + return "Error: 'question' is required." + options = [str(o) for o in (kwargs.get("options") or [])] + # The reader blocks on stdin; run it off the event loop. + answer = await asyncio.to_thread(self._ask, question, options) + if not answer or not answer.strip(): + return ( + "No answer was provided. Proceed with your best assumption and state it " + "explicitly in your final output — do not ask this again." + ) + return f"User replied: {answer.strip()}" diff --git a/tests/test_zoo_ask_tool.py b/tests/test_zoo_ask_tool.py new file mode 100644 index 0000000..3a7d157 --- /dev/null +++ b/tests/test_zoo_ask_tool.py @@ -0,0 +1,63 @@ +"""Tests for the console-backed AskUser tool used by the collection agents. + +No LLM, no real stdin — the console reader is injected so the tool's contract (what it +returns to the agent for a given human answer) is exercised deterministically. +""" + +from __future__ import annotations + +import asyncio + +from arbor.zoo import ConsoleAskUserTool + + +def _run(tool: ConsoleAskUserTool, **kwargs) -> str: + return asyncio.run(tool.execute(**kwargs)) + + +def test_returns_user_answer() -> None: + tool = ConsoleAskUserTool(cwd=".", ask=lambda q, opts: "run_naive_rag.py") + assert _run(tool, question="which baseline?") == "User replied: run_naive_rag.py" + + +def test_passes_question_and_options_to_reader() -> None: + seen: dict = {} + + def ask(question: str, options: list[str]) -> str: + seen["q"], seen["opts"] = question, options + return options[0] + + tool = ConsoleAskUserTool(cwd=".", ask=ask) + out = _run(tool, question="pick the baseline", options=["a.py", "b.py"]) + assert seen == {"q": "pick the baseline", "opts": ["a.py", "b.py"]} + assert out == "User replied: a.py" + + +def test_no_answer_tells_agent_to_proceed() -> None: + # User declined / EOF / non-interactive: the reader returns None. + tool = ConsoleAskUserTool(cwd=".", ask=lambda q, opts: None) + out = _run(tool, question="which baseline?") + assert "best assumption" in out and "do not ask this again" in out + + +def test_blank_answer_treated_as_no_answer() -> None: + tool = ConsoleAskUserTool(cwd=".", ask=lambda q, opts: " ") + assert "best assumption" in _run(tool, question="which baseline?") + + +def test_missing_question_errors() -> None: + tool = ConsoleAskUserTool(cwd=".", ask=lambda q, opts: "x") + assert _run(tool, question=" ").startswith("Error:") + + +def test_answer_is_stripped() -> None: + tool = ConsoleAskUserTool(cwd=".", ask=lambda q, opts: " run_direct_gen.py \n") + assert _run(tool, question="which?") == "User replied: run_direct_gen.py" + + +def test_tool_schema_shape() -> None: + tool = ConsoleAskUserTool(cwd=".") + schema = tool.to_api_schema() + assert schema["name"] == "AskUser" + assert schema["input_schema"]["required"] == ["question"] + assert tool.is_read_only is False diff --git a/tests/test_zoo_bringup.py b/tests/test_zoo_bringup.py new file mode 100644 index 0000000..178c75f --- /dev/null +++ b/tests/test_zoo_bringup.py @@ -0,0 +1,111 @@ +"""Tests for the bring-up agent stage (``arbor.zoo.agent_stages``). + +The agent run is injected, so the orchestration (agent writes files → eval runs → score +parsed → verify) is exercised without a live LLM. A fake runner stands in for the agent. +""" + +from __future__ import annotations + +import asyncio +from pathlib import Path + +from arbor.zoo import bringup, real_agent_runner + +_PROVENANCE = ( + "# Provenance\n\n## Source\nx\n## Setup & environment\nx\n## Baseline\nx\n" + "## Contamination assessment\nx\n## Caveats\nx\n" +) +_README = "# demo\n\nA demo benchmark.\n\n## The task\nx\n## Metric\nx\n" + + +def _fake_runner(files: dict[str, str]): + """A stand-in agent: writes the files a real bring-up agent would, returns a transcript.""" + async def _run(*, cwd: Path, system_prompt: str, task: str, max_turns: int) -> str: + for rel, content in files.items(): + p = cwd / rel + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(content) + return "bring-up done" + return _run + + +def _good_files(score: str = "1.0") -> dict[str, str]: + return { + "eval.sh": f"#!/usr/bin/env bash\necho 'score: {score}'\n", + "solution.py": "# baseline\n", + "README.md": _README, + "PROVENANCE.md": _PROVENANCE, + } + + +def test_bringup_success(tmp_path: Path) -> None: + pack = tmp_path / "p" + pack.mkdir() + res = asyncio.run(bringup(pack, run_agent=_fake_runner(_good_files("0.5")))) + assert res.dev_score == 0.5 + assert res.ran + assert res.ok + assert not [r for r in res.verify if r.status == "fail"] + assert res.transcript == "bring-up done" + + +def test_bringup_no_score_still_drafts(tmp_path: Path) -> None: + # A non-running eval is a runnable draft, not a failure (we don't force-run): ok stays + # True (artifacts verify), but ran is False and a note explains why. + pack = tmp_path / "p" + pack.mkdir() + files = _good_files() + files["eval.sh"] = "#!/usr/bin/env bash\necho 'nothing here'\n" + res = asyncio.run(bringup(pack, run_agent=_fake_runner(files))) + assert res.dev_score is None + assert not res.ran + assert res.ok + assert any("runnable draft" in n for n in res.notes) + + +def test_bringup_failed_verify_is_incomplete(tmp_path: Path) -> None: + pack = tmp_path / "p" + pack.mkdir() + files = _good_files() + del files["PROVENANCE.md"] # missing PROVENANCE → verify fails + res = asyncio.run(bringup(pack, run_agent=_fake_runner(files))) + assert not res.ok + assert any(r.status == "fail" for r in res.verify) + + +def test_bringup_threads_instruction_and_plan(tmp_path: Path) -> None: + # The user's request and the baseline plan must reach the agent's task text. + pack = tmp_path / "p" + pack.mkdir() + seen: dict = {} + + async def _spy(*, cwd: Path, system_prompt: str, task: str, max_turns: int) -> str: + seen["task"] = task + for rel, content in _good_files().items(): + (cwd / rel).write_text(content) + return "ok" + + asyncio.run(bringup( + pack, run_agent=_spy, + instruction="climb GPQA, design a self-consistency method", + baseline_plan={"source": "implement", "detail": "self-consistency over 5 samples"}, + )) + assert "self-consistency method" in seen["task"] + assert "implement" in seen["task"] and "5 samples" in seen["task"] + + +def test_bringup_surfaces_agent_error(tmp_path: Path) -> None: + pack = tmp_path / "p" + pack.mkdir() + + async def _boom(*, cwd, system_prompt, task, max_turns): + raise RuntimeError("provider exploded") + + res = asyncio.run(bringup(pack, run_agent=_boom)) + assert not res.ok + assert any("agent run failed" in n for n in res.notes) + + +def test_real_agent_runner_is_callable() -> None: + # Construction is cheap and import-light; running it needs a live provider. + assert callable(real_agent_runner()) diff --git a/tests/test_zoo_collect.py b/tests/test_zoo_collect.py index 7aa7173..2e38a6b 100644 --- a/tests/test_zoo_collect.py +++ b/tests/test_zoo_collect.py @@ -133,6 +133,51 @@ def test_cli_add(cache: Path, tmp_path: Path) -> None: assert "still to do" in result.output -def test_cli_add_rejects_bad_spec(tmp_path: Path) -> None: - result = CliRunner().invoke(app, ["benchmark", "add", "!!!", "--name", "x"]) +def test_cli_add_url_without_name_errors(cache: Path, tmp_path: Path) -> None: + # A URL/path spec skips discovery; without --name (and none to infer) it errors. + repo = _make_repo(tmp_path) + result = CliRunner().invoke(app, ["benchmark", "add", str(repo)]) assert result.exit_code == 2 + + +def test_cli_add_query_path_brings_up(cache: Path, tmp_path: Path, monkeypatch) -> None: + # A natural-language request: discovery + bring-up are faked (no LLM/network), but the + # spine in between (acquire + scaffold) runs for real, and the original request + + # baseline plan must be threaded into bring-up. + from arbor.cli.commands import benchmark_cmd + from arbor.zoo import BringupResult, DiscoveryResult + + repo = _make_repo(tmp_path) + zoo = tmp_path / "zoo" + captured: dict = {} + + async def fake_discover(query, *, run_agent, work_dir, max_turns): + captured["query"] = query + return DiscoveryResult( + choice={"name": "demo", "source": {"kind": "git", "url": str(repo)}, + "metric": "accuracy, higher better", "baseline": "naive rag", + "baseline_plan": {"source": "implement", "detail": "naive rag baseline"}, + "why": "fits"}, + ok=True, + ) + + async def fake_bringup(pack_dir, *, run_agent, materials_dir=None, instruction="", + baseline_plan=None, max_turns=40, eval_timeout=600): + captured["instruction"] = instruction + captured["plan"] = baseline_plan + return BringupResult(ok=True, ran=False, notes=["runnable draft ready"]) + + monkeypatch.setattr(benchmark_cmd, "discover", fake_discover) + monkeypatch.setattr(benchmark_cmd, "bringup", fake_bringup) + + result = CliRunner().invoke( + app, ["benchmark", "add", "get me the WebThinker GPQA benchmark", "--yes", + "--dest", str(zoo)] + ) + assert result.exit_code == 0, result.output + assert (zoo / "demo" / "README.md").exists() + assert captured["query"] == "get me the WebThinker GPQA benchmark" + # the user's original words reach bring-up (so a baseline can be implemented to them) + assert captured["instruction"] == "get me the WebThinker GPQA benchmark" + assert captured["plan"] == {"source": "implement", "detail": "naive rag baseline"} + assert "runnable draft ready" in result.output diff --git a/tests/test_zoo_discovery.py b/tests/test_zoo_discovery.py new file mode 100644 index 0000000..a78dccc --- /dev/null +++ b/tests/test_zoo_discovery.py @@ -0,0 +1,81 @@ +"""Tests for the discovery agent stage (``arbor.zoo.agent_stages.discover``). + +The agent run is injected, so the orchestration (query → agent → parse the chosen source) +is exercised without a live LLM or live web search. +""" + +from __future__ import annotations + +import asyncio +from pathlib import Path + +from arbor.zoo import discover + + +def _fake_runner(reply: str): + async def _run(*, cwd: Path, system_prompt: str, task: str, max_turns: int) -> str: + return reply + return _run + + +_CHOICE = ( + "I searched GitHub and arXiv. KernelBench fits best.\n\n" + '```json\n' + '{"name": "kernelbench", "source": {"kind": "git", ' + '"url": "https://github.com/ScalingIntelligence/KernelBench"}, ' + '"metric": "speedup, higher is better", "baseline": "torch reference in repo", ' + '"why": "ships an eval + baseline, real headroom"}\n' + '```\n' +) + + +def test_discover_picks_a_source(tmp_path: Path) -> None: + res = asyncio.run(discover("a GPU kernel optimization benchmark", + run_agent=_fake_runner(_CHOICE), work_dir=tmp_path / "w")) + assert res.ok + assert res.url == "https://github.com/ScalingIntelligence/KernelBench" + assert res.name == "kernelbench" + assert res.choice and res.choice["metric"].startswith("speedup") + + +def test_discover_no_json_is_not_ok(tmp_path: Path) -> None: + res = asyncio.run(discover("something", run_agent=_fake_runner("I couldn't find anything useful."), + work_dir=tmp_path / "w")) + assert not res.ok and res.url is None + assert any("no JSON" in n for n in res.notes) + + +def test_discover_captures_baseline_plan(tmp_path: Path) -> None: + reply = ( + '```json\n' + '{"name": "gpqa", "source": {"kind": "hf", "url": "Idavidrein/gpqa"}, ' + '"metric": "accuracy, higher is better", "baseline": "naive RAG", ' + '"baseline_plan": {"source": "implement", "detail": "write a naive RAG baseline"}, ' + '"why": "user wants to climb GPQA"}\n```' + ) + res = asyncio.run(discover("climb GPQA with a naive RAG baseline", + run_agent=_fake_runner(reply), work_dir=tmp_path / "w")) + assert res.ok + assert res.baseline_plan == {"source": "implement", "detail": "write a naive RAG baseline"} + + +def test_discover_baseline_plan_defaults_empty(tmp_path: Path) -> None: + # The KernelBench reply has no baseline_plan → property is an empty dict, not None. + res = asyncio.run(discover("a kernel benchmark", + run_agent=_fake_runner(_CHOICE), work_dir=tmp_path / "w")) + assert res.baseline_plan == {} + + +def test_discover_null_source_is_not_ok(tmp_path: Path) -> None: + reply = '```json\n{"name": null, "source": null, "why": "nothing suitable found"}\n```' + res = asyncio.run(discover("obscure", run_agent=_fake_runner(reply), work_dir=tmp_path / "w")) + assert not res.ok + assert any("no source url" in n for n in res.notes) + + +def test_discover_surfaces_agent_error(tmp_path: Path) -> None: + async def _boom(*, cwd, system_prompt, task, max_turns): + raise RuntimeError("provider missing") + + res = asyncio.run(discover("x", run_agent=_boom, work_dir=tmp_path / "w")) + assert not res.ok and any("agent run failed" in n for n in res.notes)