feat(zoo): query-driven collection — discovery + bring-up agents#35
Merged
Conversation
The collection spine acquires + scaffolds a draft; this adds the first agent-driven stage — baseline bring-up. `arbor.zoo.agent_stages.bringup()` runs one agent in the draft folder to make the baseline actually run (install deps, get the reference working, wrap an eval that prints `score:`, write the README + PROVENANCE), then checks its work by running the eval (dev) and the structural verifier. - Built on the core `Agent` runtime (`real_agent_runner` = create_provider + get_all_tools + Agent.run), the same machinery the executor uses; stays a standalone flow, never wired into the Coordinator loop. - The agent run is behind an injected `run_agent` callable, so the orchestration (write → eval → score → verify) is unit-tested with a fake runner — no live LLM. The real runner needs a configured provider; validating its *reasoning* needs live iteration. - CLI: `arbor benchmark add … --bringup [--max-turns N]` runs it after scaffolding and reports the dev score + verify; errors clearly if no provider is configured. Tests: bring-up success / no-score / failed-verify / agent-error paths. ruff + mypy + full suite green.
…e → pick)
Adds Stage 0: natural-language query → a discovery agent that searches GitHub /
HuggingFace / arXiv (keyless alphaXiv + Jina web tools), judges candidates (has an
eval + baseline? headroom? compute fit? license?), and proposes the single best source.
- `arbor.zoo.agent_stages.discover()` runs a search-enabled agent and parses its chosen
source from a trailing JSON block (`{name, source:{kind,url}, metric, baseline, why}`).
`real_agent_runner(with_search=True)` wires the keyless web search + fetch tools.
- CLI: `arbor benchmark add "<query>"` now resolves a query via discovery, shows the
chosen repo + rationale, asks for confirmation (`--yes` to skip), then runs the existing
acquire → scaffold (→ `--bringup`) flow. A URL still skips discovery.
- The agent run is behind the injected `run_agent` seam, so the orchestration (query →
parse choice; success / no-json / null-source / error paths) is unit-tested with a fake
runner — no live LLM or web. The live behavior needs an API key + prompt iteration.
Docs (overview, EN + zh) updated to the query-driven flow. ruff + mypy + full suite +
mkdocs --strict green.
ignorejjj
changed the title
benchmark add --bringup)…to 100
Two fixes found by live-testing discovery on the WebThinker case:
1. Config inheritance — the collection agents built a bare AgentConfig, which defaulted
to claude-sonnet and ignored ~/.arbor/config.yaml. They now run on the user's
configured provider (e.g. openai-oauth / gpt-5.5), via a `_user_runner` helper that
reads `llm_defaults()` in the CLI layer (keeps arbor.zoo independent of arbor.cli).
2. Turn budget — discovery over-explored and got cut off at 10–18 turns before emitting
its JSON choice (and Agent.run returns only a stub on max_turns). `--max-turns` now
defaults to 100 (discover() default raised to match) so the agent concludes naturally.
Also nudged the discovery prompt to stop and answer once it has identified a suitable
repo rather than exhaustively cloning/grepping.
Verified live: `discover("WebThinker benchmarks/baselines")` now returns ok=True with the
RUC-NLPIR/WebThinker repo and a correct metric/baseline breakdown (GPQA/GAIA/WebWalkerQA/
HLE + Search-o1 / naive-RAG / direct-gen baselines). ruff + mypy + full suite green.
Turn `arbor benchmark add "<request>"` into one interactive flow that produces a single runnable draft task, instead of guessing. - console AskUser tool (ConsoleAskUserTool) so the discovery / bring-up agents can ask the user the decisions that are genuinely theirs — which dataset, which baseline — at the terminal; TTY-gated so CI / piped runs never block - discover: ask which dataset for a work that uses several; record a baseline_plan (harvest | implement | web) - bring-up: baseline may be harvested from the repo, implemented to the user's described method, or found online; thread the user's original request into the task; produce a runnable draft and do NOT force-run the eval (ok = artifacts verify; a new `ran` flag marks whether it actually scored) - a natural-language request runs find → acquire → bring-up by default; a bare URL/HF spec still does the single deterministic collect (+ optional --bringup) All orchestration tested via injected fake runners (no LLM/network).
- README: add a "Collect a benchmark from a request" section - zoo-overview EN/zh: rewrite the collect bullet, status line, entry-point row - roadmap EN/zh: mark `arbor benchmark add` as built (discovery + interactive bring-up), reasoning still maturing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes
arbor benchmark addquery-driven: describe what you want in natural language and Arbor searches, judges, fetches, and brings up a benchmark.Flow
arbor benchmark add "<query>" [--bringup]:--yesto skip).--bringup) — an agent makes the baseline run, writes README + PROVENANCE, then the code runs the eval (dev) + structural verify to check the result.A repo URL skips discovery and goes straight to acquire.
What's solid vs. what needs live iteration
run_agent, no live LLM/web): the whole orchestration — discovery's query→parse-choice paths (ok / no-json / null-source / error) and bring-up's write→eval→score→verify paths. Tool wiring uses the realAgentruntime (create_provider+get_all_tools+ keyless web factories), the same machinery the executor/search-agent use. Never wired into the Coordinator loop (§2.1).Tests
test_zoo_discovery.py+test_zoo_bringup.pycover the agent-stage orchestration via fake runners. ruff + mypy + full suite +mkdocs build --strictgreen.