From b84571c3d7ee257503a4954d44e69d0090a5fa1a Mon Sep 17 00:00:00 2001 From: ignorejjj Date: Sun, 28 Jun 2026 16:21:35 +0800 Subject: [PATCH] docs(roadmap): rewrite around open problems and preliminary ideas MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Recast the roadmap as a few categories of open problems, each paired with the team's current preliminary thinking, in a leaner and more restrained voice. Reorganize into three directions — core capability, external resources, and user experience — covering long-horizon process management, idea-quality assessment, self-evolution, composite optimization objectives, stronger agentic research ability, stateful long-horizon environments, automated environment collection, a worked-examples gallery, and better single-run export/comparison. EN and ZH kept in sync. --- docs/roadmap.md | 237 ++++++++++----------------------------------- docs/roadmap.zh.md | 189 +++++++----------------------------- 2 files changed, 86 insertions(+), 340 deletions(-) diff --git a/docs/roadmap.md b/docs/roadmap.md index 1aa95f1..f46c463 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1,215 +1,78 @@ # Roadmap -This is a direction document, not a release schedule. It lists the directions we -want to push on and a few concrete ideas under each. Items move, merge, or get -dropped as we learn. +This document records the directions we plan to push Arbor in: a few categories of open +problems and our current preliminary thinking on each. These problems and ideas will be +adjusted, merged, or dropped as the team's understanding deepens — contributions and +suggestions are welcome. -## Positioning +## Direction 1 — Core capability -A few autonomous-research systems already overlap with parts of this plan. -[AutoSOTA](https://github.com/tsinghua-fib-lab/AutoSOTA) is the closest: a -closed-loop system that reviews literature, edits research code, runs -experiments, and maintains a per-paper leaderboard across 100+ papers organized -by domain, with reproduced baselines and tamper-protected evals. +### 1. Better long-run process management -That overlap is real, so we are explicit about where Arbor is *not* trying to be a -second "auto-optimize published papers" leaderboard: +Arbor currently maintains the entire hypothesis tree with a single Coordinator, which can +degrade over longer research horizons. We plan to improve its context management so that +the key information accumulated during a long run isn't gradually distorted or lost. -- The value we want to build is a **reusable, checkable benchmark format** anyone - can re-run, not a catalog of our own wins. -- We lean on **held-out discipline** — a change is kept only if it clears a margin - on a protected test split — over "produced a higher number". -- We treat the benchmark collection first as **our own regression harness** for the - Coordinator/Executor, and only second as something public-facing. +### 2. Idea quality assessment ---- - -## Direction 1 — Core functionality - -### 1.1 Search and literature-grounded ideation ✅ *(shipped)* - -Originally search lived in an isolated `SearchAgent`; idea generation and the -Coordinator could not search the open web. That kept benchmark runs fair — the -system couldn't copy a finished idea off the web — but it was too strict for real -research, where you read prior work before proposing a direction. - -Shipped (default-off, so benchmark runs stay fair). See the -[Search & External Knowledge](search.md) guide for usage: - -- **Grounded ideation** (`search.grounded_ideation`, default off) — the - coordinator gets a `ResearchSearch` tool during ideation (intents: - related-work / survey / lookup / explore). -- **Separation, not prohibition** — the grounding lane and the novelty-audit lane - don't share state; a source that shapes an idea is recorded on the node's - `grounding` field, separate from the audit's `related_work`. -- **Pluggable backends** behind `search.backends`, fanned out and merged: - alphaXiv + Jina (keyless), Serper + Exa REST (keyed), **Exa via MCP** (keyless), - and the legacy self-hosted endpoint. Keyless page reading via the Jina reader - (raw-`requests` fallback), no browse endpoint needed. -- **Full-text and PDF ingestion** — the grounded lane reads with a larger token - budget (`research_visit_tokens`) and parses PDFs, so a model can read a paper's - method/results sections, not just its abstract. - -Still open: enforcing the per-search round/visit caps as a hard cost bound, and -surfacing per-run search cost (tracked under [1.3](#13-cost-and-scheduling)). - -### 1.2 Evaluation discipline ✅ *(shipped)* - -Shipped: - -- **Split provenance** — every score is tagged with the split it came from - (`dev`/`test`) at the data-model level and rendered labeled in REPORT.md, the - CLI dashboard, and the WebUI. The verified B_test score is recorded on the node - and trunk meta automatically at merge. -- **Tamper-proof evals** — protected paths are hash-verified at runtime, not only - at merge. Each executor worktree gets a SHA-256 manifest of its protected files - plus best-effort OS read-only; any mid-run change discards the node's dev score - and blocks the merge (emitting `eval.protected_tamper`). This closes the gap - where an executor could inflate B_dev by writing to `data/`/`evaluation/`. -- **Contamination checks** — a declarative `eval_contract.contamination` block - (release date, `is_public`, canaries) drives a non-blocking preflight warning - and an INIT-time probe (`eval.contamination_assessed`, recorded in tree meta). - The declarative heuristic + canary scan ship now; an LLM membership-inference - probe is a planned follow-up. - -See the [Plugins](plugins.md) guide for the `contamination` block and runtime -protected-path enforcement. - -### 1.3 Cost and scheduling - -- Budget tiers (smoke → pilot → full) so larger sweeps stay predictable. -- Per-backend / per-run cost accounting surfaced before a run starts, not after. - ---- +The novelty and feasibility of a proposed idea currently rest mainly on the base model's +own ability, with no explicit external feedback to supervise it. We are considering +introducing a critic-style role that gives candidate ideas an independent assessment, to +raise the quality of this step. -## Direction 2 — External resources - -### 2.1 Benchmark zoo, organized by domain 🚧 *(format + tooling shipped; collection growing)* - -A curated collection of tasks in one standard format, grouped by domain (e.g. -vision, NLP, time series, optimization), each using a published paper's result as -the baseline to beat. It lives in the repo as `arbor-zoo/`, one folder per -benchmark, and serves first as Arbor's own regression harness — not a leaderboard -of our wins. - -Shipped — the Task Pack format, the verifier, and a first reference pack. See the -[Benchmark Zoo](zoo.md) guide for the full spec and the verifier's check list: - -- **Task Pack format**, one folder per benchmark, with the contract carried in the - **README front-matter** (metric, dev/test split, baseline, edit surface) — there - is *no separate manifest file*. Alongside it: a runnable baseline (e.g. - `solution.py`), a protected eval entrypoint (`eval.sh` / `eval.py`) that prints - exactly one `score: ` line for `dev`|`test`, an optional protected - `task.py` (deterministic `generate_problem` + an *independent* `is_solution` - verifier, so "fast but wrong" can't score), and a `PROVENANCE.md` card (source, - license, setup/environment, baseline reproduction, contamination, caveats). -- **`arbor benchmark verify`** gates a pack: front-matter + `PROVENANCE.md` parse - and are complete, the eval emits a parseable score on dev and test, the baseline - reproduces its claimed number, dev/test are held out, protected paths hold, and - the eval is deterministic/offline. Exits non-zero on any failure — an unverified - pack does not enter the zoo. **`arbor benchmark list`** indexes a zoo directory - (a plain index, not a leaderboard). -- **Reference pack + scaffolding**: - [`algotune_knn`](https://github.com/RUC-NLPIR/Arbor/tree/main/arbor-zoo/algotune_knn) - (verified) and a `_template` to copy. Folders prefixed `_` are skipped by tooling. - -Still open: - -- **Grow the collection** to 3–5 high-quality, human-checked packs across distinct - task shapes, using `algotune_knn` as the reference. Cap on quality, not count. -- **`arbor benchmark add`** — semi-automatic conversion: from a one-line request the - agent finds the dataset, asks (on an interactive terminal) which dataset to use and - where the baseline comes from (harvest an existing one / implement the method you - described / find one online), and brings up a runnable draft — gated behind the - verifier and a human accept step (draft-automatic, accept-verified — never - auto-accepted). The baseline-implementing agent stays separate from the loop that - later optimizes it, so evaluation isn't self-certifying. *(Built: discovery + - interactive bring-up; bring-up reasoning still maturing.)* -- **Lower a pack into a [plugin](plugins.md)** for one-line retargeting — the - front-matter contract reuses the `plugin` vocabulary (`eval_contract` / - `protected_paths`), so it should fall out with little rework (pairs with 2.2). - - -### 2.2 Plugin gallery - -More worked domain plugins beyond `mle_kaggle`, paired with the Task Packs above, -so retargeting Arbor to a domain is a one-line `plugin:` change. - -### 2.3 Search backends ✅ *(shipped)* - -The pluggable backends from Direction 1 (alphaXiv, Jina, Serper, Exa REST, Exa -via MCP, self-hosted endpoint) are also external resources users wire in once and -reuse across runs. See [1.1](#11-search-and-literature-grounded-ideation-shipped) -and the [Search guide](search.md). - ---- +### 3. Self-evolution -## Direction 3 — User presentation +Arbor currently optimizes around a single benchmark, and the experience it gains is hard +to transfer to similar or same-domain tasks. We plan to add the ability to export research +trajectories, and on top of that to extract and consolidate reusable skills and experience. -### 3.1 Zoo / leaderboard view +### 4. Support for composite optimization objectives -A browsable page over the benchmark zoo: per domain, show the paper baseline, the -Arbor result, the gain, and the exact command to reproduce it. The point is -reproducibility — every row is something a reader can re-run — not a scoreboard. +Arbor's optimization objective is currently limited to a scalar score. We are considering +support for more composite objective forms — such as rubric- and LLM-judge-based scoring +and multi-objective optimization — and the comparability and anti-gaming problems such +objectives bring. -### 3.2 Run comparison +### 5. Stronger agentic research ability -- Diff two runs of the same benchmark. -- Compare Idea Trees across models/providers on the same task. +Let the agent autonomously gather more experiment information as a run progresses, rather +than being confined to a pre-defined metric, giving subsequent decisions a fuller basis. -### 3.3 Reports and export - -Build on today's `REPORT.md` and HTML export with citations (the grounding -sources behind each idea) and a per-run cost breakdown. - ---- - -## Direction 4 — Adoption, DX & community - -The first three directions grow what Arbor *does*. This one lowers the barrier -between a curious visitor and a first successful run, and makes the project's -momentum visible. The goal is a shorter path to the "aha" moment, not more -surface area. - -### 4.1 Live, zero-install demo +## Direction 2 — External resources -`arbor replay --demo --html` already emits a self-contained, dependency-free page -of a real run. Publish that page (e.g. on GitHub Pages alongside the docs) and -link it from the top of the README, so a visitor can watch the hypothesis tree -grow **without installing anything**. One recorded run, refreshed when the -dashboard changes. +### 1. Extend to more diverse long-horizon optimization settings -### 4.2 Examples gallery +The current task format assumes static optimization (a fixed, stateless environment). We +plan to add tasks that carry environment state, to support optimizing agent harnesses in +concrete settings such as office work. -Today `examples/algotune_knn` is the only end-to-end task a newcomer can run. -Grow this into a small gallery so different audiences can self-identify, each -with a copy-pasteable command and a short asciinema recording: +### 2. Automated environment collection -- a Kaggle / MLE-style task (pairs with the `mle_kaggle` plugin), -- a prompt / harness-engineering task, and -- a small training / fine-tuning task. +Collect and grow a more comprehensive set of benchmark scenarios, covering a range of LLM +and agent evaluation benchmarks, so that Arbor can support a wider range of optimization +settings. -Keep the bar at "runs in minutes on a laptop or a free key", reusing the -zero-config discipline of `algotune_knn`. +## Direction 3 — User experience -### 4.3 Regular releases and a changelog +### 1. More worked examples and scenarios -Versions are already derived from `v*` tags via setuptools-scm, so the cost of a -release is low. Cut a release on a predictable cadence with human-readable notes, -and keep a `CHANGELOG.md` (or GitHub Releases as the source of truth) so the -project's progress is legible from the outside. +The zero-install demo has shipped, but `algotune_knn` is still the only example a newcomer +can run end to end. We plan to grow it into a small examples gallery covering different +task types and audiences (e.g. Kaggle / MLE, prompt and harness engineering, small-scale +training), each with a copy-pasteable command and a short recording, keeping the barrier to +"runs in minutes on a laptop or a free API key". -### 4.4 Public roadmap and issue board +### 2. Better export and presentation of a single run -Surface this document as a public board (GitHub Projects) and tag tracked work -with `good first issue` / `help wanted`. Several threads are already -contribution-shaped — growing the [benchmark zoo](#21-benchmark-zoo-organized-by-domain-format-tooling-shipped-collection-growing) -to 3–5 packs ([2.1](#21-benchmark-zoo-organized-by-domain-format-tooling-shipped-collection-growing)) -and adding domain plugins ([2.2](#22-plugin-gallery)) — so opening them up turns -readers into contributors. +There's already `REPORT.md`, HTML export, a live dashboard, and a read-only WebUI — what's +missing is understanding and comparing a single run. We plan to support comparison across +runs (diff multiple runs of the same task, or compare idea trees across models / providers), +add citations (the sources behind each idea) and a cost breakdown to the report, and give +the benchmark collection a reproducibility view where every entry carries a command you can +re-run directly. --- -Have an idea or want to own one of these threads? Open a +Have an idea, or want to own one of these threads? Open a [discussion](https://github.com/RUC-NLPIR/Arbor/discussions) or see [Contributing](contributing.md). diff --git a/docs/roadmap.zh.md b/docs/roadmap.zh.md index 5060c77..ca970db 100644 --- a/docs/roadmap.zh.md +++ b/docs/roadmap.zh.md @@ -1,179 +1,62 @@ # 路线图 -这是一份方向文档，不是发布排期。它列出我们想推进的几个方向，以及每个方向下的几个具体 -思路。条目会随着认识的深入而移动、合并或删除。 +这是一份记录 Arbor 后续优化方向的文档，列出我们关心的几类开放问题及目前的初步设想。 +这些问题与设想会随团队认识的深入而调整、合并或删除，欢迎贡献与建议。 -## 定位 +## 方向一：核心功能优化 -已经有一些自动研究系统和本计划的一部分重叠。最接近的是 -[AutoSOTA](https://github.com/tsinghua-fib-lab/AutoSOTA)：一个闭环系统，会做文献调研、 -改研究代码、跑实验，并维护一个按领域组织、覆盖 100+ 篇论文的 per-paper leaderboard， -带 baseline 复现和防篡改的 eval。 +### 1. 实验进程管理优化 -这种重叠是真实的，所以我们明确：Arbor **不打算**做第二个"自动优化已发表论文"的榜单。 +目前 Arbor 由单一 Coordinator 维护整棵 hypothesis tree，在更长的研究周期下可能出现能力 +退化。我们计划优化其上下文管理，使长程研究中积累的关键信息不致逐步失真或丢失。 -- 我们要做的价值是一个**可复用、可校验、别人能复跑的基准格式**，而不是我们自己战绩的目录。 -- 我们靠 **held-out 纪律**——只有在受保护的 test split 上超过一定 margin 才保留改动—— - 而不是"刷出了更高的数字"。 -- 我们首先把这套基准集合当成 **Arbor 自己的回归 harness**（用于检验 Coordinator/Executor）， - 其次才是对外展示。 +### 2. Idea 质量评估 ---- - -## 方向一 —— 基本功能 - -### 1.1 搜索与文献接地的 ideation ✅ *（已完成）* {#sec-1-1} - -最初搜索被隔离在独立的 `SearchAgent` 里，idea 生成与 Coordinator 不能直接搜索开放网络。 -这让基准运行保持公平——系统无法从网上抄一个成品 idea——但对真实研究太严：真实研究里， -人会先读相关工作再提方向。 - -已落地（默认关，基准运行保持公平），用法见 -[检索与外部知识](search.zh.md)指南： - -- **接地 ideation**（`search.grounded_ideation`，默认关）——ideation 阶段 Coordinator 获得 - `ResearchSearch` 工具（intent：related-work / survey / lookup / explore）。 -- **分离而非禁止**——接地 lane 与新颖性审计 lane 不共享状态；塑造了某个 idea 的来源记录在该 - 节点的 `grounding` 字段，与审计的 `related_work` 分开。 -- **可插拔后端**，在 `search.backends` 后扇出合并：alphaXiv + Jina（免 key）、Serper + Exa REST - （需 key）、**Exa via MCP**（免 key）、以及自托管 endpoint。读取页面经 Jina reader 免 key - （raw-`requests` 兜底），无需 browse 端点。 -- **全文与 PDF 摄取**——接地 lane 用更大的 token 预算（`research_visit_tokens`）读取并解析 PDF， - 让模型能读到论文的方法/结果章节，而不只是摘要。 - -尚未做：把每次检索的轮次/visit 上限做成硬性成本约束，以及给出按 run 的检索成本（归在 -[1.3](#sec-1-3)）。 - -### 1.2 评测纪律 ✅ *（已完成）* - -已完成： - -- **Split 溯源**——每个分数在数据模型层就标注来自哪个 split（`dev`/`test`），并在 - REPORT.md、CLI dashboard 和 WebUI 中带标签渲染。验证过的 B_test 分数在 merge 时自动 - 记录到节点与 trunk meta。 -- **eval 防篡改**——受保护路径在运行中（而不仅 merge 时）做哈希校验。每个 executor 的 - worktree 会拿到其受保护文件的 SHA-256 清单，外加尽力而为的 OS 只读；运行中任何改动都会 - 作废该节点的 dev 分数并阻止 merge（发出 `eval.protected_tamper`）。这堵上了 executor - 通过写 `data/`/`evaluation/` 抬高 B_dev 的口子。 -- **污染自检**——声明式的 `eval_contract.contamination` 块（发布日期、`is_public`、 - canary）驱动一个非阻塞的 preflight 警告和一次 INIT 期探测（`eval.contamination_assessed`， - 记录在 tree meta）。声明式启发式 + canary 扫描现已随包；LLM 成员推断探测作为后续。 - -关于 `contamination` 块与运行时受保护路径强制，见 [Plugins](plugins.md) 指南。 - -### 1.3 成本与调度 {#sec-1-3} - -- 预算分层（smoke → pilot → full），让较大的扫描可预估。 -- 在运行开始前就给出按 backend / 按 run 的成本核算，而不是事后。 - ---- - -## 方向二 —— 外部资源 - -### 2.1 按 domain 划分的 benchmark zoo 🚧 *（格式与工具已完成；集合扩充中）* {#sec-2-1} +目前提出的 idea，其新颖性与可行性主要依赖基础模型自身的能力，缺乏外部的显式反馈进行 +监督。我们考虑引入类似批判者（critic）的角色，对候选 idea 给出独立评估，以提升这一环节 +的质量。 -一个经过筛选、统一格式的任务集合，按领域分组（如 CV、NLP、时序、优化），每个任务用一篇已 -发表论文的结果作为要超越的 baseline。它以 `arbor-zoo/` 放在仓库里，每个基准一个文件夹， -首要用途是 Arbor 自己的回归 harness——而非我们战绩的榜单。 +### 3. 自进化能力 -已完成——Task Pack 格式、校验器，以及第一个参考 pack。完整规格与校验器的检查清单见 -[Benchmark Zoo](zoo.md) 指南： +目前 Arbor 围绕单一 benchmark 进行优化，所得经验难以迁移到相似或同领域的其他任务。我们 +计划增加研究轨迹的导出能力，并在此基础上抽取、整合可复用的技能与经验。 -- **Task Pack 格式**，每个基准一个文件夹，契约写在 **README front-matter** 里（metric、 - dev/test 切分、baseline、编辑面）——*没有单独的清单文件*。同目录还有：可运行的 baseline - （如 `solution.py`）、一个受保护的 eval 入口（`eval.sh` / `eval.py`），对 `dev`|`test` - 各打印恰好一行 `score: `、一个可选的受保护 `task.py`（确定性的 `generate_problem` - + *独立的* `is_solution` 验证器，让“快但错”无法得分），以及一份 `PROVENANCE.md` 卡片 - （来源、license、setup/环境、baseline 复现、污染、注意事项）。 -- **`arbor benchmark verify`** 为一个 pack 把关：front-matter + `PROVENANCE.md` 可解析 - 且完整、eval 在 dev 与 test 上各产出一个可解析的分数、baseline 能复现声称的数字、 - dev/test held out、受保护路径不可写、eval 确定且离线。任何一项失败即非零退出——未经校验 - 的 pack 不进 zoo。**`arbor benchmark list`** 索引一个 zoo 目录（只是索引，不是榜单）。 -- **参考 pack + 脚手架**： - [`algotune_knn`](https://github.com/RUC-NLPIR/Arbor/tree/main/arbor-zoo/algotune_knn) - （已校验）和一个可复制的 `_template`。以 `_` 开头的文件夹被工具跳过。 +### 4. 对复合优化目标的支持 -尚未做： - -- **扩充集合**到 3–5 个高质量、人工核对的 pack，覆盖不同任务形态，以 `algotune_knn` 为 - 参考。质量封顶，不是数量封顶。 -- **`arbor benchmark add`**——半自动转换:从一句话需求出发,agent 找到数据集,在交互终端里 - 询问用户**用哪个数据集、baseline 从哪来**(收割现成的 / 按你描述的方法实现 / 上网找),并产出 - 一个可运行草稿,再由校验器和人工接受这一步把关(起草自动、接受需校验——绝不自动接受)。*实现* - baseline 的 agent 与之后优化它的 loop 分开,使评测不自证。*(已实现:discovery + 交互式 - bring-up;bring-up 的推理仍在打磨。)* -- **把一个 pack 降级成 [plugin](plugins.md)** 以实现一行改写重定向——front-matter 契约 - 复用 `plugin` 词汇（`eval_contract` / `protected_paths`），应能几乎无返工地导出（与 2.2 - 配套）。 - -### 2.2 插件库 {#sec-2-2} - -在 `mle_kaggle` 之外提供更多范例领域插件，与上面的 Task Pack 配对，让把 Arbor 重定向到 -一个领域只需改一行 `plugin:`。 - -### 2.3 搜索 backend ✅ *（已完成）* - -方向一里的可插拔后端（alphaXiv、Jina、Serper、Exa REST、Exa via MCP、自托管 endpoint）同样是 -外部资源——用户接入一次，便可跨 run 复用。见 -[1.1](#sec-1-1)与[检索指南](search.zh.md)。 - ---- - -## 方向三 —— 用户展示 - -### 3.1 Zoo / leaderboard 视图 - -一个可浏览的 benchmark zoo 页面：按领域展示论文 baseline、Arbor 结果、提升，以及复现它的 -确切命令。重点是可复现——每一行都是读者能复跑的东西——而不是一个计分牌。 - -### 3.2 运行对比 - -- 对同一基准的两次 run 做 diff。 -- 在同一任务上跨 model/provider 比较 Idea Tree。 - -### 3.3 报告与导出 - -在今天的 `REPORT.md` 和 HTML 导出基础上，加入引用（每个 idea 背后的接地来源）和按 run -的成本明细。 - ---- +当前 Arbor 的优化目标仅支持标量分数。我们考虑支持更复合的目标形式——如 rubric、LLM judge 评分与多目标优化——并相应解决这类目标下的可比较性与防作弊问题。 -## 方向四 —— 采用、开发者体验与社区 +### 5. Agent 科研能力的提升 -前三个方向扩展 Arbor *能做什么*；这个方向降低"好奇的访客"到"第一次成功运行" -之间的门槛，并让项目的推进势头可见。目标是缩短到达"aha 时刻"的路径，而不是堆叠 -更多功能面。 +让 Agent 能根据实验进展自主获取更多实验信息，而不局限于预先给定的 metric，从而为后续 +决策提供更充分的依据。 -### 4.1 免安装的在线 Demo +## 方向二：外部资源优化 -`arbor replay --demo --html` 已经能导出一个自包含、零依赖的真实 run 页面。把它 -发布出去（例如随文档一起部署到 GitHub Pages），并从 README 顶部链接过去，让访客 -**无需安装任何东西**就能看假设树生长。一份录制好的 run，随仪表盘变化刷新即可。 +### 1. 扩展到更多样化的长程优化场景 -### 4.2 示例画廊 +现有任务格式假设静态优化（环境固定、无状态）。我们计划增加带环境状态的任务形态，以 +支持办公等真实场景下的 agent harness 优化。 -目前 `examples/algotune_knn` 是新手唯一能端到端跑通的任务。把它扩成一个小型画廊， -让不同受众都能对号入座，每个都配可复制的命令和一段 asciinema 录屏： +### 2. 自动化环境收集 -- 一个 Kaggle / MLE 风格的任务（与 `mle_kaggle` 插件配套）， -- 一个 prompt / harness 工程任务，以及 -- 一个小规模训练 / 微调任务。 +收集并扩充更全面的 benchmark 场景，覆盖各类 LLM 与 Agent 评测基准，使 Arbor 能支持更广 +的优化场景。 -门槛保持在"笔记本上几分钟、或用免费 key 即可跑通"，复用 `algotune_knn` 的零配置 -纪律。 +## 方向三：用户体验 -### 4.3 规律发布与更新日志 +### 1. 扩展更多实际示例与场景 -版本号已经通过 setuptools-scm 从 `v*` tag 自动派生，发布成本很低。以可预期的节奏 -发布版本并附上人类可读的发布说明，并维护一份 `CHANGELOG.md`（或以 GitHub Releases -为准），让项目进展从外部看也一目了然。 +目前免安装 demo 已经落地，但能端到端跑通的示例仅有 `algotune_knn` 一个。我们计划将其 +扩展为一个小型示例画廊，覆盖不同类型的任务与受众（如 Kaggle / MLE、prompt 与 harness +工程、小规模训练等），每个示例配以可复制的命令与简短录屏，并将上手门槛控制在笔记本或 +免费 API key 即可在数分钟内跑通。 -### 4.4 公开路线图与 issue 看板 +### 2. 优化单次自动实验的导出与展示能力 -把本文档以公开看板（GitHub Projects）的形式呈现，并给跟踪中的工作打上 -`good first issue` / `help wanted` 标签。有几条线索天然适合外部贡献——把 -[benchmark zoo](#sec-2-1) 扩到 3–5 个 pack（[2.1](#sec-2-1)）、增加领域插件 -（[2.2](#sec-2-2)）——把它们开放出来，就能把读者变成贡献者。 +目前已有 `REPORT.md`、HTML 导出、实时 dashboard 与只读 WebUI，仍缺少对单次实验的理解 +与横向对比。我们计划支持 run 之间的对比（对同一任务的多次 run 做 diff，或跨 model / +provider 比较 idea tree），在报告中补充引用（每个 idea 背后的文献来源）与成本明细，并为 +benchmark 集合提供可复现视图——每一条目均附带可直接复跑的命令。 ---