[codex] #87 补充 agent recall 基线报告

[catoncat]

Mainline Intent

Intent: int_59a088ab
Status: proposed
Title: 补充 agent recall 基线报告

What changed

扩展 eval/perf-bench.ts，使 agent-recall 基线报告覆盖 active Sherlog DB、find 输出尺寸、read-range/read-page 上下文尺寸、SQLite table-size breakdown，以及可选 dogfood scorecard 摘要。

Why

#87 需要在 recall packet、projection、dogfood gate 等后续改动前，先拿到一份可复现、机器可读且人类可扫的现状基线，避免凭感觉判断搜索/上下文/数据库成本。

Decisions

• 默认测量哪个数据库: 默认使用 DEFAULT_DB_PATH，并保留 --db 覆盖 (issue 明确要求 active Sherlog DB 的大小和 table-size breakdown；自定义 --db 仍支持 fixture 或 scratch benchmark。)
  • Rejected: 继续默认写入临时 scratch DB
• 如何记录返回上下文大小: find/read 命令记录 outputBytes/outputChars，read-range/read-page 额外记录 messages 的 contentText 字符总量 (output size 反映 agent 实际收到的 JSON 负载，messageContentChars 反映 transcript 上下文实体大小。)
• session-level top hit 如何测 read-range: matchSeq 缺失时用 read-range --query 回退定位，同步测量 read-page (当前 recall 会返回 session-level hits；用 --query 可覆盖这类真实 top hit，而不是把 read-range 指标置空。)
• dogfood 汇总集成方式: 在 perf bench 增加可选 --dogfood，并解析 run-dogfood-eval.ts 的 scoreboard/outDir/scorecard (Baseline agent-recall metrics #87 要求同一 baseline report 在提供 dogfood 文件时记录当前 scorecard summary；runner 即使存在 failing hard cases，也应被记录而不是吞掉。)
  • Rejected: 本轮拆出新的 dogfood adapter 模块或新命令
• SQLite 空间拆解: 用 better-sqlite3 只读打开 benchmark DB，通过 dbstat 聚合 table/index bytes，并记录 page_size/page_count/freelist_count (dbstat 是 SQLite 原生结构化来源，能避免 ad-hoc 文件推断。)

Subsystems: eval, perf, dogfood, sqlite-storage, agent-recall

---

Summary by cubic

Adds a reproducible baseline report for agent recall, covering DB storage and response sizes, with optional dogfood summary. Addresses #87 by making current recall performance and costs easy to compare before further changes.

• New Features
  • Measures active Sherlog DB: total size, page stats, and per-table bytes via better-sqlite3/dbstat.
  • Tracks response sizes: outputBytes/outputChars for find/read; messageContentChars for read-range/read-page.
  • Adds read-range fallback when matchSeq is missing (uses --query) and always probes read-page.
  • Optional --dogfood runs run-dogfood-eval.ts and embeds scoreboard/scorecard in JSON and Markdown.
  • Defaults to DEFAULT_DB_PATH (still supports --db); sync runs with --best-effort.
  • Markdown adds per-query size columns and top table sizes; JSON adds storage/dogfood fields.

^{Written for commit 5dd6eac. Summary will update on new commits.}

[cubic-dev-ai]

1 issue found across 1 file

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="eval/perf-bench.ts">

<violation number="1" location="eval/perf-bench.ts:338">
P2: Baseline metrics can be produced from partially failed syncs without any warning, which makes recall/latency numbers look valid even when indexing failed for part of the corpus. This comes from forcing `sync --best-effort` while not validating the returned failure/error fields; using normal sync behavior here (or explicitly failing when errors > 0) keeps the baseline trustworthy.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Baseline metrics can be produced from partially failed syncs without any warning, which makes recall/latency numbers look valid even when indexing failed for part of the corpus. This comes from forcing sync --best-effort while not validating the returned failure/error fields; using normal sync behavior here (or explicitly failing when errors > 0) keeps the baseline trustworthy.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At eval/perf-bench.ts, line 338:

<comment>Baseline metrics can be produced from partially failed syncs without any warning, which makes recall/latency numbers look valid even when indexing failed for part of the corpus. This comes from forcing `sync --best-effort` while not validating the returned failure/error fields; using normal sync behavior here (or explicitly failing when errors > 0) keeps the baseline trustworthy.</comment>

<file context>
@@ -287,7 +335,7 @@ if (!args.jsonOnly) {
 
 // 1. sync
-const syncRun = await runOrThrow(cliCommand("sync", "--source", args.source, "--db", args.db, "--root", args.root, "--json"));
+const syncRun = await runOrThrow(cliCommand("sync", "--source", args.source, "--db", args.db, "--root", args.root, "--best-effort", "--json"));
 const syncMs = syncRun.ms;
 let sessionCount = 0;
</file context>

Suggested change

	const syncRun = await runOrThrow(cliCommand("sync", "--source", args.source, "--db", args.db, "--root", args.root, "--best-effort", "--json"));
	const syncRun = await runOrThrow(cliCommand("sync", "--source", args.source, "--db", args.db, "--root", args.root, "--json"));