test(mcp): benchmark every MCP tool against a committed baseline

[HackPoint]

Token-efficiency benchmarks for all four MCP tools, compared against a committed baseline so each
fix's effect is visible as a delta.

Not wired into CI — that is the follow-up once you have looked at the output.

Coverage

20 scenarios across smart_read, recall_file, compress_logs and lumen_ping, driven through
handle_tools_call (the dispatch the stdio server uses) so argument parsing and error paths are
exercised rather than bypassed.

Coverage is checked, not claimed: every_advertised_tool_is_benchmarked reads tools/list and
fails if any advertised tool has no scenario — so a fifth tool fails this suite until someone
measures it. It fails in reverse too, if a scenario names a tool that no longer exists.

Scenarios deliberately include the cases where the tools should not win: an outline larger than its
file, a tiny file, an unparseable language, a range covering everything, an incompressible log.
Measuring only the good cases would make the table meaningless.

Current output

scenario                                   full returned    saved  vs base
smart_read/outline_few_big_items           7495      278     7217    +0.0%
smart_read/outline_many_tiny_items          720      703       17    +0.0%
smart_read/outline_unsupported_language    6400     6451      -51    +0.0%
smart_read/full_mode                       7495     7506      -11    +0.0%
recall_file/exact_one_name                 7495      768     6727    +0.0%
recall_file/exact_three_names              7495     2250     5245    +0.0%
recall_file/substring_narrow               3334       68     3266    +0.0%
recall_file/substring_broad                3334       66     3268    +0.0%
recall_file/range_whole_file               7495     7522      -27    +0.0%
compress_logs/repetitive_path             18380     1180    17200    +0.0%
compress_logs/incompressible               9472     9472        0    +0.0%
...20 scenarios total

across scenarios with a file: 39280 returned vs 123797 full = 31.7%

The negative rows are the honest ones: many_tiny_items, unsupported_language, full_mode and
range_whole_file all cost slightly more than the file, which is the inflation guard returning the
file plus one explanatory line. incompressible is exactly break-even. Those numbers should stay
small and negative; if one grows, something regressed.

Why this is separate from efficiency.rs

efficiency.rs measures this repository. That is right for "what does the optimizer save on real
code" and wrong for showing progress — its numbers move when the corpus moves. Its break-even count
went 17 → 16 within an hour of being published, because a commit added a file. A benchmark whose
numbers drift for unrelated reasons cannot attribute a change to a fix.

This one measures frozen generated inputs, so a delta can only mean the code changed.

Two things that had to be fixed before the baseline meant anything

It was not reproducible. Every tool embeds the path it was given in its header, so the path's own
token cost lands in returned_tokens. With a random TempDir, three runs with no code change gave
351, 351 and 357 tokens for one scenario — noise larger than most real improvements, and enough to
fail CI at random. An absolute path is no better: /Users/... and /home/runner/... tokenize
differently, so a baseline blessed locally could never match in CI. Fixtures now live at a short
relative path; five consecutive runs are bit-identical.

Measurements alone are not comparable. An edited fixture would pass as an optimizer improvement —
a smaller number for a smaller input. The baseline records an FNV-1a hash of every input beside the
measurements, and a changed generator fails with "the deltas are not comparable" instead of reporting
a win. FNV rather than DefaultHasher, whose cross-release stability is unspecified.

Verified in both directions

Improvement — LUMEN_LINE_NUMBERS=0 drops the per-line gutter:

recall_file/exact_one_name      768 ->  697   -12.2%
recall_file/exact_three_names  2250 -> 1994   -12.4%
recall_file/range_narrow        647 ->  598   -11.1%

Regression — re-widening the gutter to {lineno:>5}: :

the optimizer got worse:
  recall_file/exact_one_name: returned 768 -> 870 (+102)
  recall_file/exact_three_names: returned 2250 -> 2546 (+296)
  recall_file/range_narrow: returned 647 -> 721 (+74)

How to use it

cargo test --release -p lumen-mcp --test optimizer_bench -- --nocapture   # see the table
LUMEN_BENCH_BLESS=1 cargo test --release -p lumen-mcp --test optimizer_bench   # after a fix

Commit the regenerated baseline with the fix that moved it. The git history of one JSON file then
becomes the record of the optimizer's progress, fix by fix — which is the thing you asked for.

Judgement calls worth your attention

• Wall-clock is reported, never asserted. A shared CI runner cannot support a latency claim, and a
  flaky perf gate teaches people to re-run until it passes. The µs column is there to spot something
  pathological, not to gate.
• Fixtures are generated, not committed files. The generator is visible in the diff where a blob
  of fixture text would not be, and nothing can reformat it. The tradeoff is that they are synthetic
  rather than real-world shapes — efficiency.rs covers real code, so the two are complementary.
• Any increase fails, with no tolerance band. Token counts are deterministic, so a band would only
  hide small regressions. If an increase is intended, re-bless in the same commit.
• A missing baseline entry is a hard failure, not a warning: a scenario with no baseline has no
  history, which is the one thing this file exists to provide.

Once you are happy with the numbers, the CI step is small — add it to the existing test job and
require it, the same way the other checks are required.

---

Correction to the table above. It originally quoted figures taken before the relative-path fix
(outline_few_big_items at 351 rather than 278). Those were measured against a long random tempdir
path whose token cost was inside the numbers — exactly the noise this PR removes. Replaced with the
committed baseline.

Windows. The first push failed Test (windows-latest): nine scenarios off by +1 to +3 tokens.
PathBuf::join produces target/bench-fixtures\big.rs there, and the separator is in the header and
therefore in the measurement. Same defect as the tempdir, one level down. Fixed by building the path
as a forward-slash string, which Windows accepts. macOS numbers were unchanged, so the baseline still
stands. All 11 checks green.