Skip to content

Files just over the 300-line threshold fall below the interception break-even #12

Description

@HackPoint

What

The 300-line interception threshold admits files that cannot repay what interception costs. Interception blocks a Read, so the model spends another round; break-even is 5,383 avoided tokens per read. Files just over the threshold avoid roughly 2,100-3,300 tokens, which is less than they cost. 17 of the 31 files in the repository's corpus fall short. The corpus is still net positive (+$2.64), so the threshold pays in aggregate, but "the optimizer saves 94%" is false for those 17 files. What is missing is a measured threshold rather than another guess at one.

Evidence

  • Break-even of 5,383 avoided tokens per read: Econ::s_min(), crates/lumen-core/src/econ.rs:139.
  • 17 of 31 files below break-even, named individually rather than averaged away: the_files_where_interception_does_not_pay_are_named, crates/lumen-mcp/tests/efficiency.rs:267.
  • Net +$2.64 across the corpus, with the extra round subtracted: the_net_value_prices_in_the_round_interception_costs, crates/lumen-mcp/tests/efficiency.rs:219.
  • Largest single win in the corpus: lumenator/src-tauri/src/setup.rs, 41,649 tokens avoided.
  • Threshold default: DEFAULT_LINE_THRESHOLD, crates/lumen-core/src/coverage.rs:44. The per-run override is read by the hook — .claude/hooks/lumen_read_intercept.sh:31, THRESHOLD="${LUMEN_LINE_THRESHOLD:-300}" — not by the Rust classifier, which uses the constant at coverage.rs:108.
  • The efficiency suite pins its own copy at crates/lumen-mcp/tests/efficiency.rs:31, which is what a sweep has to parameterize. The suite reads no environment variable, so LUMEN_LINE_THRESHOLD does not move it today.

Raising the threshold is the obvious response and it is not clearly right. The loss on a 310-line file is small and bounded; the gain on setup.rs is 41,649 tokens. Moving the threshold up to avoid small losses also forfeits the long tail of moderate wins, and nobody has measured where the crossover actually sits. Since the threshold is already configurable per run through the hook's LUMEN_LINE_THRESHOLD, the sweep needs no product change to perform — only the pinned constant in the test has to give way.

Acceptance criteria

  • The efficiency suite prints net value for candidate thresholds 300, 400, 500, 600 and 800, one line per candidate.
  • Each candidate line also reports how many corpus files fall below break-even at that threshold.
  • The 300 row reproduces the figures recorded here — 17 of 31 files below break-even, net +$2.64 — allowing for drift, since the corpus is walked from the working tree and moves as files are added or grow.
  • The sweep runs from the existing test binary with no source change to the MCP server.
  • Any published savings percentage states the corpus and threshold it was measured on, or is removed.
  • A new default threshold is adopted only if it improves net value on at least two independent corpora; otherwise the default stays 300 and the measured curve is recorded on this issue.

How to test

From the repository root:

cargo test --release -p lumen-mcp --test efficiency -- --nocapture

Today this prints a single net-value block and one named list of the files below break-even. Once the sweep lands, expect one net-value line per candidate threshold alongside that list, with the 300 candidate reproducing 17 of 31 files short and net +$2.64.

The environment variable does not reach this test:

LUMEN_LINE_THRESHOLD=500 cargo test --release -p lumen-mcp --test efficiency -- --nocapture

This runs and changes nothing — the suite reads no environment variable, so the constant at crates/lumen-mcp/tests/efficiency.rs:31 still selects the corpus. Whether the sweep is driven by that variable or by an in-test list of candidates is an implementation choice; either way the pinned constant has to go. The variable is honoured only by the hook (.claude/hooks/lumen_read_intercept.sh:31), which is where a candidate threshold can be tried against real reads without editing anything.

Metadata

Metadata

Assignees

No one assigned

    Labels

    measurementMetering or reporting accuracy

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions