fix(observability): count the prompt-cache tiers as input tokens for the CLI providers - #10251
Merged
Merged
Conversation
…the CLI providers Every claude-code call recorded exactly ~2 input tokens. Measured on edge-nl-01 over 48h: 2048 input tokens across 1000 calls -- an average of 2.0 -- against 1051 output tokens per call and $225.06 of real spend. An average of 2.0 over a thousand calls is not a measurement, it is a constant. Anthropic, and therefore Claude Code, splits one prompt across three counters: input_tokens carries only the portion neither read from nor written to the prompt cache, with the rest in cache_read_input_tokens and cache_creation_input_tokens. INPUT_TOKEN_KEYS read only the first, so with caching active -- which it is on every review the CLI runs -- the figure degenerated to the handful of genuinely-new tokens. These are real input tokens: the model processed them and they are billed, cache reads at a reduced rate. Each tier gets its OWN alias group because the three are additive components of one prompt, not names for one value; folding them into INPUT_TOKEN_KEYS would take the maximum of the three and silently under-report again, just less severely. Absence is still absence -- an envelope reporting no input counter at all yields undefined rather than a fabricated 0 (#10207), and a tier present but zero contributes a real zero. Providers that emit no cache keys -- codex and the OpenAI-compatible bindings -- are byte-identical to before, which is what makes this safe at the shared extraction point. loopover_ai_input_tokens_total is corrected by the same change, since it reads the same usage.inputTokens. coerceByokUsage (src/services/ai-review.ts) is deliberately NOT changed. It already documented these two keys and declined them, because callAiProvider never sends cache_control so the provider never populates them there, and it feeds BYOK_MODEL_PRICING_USD_PER_MTOK where a cache read bills at a different rate than fresh input. Its comment now records that the divergence from the CLI path is intentional rather than an oversight. Closes #10235
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Contributor
|
Tip ✅ LoopOver review result - approve/merge recommendedReview updated: 2026-07-31 14:45:07 UTC
Review summary Nits — 4 non-blocking
Decision drivers
Context & advisory signals — never blocks the verdict
Linked issue satisfactionPartially addressed Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #10251 +/- ##
==========================================
- Coverage 92.25% 91.39% -0.87%
==========================================
Files 938 938
Lines 114702 114707 +5
Branches 27698 27698
==========================================
- Hits 105821 104832 -989
- Misses 7575 8764 +1189
+ Partials 1306 1111 -195
Flags with carried forward coverage won't be shown. Click here to find out more.
|
![loopover-orb[bot]](https://avatars.githubusercontent.com/in/4139483?s=60&v=4){kind=small}
JSONbored
added a commit
that referenced
this pull request
Jul 31, 2026
…e's own parsers #10251 fixed this on the ORB side. Both of the engine's deliberately-parallel copies were left reading only `input_tokens`, so the miner still records a near-constant handful for every cached claude attempt. Anthropic, and therefore Claude Code and the Agent SDK, splits one prompt across three counters: input_tokens carries only the portion neither read from nor written to the prompt cache, with the rest in cache_read_input_tokens and cache_creation_input_tokens. With caching active -- which it is for every attempt the CLI runs -- essentially the whole prompt lands in the latter two. Sum the three, matching the ORB's totalInputTokens exactly, so the two parsers cannot report different numbers for the same envelope. Each tier keeps its own alias group because the three are additive components of one prompt, not names for one value; folding them into INPUT_TOKEN_KEYS would take the maximum and under-report again, just less severely. Absence stays absence: an envelope with no input counter at all yields undefined rather than a fabricated 0, and a tier present but zero contributes a real zero. Providers emitting no cache keys -- codex and the OpenAI-compatible bindings -- are byte-identical to before, which is what makes the shared extraction point safe to change. Closes #10246
loopover-orb Bot
pushed a commit
that referenced
this pull request
Jul 31, 2026
…e's own parsers (#10252) * fix(miner): count the prompt-cache tiers as input tokens in the engine's own parsers #10251 fixed this on the ORB side. Both of the engine's deliberately-parallel copies were left reading only `input_tokens`, so the miner still records a near-constant handful for every cached claude attempt. Anthropic, and therefore Claude Code and the Agent SDK, splits one prompt across three counters: input_tokens carries only the portion neither read from nor written to the prompt cache, with the rest in cache_read_input_tokens and cache_creation_input_tokens. With caching active -- which it is for every attempt the CLI runs -- essentially the whole prompt lands in the latter two. Sum the three, matching the ORB's totalInputTokens exactly, so the two parsers cannot report different numbers for the same envelope. Each tier keeps its own alias group because the three are additive components of one prompt, not names for one value; folding them into INPUT_TOKEN_KEYS would take the maximum and under-report again, just less severely. Absence stays absence: an envelope with no input counter at all yields undefined rather than a fabricated 0, and a tier present but zero contributes a real zero. Providers emitting no cache keys -- codex and the OpenAI-compatible bindings -- are byte-identical to before, which is what makes the shared extraction point safe to change. Closes #10246 * test(engine): cover the cli driver's input-tier sum in the engine's own suite codecov/patch failed at 83.78% because totalInputTokens' body and its mergeCliUsage call site had zero hits in the engine flag's c8 report: the driver's behavior tests live entirely in the root vitest copy, which the engine flag never sees. Port the tier-sum scenarios into packages/loopover-engine/test/, the suite that actually carries this file's coverage. Verified locally: every changed line in both drivers now has hits in the engine lcov. --------- Co-authored-by: JSONbored <aetherealdev@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every
claude-codecall recorded exactly ~2 input tokens. Measured on edge-nl-01 over 48h:claude-codeAn average of 2.0 across a thousand calls is not a measurement, it is a constant. The output side is healthy and the dollar cost is real, so the event was arriving and being parsed — the input figure specifically carried nothing.
Cause
Anthropic — and therefore Claude Code — splits one prompt across three counters.
input_tokenscarries only the portion that was neither read from nor written to the prompt cache; the rest lands incache_read_input_tokensandcache_creation_input_tokens.INPUT_TOKEN_KEYS(src/selfhost/ai.ts) read only the first group:With caching active — which it is on every review the CLI runs, since the system prompt and tool definitions are stable — that degenerates to the handful of genuinely-new tokens per call. Nothing was malformed and nothing was being dropped defensively: the extractor read a real field that does not mean what its name suggests under caching.
Fix, and the trap inside it
The two cache tiers are counted as input tokens, because they are: the model processed them, and they are billed (cache reads at a reduced rate).
Each tier gets its own alias group, and the groups are SUMMED — not appended to
INPUT_TOKEN_KEYS.maxNumbertakes the maximum across a key list, which is correct for aliases (input_tokensandprompt_tokensare two names for one number) but wrong here, because the three are additive components of a single prompt. Folding them into one list would take the max of the three and silently under-report again, just less severely — the same failure wearing a smaller number.Absence stays absence: an envelope reporting no input counter at all yields
undefined, never a fabricated0(#10207's rule), while a tier that is present but zero contributes a real zero.Blast radius
codexand the OpenAI-compatible bindings — are byte-identical to before. That is what makes this safe to apply at the shared extraction point rather than per-provider, and it is pinned by a test.loopover_ai_input_tokens_totalis corrected by the same change, since it reads the sameusage.inputTokens. The Prometheus/Grafana view was wrong in the same direction.What is deliberately NOT changed
coerceByokUsage(src/services/ai-review.ts) — the BYOK API path — already considered these exact two keys and declined them:That reasoning still holds, and reading them there would be worse than dead code: it feeds
BYOK_MODEL_PRICING_USD_PER_MTOK, where a cache read bills at a different rate than fresh input, so ifcache_controlis ever introduced it would mis-price. The two paths naming the same keys but disagreeing on whether to read them is correct, not drift — its comment now says so explicitly, so the next reader sees an intentional divergence rather than an oversight.Closes #10235
Scope
type(scope): short summaryConventional Commit format, for examplefix(api): restore profile access checks.CONTRIBUTING.mdand does not reintroduce GitHub Pages, VitePress,site/, orCNAME.Closes #123) — a linked open issue is required for every contributor PR.Validation
git diff --checknpm run actionlintnpm run typechecknpm run test:coveragelocally;codecov/patchrequires ≥99% coverage of the lines AND branches you changednpm run test:workersnpm run build:mcpnpm run test:mcp-packnpm run ui:openapi:checknpm run ui:lintnpm run ui:typechecknpm run ui:buildnpm audit --audit-level=moderateDetail:
main: 26,623 passed, 0 failed.totalInputTokensto the oldmaxNumber(entry, INPUT_TOKEN_KEYS)fails both new tests.input_tokens: 2alongsidecache_read_input_tokens: 41820/cache_creation_input_tokens: 1140), reproducing the exact ~2 observed in production.selfhost:env-reference:check,docs:drift-check,coverage-boltons:check,dead-exports:check,manifest:drift-check.npm auditreports only pre-existing advisories transitive underrelease-please; this PR changes no dependencies.Safety
UI Evidencesection below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails.Token counts only — no prompt, diff, transcript or completion content is read or emitted by this path, and the metadata-only policy on
$ai_generationis unchanged.UI Evidence
Not applicable — no visible UI, frontend, docs, or extension change.
Notes
Historical
ai_usage_eventsrows keep their wrong figure; this corrects the reading going forward rather than backfilling. Worth knowing when comparing across the cutover:claude-codeinput tokens will jump by roughly three orders of magnitude, and that jump is the fix landing, not a change in usage. Cost is unaffected —$ai_total_cost_usdwas always read from the provider's owntotal_cost_usdand was never derived from these counters.