observability: cache effectiveness is unanswerable because three surfaces use three different words for a hit

[JSONbored]

The premise was wrong, and that is itself the finding

The standing note on this work said "there is no cache hit/miss telemetry, so 'is our caching working?' is unanswerable." The first half is false — the second half is true, for a different and more interesting reason.

Telemetry exists and is thorough. src/selfhost/metrics.ts declares 15 *_cache_hit_total / *_cache_miss_total counters, and all 15 are actually incremented (verified: no declared-but-unwired metric). On top of that, src/selfhost/orb-collector.ts:81 already computes a combined hit ratio, and src/services/public-reuse-rate-trend.ts computes a weekly reuse rate.

So why is the question still unanswerable? Because each cache reports its avoidance under a different event name, and no single view knows all of them.

The trap, measured

Asking the obvious question — %cache_hit% vs %cache_miss% over audit_events — gives this for the AI review cache over 24h:

github_app.ai_review_cache_hit	1
github_app.ai_review_cache_miss	228
apparent hit rate	0.44%

That reads like a completely dead cache. It is not. The durable cache is bypassed by design for dynamic-context repos (grounding/RAG/enrichment/reputation — see the features comment in src/review/ai-review-cache-input.ts), and reuse is instead served by the #regate-churn cooldown, which emits different events:

github_app.ai_review_one_shot_reuse	561
github_app.ai_review_frozen_reuse	249
true rate (811 hits / 228 misses)	78.1%

The naive query understates the real answer by 177x. Only orb-collector.ts knows to add AI_REVIEW_REUSE_EVENT_TYPES, and even it does not know about the *_one_shot_skip family below.

Full picture (Orb, 24h to 2026-07-31T12:35Z)

cache	avoidance event family	hits	misses	rate
miner_detection	cache_hit	1834	304	85.8%
ai_review	*_reuse	811	228	78.1%
ai_slop	one_shot_skip	850	302	73.8%
linked_issue_satisfaction	one_shot_skip	690	248	73.6%
grounding	cache_hit only	142	753	15.9%

Three different vocabularies for one concept: cache_hit, *_reuse, *_one_shot_skip.

Two pieces of work

1. One vocabulary, one view. Every cache-like surface should report avoidance under a shape a single query can aggregate — either a shared event suffix, or a cache_outcome metadata field on a common event type, with the existing per-surface events kept as-is for back-compat. Then "is our caching working?" is one query instead of five plus tribal knowledge. Right now the answer depends on knowing that three of the five surfaces do not use the word "cache" for a hit.

2. grounding is a genuine outlier and worth a look on its own. It is the highest-volume surface here (895 calls/24h, more than any other) and the only one with no reuse or skip path — 15.9%, against 74-86% everywhere else. Either its key includes something per-evaluation (the exact failure #regate-churn already fixed once for baseSha in the AI review fingerprint, and that #10184 found again in record_digest), or it legitimately cannot be cached and should say so explicitly rather than looking broken.

Why this is a prerequisite

Any further caching work needs a trustworthy baseline. Today the most natural query says the AI review cache is 0.44% effective when it is 78%, which would send someone optimising a cache that already works while the actual outlier (grounding) sits unexamined.

Verification notes

All numbers queried directly against the Orb Postgres primary, not derived from code. audit_events retains from 2026-06-11, so the 24h window is well inside retention. The declared-vs-wired metric check was done by diffing the metric names in metrics.ts against every incr(/gauge( call site.

[linear-code]

JSO-7128

[JSONbored]

Half (b) resolved by measurement: grounding is not a broken cache, and it is not fixable by re-keying

All figures queried directly against the Orb primary on edge-nl-01.

It is worse than 15.9% — it is now ~0%

Hourly, github_app.grounding_cache_hit vs _miss:

window	hits	misses	rate
07-30 19:00–23:00	120	160	27–61%
07-31 01:00–13:00	2	~500	0.0% in 11 of 13 hours

So the 15.9% in the issue body is an average across a window whose first half still had reuse. The current rate is effectively zero.

Two candidate causes, both checked, one falsified

#10204 is NOT the cause, despite fitting the story. It merged at 2026-07-31T12:34:36Z; the collapse begins at 01:00Z, ~11.5 hours earlier. Ruled out on timing alone.

The real mechanism

getCachedGroundingFileContent (src/db/repositories.ts:6020) keys on (repo_full_name, path, head_sha) with a 24h TTL. Its own comment states the intended hit: "reusing a commit's content across review lanes / re-runs within the same PR, which happens within minutes."

That is the only hit it can ever get. A new push means a new head SHA, so every file is a guaranteed miss. Which means:

The grounding cache's hit rate is a measure of re-evaluation churn, not of caching effectiveness.

Every hit requires the same file to be grounded twice at the same commit. As same-SHA re-evaluation was deliberately driven down — the whole point of #regate-churn and #10204 — this cache's hit rate necessarily goes to zero. It is falling because the system got more efficient. Reading it next to miner_detection at 85.8% and concluding "grounding is broken" inverts the truth.

Re-keying on blob content would not fix it either — measured, not assumed

The obvious fix is to key on content rather than commit, so an unchanged file shared across commits hits. The gain is negligible:

rows keyed (repo, path, head_sha)	1146
distinct (repo, path, md5(content))	1047
distinct (repo, path)	672

Content-keying collapses 1146 → 1047: 8.6% reuse, not the large win the framing suggests.

The reason is structural and obvious in hindsight: grounding fetches the files the PR changed. A changed file has different content at every head SHA by construction. 672 distinct paths carrying 1047 distinct contents means each path really does appear with ~1.6 different bodies across the window — the files are changing, because that is what makes them interesting to a reviewer.

Recommendation

This is the issue's own second fork — "it legitimately cannot be cached and should say so explicitly rather than looking broken" — and the measurement now supports it over the first fork.

• Do not re-key, and do not optimise this cache. It is working correctly; there is simply almost nothing to reuse.
• Do exclude it from the unified caching view half (a) builds, or report it under its own label. Listing a per-commit blob cache next to fingerprint-keyed caches invites exactly the misreading this issue was opened about — and a future reader watching this number "improve" would be watching re-evaluation churn come back.
• The 24h TTL and the safety-net reasoning behind it (test(ci): cover the extension build scripts' hand-written zip writer #7481-class, mutable-ref misuse) are untouched by any of this and should stay.

Half (a) — the one vocabulary, one view work — is unaffected and still stands on its own.