fix(rpc): sum both usage stores behind one composer all three surfaces call - #9296
Merged
Conversation
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
metagraphed-registry-sync-api | 214b37b | Aug 03 2026, 01:49 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
metagraphed-data-api | 214b37b | Aug 03 2026, 01:48 PM |
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
❌ 5 Tests Failed:
View the full list of 5 ❄️ flaky test(s)
To view more test analytics, go to the Test Analytics Dashboard |
…s call /api/v1/rpc/usage had two defects that are the same defect seen from two sides, so they are fixed together rather than as two patches that would each have to remember the other. MAGNITUDE (#9293). The Analytics Engine hot tier DISPLACED the lakehouse cold tier instead of joining it. Measured 13:03 UTC: `window=7d` reported total_requests 3,990 across three buckets spanning two hours of today, where the same window served 578,506 across 151 buckets the day before -- a 99.3% under-report carrying a confident `7d` label. It does not resolve itself: the lakehouse is frozen permanently at 2026-08-02 22:00, so AE only reaches a rolling 7d from 08-10 and 30d from 09-02, and the older days are ABSENT from the number rather than diluted. Counts are additive across disjoint ranges, so they are summed: total, ok, error, failover and cache_hits, plus the per-endpoint, per-network and per-bucket series. The lakehouse read is bounded strictly below the oldest event AE holds, so the two stores can never describe the same event twice, and it is only issued while AE does not already span the requested window -- once it does, the second scan stops for good rather than becoming permanent cost. Percentiles are NOT summed, because they are not additive. quantileExactWeighted cannot be merged with a store that has no percentile function at all. They stay AE-only, and the payload now says which sub-range they describe. The payload also stops claiming a range it did not measure. A new `coverage` block reports the span each store actually contributed, per store -- two non-adjacent segments make the real hole between the frozen lakehouse and the start of AE capture visible rather than papering over it with a single start/end pair. SURFACES (#9269). REST, MCP and GraphQL each ran their own cascade. #9207 wired the cold tier into the REST handler only, so with the Postgres box destroyed the other two fell through `tryPostgresTier -> loadRpcUsage` to the schema-stable ZEROED card: REST served 118,309 requests on its top endpoint while `rpc_usage(window:"7d")` served `total_requests: 0, endpoints: []` at the same instant. An MCP client cannot tell that apart from an idle proxy. The cascade is now ONE composer, `answerRpcUsage` in src/rpc-usage-answer.ts -- the same shape as src/account-summary-card.ts's answerAccountSummary, which exists for the same reason. All three surfaces call it and compose nothing themselves, so src/rpc-usage-loader.ts's zeroed floor is reachable only where it is genuinely correct: when every store declined. The tests assert ACROSS the surfaces, not per surface -- structurally (no surface imports a tier reader) and behaviourally (one set of store responses, three identical cards). Per-surface tests are what let this class of bug land twice. The MCP tool and GraphQL SDL descriptions no longer claim D1 telemetry, which has not existed since #9228. Closes #9293 Closes #9269
JSONbored
force-pushed
the
fix/rpc-usage-one-composer
branch
from
August 3, 2026 13:48
5cb40c4 to
214b37b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/api/v1/rpc/usagehad two defects that are one defect seen from two sides, which iswhy they are fixed in one PR rather than two patches that would each have to remember
the other.
displaced the lakehouse cold tier instead of joining it. Measured 2026-08-03 13:03
UTC:
window=7dreturnedtotal_requests: 3990over three buckets spanning twohours of today, where the day before the same window served 578,506 over 151 buckets.
It never resolves itself — the lakehouse is frozen permanently at 2026-08-02 22:00,
so AE reaches a rolling 7d only from 08-10 and 30d from 09-02, and the older traffic
is absent from the number rather than diluted.
GraphQL each ran their own cascade. feat(api): serve /api/v1/rpc/usage from the lakehouse #9207 wired the cold tier into the REST handler
only, so with the Postgres box destroyed the other two fell through
tryPostgresTier → loadRpcUsageinto the schema-stable zeroed card. Verifiedlive: REST 118,309 requests on its top endpoint,
rpc_usage(window:"7d")total_requests: 0, endpoints: [], same moment.The shared cause is that three surfaces each assembled the card, so a fix could land on
one leg and leave the others a version behind — exactly the pattern #9285 documented
and fixed for
/accounts/{ss58}.What Changed
One composer.
answerRpcUsage(src/rpc-usage-answer.ts) owns the whole tierdecision — deliberately the same shape as
src/account-summary-card.ts'sanswerAccountSummary, so this reads as one pattern rather than two solutions to oneproblem. REST (
workers/request-handlers/rpc-proxy.ts), theget_rpc_usageMCP tooland the
rpc_usageGraphQL resolver all call it and compose nothing themselves.src/rpc-usage-loader.ts's zeroed floor is now reachable only where it is genuinelycorrect: when every store declined.
Merge strategy: full sum, not fall-through. Counts are additive over disjoint
ranges, so
total_requests,ok_requests,error_requests,failover_requests,cache_hitsand the per-endpoint / per-network / per-bucket series are summed acrossboth stores. Two properties keep that sound:
(
loadRpcUsageColdTier's newuntil), so the two stores can never describe the sameevent twice. That is why the reads are sequential rather than parallel — the cold
query cannot be bounded until the hot answer is in hand.
(one bucket of tolerance, since AE's own
timestamp > now() - N DAYpredicate alwaysstarts a moment after the cutoff). Once AE's retention covers the window — 7d from
2026-08-10, 30d from 09-02 — the second scan stops being issued and never resumes, so
this is a transitional cost, not a permanent one.
Percentiles are not merged, and say what they cover.
quantileExactWeightedcannotbe combined with a store that has no percentile function at all, so p50/p95 stay
AE-only. Reporting them as the whole window's percentiles would be a claim about a
sub-range, so the sub-range is published instead of implied.
The payload no longer claims a range it did not measure. New top-level
coverage:Segments are per store rather than one start/end pair on purpose: the two stores are
disjoint and non-adjacent, so the honest picture has a hole in the middle that a
single pair would paper over.
latency_percentilesisnullwhen nothing measuredthem —
nullkeeps meaning "not measured", exactly as it already does everywhere inthis payload.
Descriptions corrected. The MCP tool description and the GraphQL SDL no longer
claim
rpc_proxy_eventsD1 telemetry (gone since #9228); both now name the twotiers and the deliberate percentile difference.
docs/backend-artifact-contracts.mdlikewise.
Requirement 4 of #9269 (audit the other
tryPostgresTiercall sites): the flag forthis route is
"retired"inwrangler.jsoncandworkers/data-api.tsno longer has anrpc/usage read route at all, so the Postgres leg is dead in both directions. It is kept
in the composer, in its historical position ahead of the lakehouse, purely as the
documented kill switch — no behaviour change.
src/account-summary-card.ts(#9285)already closed the other instance of "cold tier wired into REST but not the siblings";
no further instance of the pattern was found for this route family.
Tests
The tests assert across the surfaces, not per surface — per-surface tests are what
let this class of bug land twice:
tests/rpc-usage-surface-parity.test.ts— (1) structural: no surface imports a tierreader (
loadRpcUsageHotTier/loadRpcUsageColdTier/loadRpcUsage), each importsthe composer; (2) behavioural: one set of Analytics Engine + R2 SQL responses driven
through the real readers, and REST / MCP / GraphQL come back byte-identical on
summary,endpointsandcoverage.tests/rpc-usage-answer.test.ts— the sum, the weighted latency mean, the refusal tomerge percentiles, the coverage segments (including the gap), endpoint/network/bucket
folding, the cascade order, and the floor being reachable only when every store
declined.
min()selections, theuntilceiling (includingthat an unusable ceiling is dropped rather than interpolated — R2 SQL has no bound
parameters), and each tier's own
coverage.All generated SQL still passes
unsupportedAeFunctions(minis documented; nonullif/ifNull/coalesce/NULL literal anywhere), and no ungroupedcount(DISTINCT)is introduced.
Patch coverage
100% of changed lines in
src/**+workers/**, branch-counted, measured byintersecting the diff's changed lines with v8's uncovered statement + branch set (not a
whole-file percentage):
No
v8 ignorewas added — two defensive branches that no fixture could reach on bothsides were removed instead (the merge's inputs are
formatRpcUsageoutput, sore-guarding fields that are unconditionally present was untested branching wearing the
costume of safety).
Note on the base
tests/account-summary-card.test.tsandtests/cross-view-agreement.test.tsfail with5 assertions on
origin/mainat b852dbd — verified by stashing this branch andrunning them against a clean tree, and confirmed in main's own Validate run
(30817296139,
testjob:2 failed | 636 passed). Those are #9285's own tests and areuntouched by this PR; main's
uie2e job is also red there. Nothing in this diff canturn either green.
Registry Safety
Closes #<n>) — required.validator-local state.
(
public/metagraph/r2-manifest.jsonauto-reverted by the build;public/metagraph/schemas/index.jsonuntouched).coverageisadditive on
RpcUsageArtifactandget_rpc_usage's output schema, plus the newRpcUsageCoverage/RpcUsageCoverageSegment/RpcUsageCoverageRangeGraphQLtypes;
openapi.json,packages/contract/index.d.ts,public/metagraph/types.d.tsand
generated/graphql/types.tsregenerated and committed. The api-reference docsgenerator was re-run from
apps/uiand produced no drift.Validation
npm run lint·npm run format:check·npm run typechecknpm run build(artifacts regenerated + committed)npm run validatenpm run validate:schemasnpm run validate:apinpm run validate:openapinpm run validate:typesnpm run validate:contract-driftnpm run validate:schema-enumsnpm run validate:graphql-types-driftnpm run validate:mcpnpm run validate:ainpm run validate:docsnpm run validate:module-state-resetsnpm run validate:no-hand-written-mjsnpm run scan:public-safetynpm run validate:private-boundarynpm test(647/649 files pass; the 2 failing files are the pre-existing mainbreakage noted above)
git diff --checkTemplate Used
backend-code.mdCloses #9293
Closes #9269