feat(validators): give the nominator-counts lane a D1 sink so it can advance again - #9302
Merged
Merged
Conversation
…advance again
`nominator_count` on /api/v1/validators and /api/v1/validators/{hotkey} is
served from the lakehouse mirror of `chain.validator_nominator_counts` (#9276),
and that mirror is frozen. Measured live 2026-08-03: 112,550 rows, newest
`captured_at` 2026-08-02T01:38Z -- the retired box's last scan -- covering only
564 of the 1,031 validators the leaderboard serves. Coverage can only fall from
here as new validators register.
This is a SINK problem, not a producer problem. metagraphed-infra's poller
Container already carries the full SubtensorModule::Alpha scan this table is
derived from (src/bin/poller/jobs/validator_nominators.rs, 24h tick); it is one
of the five lanes Dockerfile.poller leaves disabled "until they have a
Cloudflare-native sink", because it writes to a Postgres that no longer exists.
The scan needs no rehosting. Its write target does.
The lane, on the pattern #9273/#9292 used for `nominator_positions` -- the other
output of that same scan:
- migrations/d1/0012 adds `validator_nominator_counts` on D1, upserted on
(hotkey) with the usual `captured_at <= excluded.captured_at` staleness guard.
The writer's column list and the table are asserted against each other in both
directions, the same anti-drift guarantee 0007 has. No prune, and for a
sharper reason than account_identity's: the producer's pass over Alpha IS
exhaustive, but it chunks across requests, so "absent from this batch" and
"absent from the scan" are different statements and only the latter would
license a delete.
- src/validator-nominator-counts-d1-write.ts reuses neurons-d1-write's statement
building, so one place owns the Workers-binding budget of 100 bound parameters
per statement -- asserted as the platform limit in the tests, not as a constant
we picked. At 3 columns this chunks to 30 rows a statement, so a full scan is
~3,752 statements.
- workers/data-api.ts's handleValidatorNominatorCountsSync stops answering
`503 hyperdrive binding unavailable` and writes that batch to D1.
`nominator_count` is required to be a non-negative integer rather than
coerced: the read side already discards anything else, and a value the route
accepted but every reader silently drops is worse than a 400 the producer can
see.
- The read is wired into buildGlobalValidators/buildValidatorDetail, which were
handed a hardcoded empty map / null count. It joins against `neurons` inside
SQLite rather than inlining a key list -- the leaderboard covers ~1,031
hotkeys against a 100-parameter cap, so a correlated subquery costs zero bound
parameters and one query where an IN list would cost a dozen round trips.
#9276's serving-Worker lakehouse overlay is deliberately left in place. It only
collects hotkeys whose count is still null and returns early when there are
none, so it covers what D1 has not received yet and stops firing on its own once
a full scan lands -- the cutover is a property of the data, not of a deploy.
Failure posture is the family's throughout: any failed read degrades to null,
which is exactly what this tier served before the table existed. A broken read
is a lost enrichment, never a wrong number.
Closes #9301
Part of #9146
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
metagraphed-data-api | 2b53fe6 | Aug 03 2026, 08:26 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
metagraphed-registry-sync-api | 2b53fe6 | Aug 03 2026, 08:26 PM |
This was referenced Aug 3, 2026
Merged
JSONbored
added a commit
that referenced
this pull request
Aug 3, 2026
…sholds to the producer (#9311) Two gaps in the alarm coverage over the two outputs of the poller's one Alpha scan. THE COUNTS LANE HAD NO WATCHDOG. #9302 gave it a D1 sink and metagraphed-infra #243 re-enabled its producer, but nothing watched it -- which is the exact shape of the failure that produced the lane's outage in the first place. Its writer targeted a Postgres that went away, nominator_count degraded to null or to a frozen 2026-08-02 mirror covering 564 of 1,031 validators, and no probe, no red check and no exception fired. The read path degrades so gracefully that the outage was invisible. nominator_positions -- the other output of the same scan -- got a watchdog in #9273 for precisely this reason; its sibling now has the matching one, same shape as neurons/positions/chain-detail before it: one MAX() read, a pure rule, a summary rather than a throw, one exception per stale tick. An EMPTY table alerts too, since that is the state in which every nominator_count is still coming from the frozen mirror. THE POSITIONS THRESHOLD UNDERCUT ITS OWN PRODUCER. It was 6 hours, chosen while that lane had no producer at all, on the reasoning that six hours was "several missed passes at any plausible cadence". The producer now feeding it runs on a 24h tick (VALIDATOR_NOMINATORS_POLL_SECS defaults to 24*3600, and one scan writes both tables), so a healthy lane presents an age anywhere in [0h, 24h+scan] and a 6h threshold would have alerted for roughly three quarters of every day. It has not fired yet only because the table is still empty and takes the no_rows branch instead; it would have started the moment the re-enabled producer posted. An alarm that always fires is one nobody reads. Both thresholds are now 30h -- one missed pass plus slack for the scan itself (~4 min at the measured ~3,100 rows/sec) and cron jitter. Derived from the cadence, not picked, and stated as such in both headers so the next cadence change has somewhere to land. Tests pin the regression directly: a capture from the middle of a 24h cycle must be quiet. Cron 19,49 collides with nothing in workers/config.ts and stays off the */5 raw-capture and */15 probe grids; the suite asserts both that uniqueness and that wrangler.jsonc actually declares the trigger, since dispatch keys on the literal string and an undeclared cron is silently dead code. Closes #9310 Part of #9146
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
nominator_countis served from the lakehouse mirror ofchain.validator_nominator_counts(#9276), and that mirror is frozen. Measured live 2026-08-03 via R2 SQL: 112,550 rows, newestcaptured_at2026-08-02T01:38Z— the retired box's last scan — covering only 564 of the 1,031 validators the leaderboard serves. 467 have no row at all, and the scan never recorded a zero, so an absent row conflates "no nominators" with "not scanned". Coverage can only fall from here.This is a sink problem, not a producer problem.
metagraphed-infra's poller Container already carries the fullSubtensorModule::Alphascan this table is derived from (src/bin/poller/jobs/validator_nominators.rs, 24h tick). It is one of the five lanesDockerfile.pollerleaves disabled, verbatim: "The five Postgres-backed lanes stay disabled until they have a Cloudflare-native sink." The scan needs no rehosting — only its write target does.Same shape as #9273/#9292, which gave
nominator_positions— the other output of that same scan — its D1 sink.What changed
migrations/d1/0012addsvalidator_nominator_countson D1, upserted on(hotkey)with the usualcaptured_at <= excluded.captured_atstaleness guard. Table and writer column list are asserted against each other in both directions (0007's anti-drift guarantee). No prune — the producer's Alpha pass is exhaustive, but it chunks across requests, so "absent from this batch" ≠ "absent from the scan", and only the latter would license a delete.src/validator-nominator-counts-d1-write.tsreusesneurons-d1-write's statement building, so one place owns the Workers-binding budget of 100 bound parameters per statement (not the 1,200wrangler d1 executepermits). Asserted as the platform limit in the tests, not as a constant we picked.handleValidatorNominatorCountsSyncstops answering503 hyperdrive binding unavailable(chore(workers): delete the dead Postgres tier now that no Hyperdrive binding exists #9193) and writes the batch to D1.buildGlobalValidators/buildValidatorDetail, which were handed a hardcoded empty map / null count. It joins againstneuronsinside SQLite rather than inlining a key list: ~1,031 hotkeys against a 100-parameter cap means a correlated subquery costs zero bound parameters and one query, where an IN list would cost a dozen round trips.On #9276's overlay
Deliberately left in place.
validatorHotkeysNeedingCountonly collects hotkeys whose count is still null and returns early when there are none — so the overlay covers what D1 hasn't received yet and stops firing on its own once a full scan lands. The cutover is a property of the data, not of a deploy. Removing it is a follow-up once the table is populated, not part of this PR.Failure posture
Any failed read degrades to null — exactly what this tier served before the table existed. A broken read is a lost enrichment, never a wrong number. A D1 write failure is a 502, never a silent success.
Validation
Patch coverage verified by diff intersection (not whole-file %): 0 uncovered lines and 0 uncovered branches across the changed lines of
workers/data-api.tsand all ofsrc/validator-nominator-counts-d1-write.ts.Follow-ups (deliberately not in this PR)
validator_nominators.rsat this route and enabling the lane inPOLLER_ONLY. Until that lands, the table stays empty and the overlay keeps serving the frozen mirror, which is exactly today's behavior.nominator_positions. This gap existed because nothing noticed the writer was gone; that applies identically here, but it's a separate concern from restoring the lane and would near-double this diff.Closes #9301
Part of #9146