canonical_beacon_validators: store the real API value (FAR_FUTURE_EPOCH/0) instead of NULL#863
Merged
Merged
Conversation
The route's setOptionalEpoch mapped both 0 and FAR_FUTURE_EPOCH (2^64-1) to NULL and nulled a zero balance — discarding values the beacon API returns verbatim (the Validator spec marks these required and documents 'FAR_FUTURE_EPOCH if not exited/activated'). Conflating real data with absence made NULL ambiguous. Keep the columns Nullable(UInt64) (NO type migration — the table is ~350B rows; a type-change mutation would be huge and pointless) and instead always write the exact uint64 the API reports as a set value: balance 0, genesis epoch 0, and FAR_FUTURE_EPOCH are all preserved. Halt (route.ErrInvalidEvent) on a missing index/status/data/slashed, and on empty pubkey/withdrawal_credentials in the fanout tables, rather than skipping or empty-filling. New rows are unambiguous immediately; historical NULLs stay losslessly recoverable by status and can be backfilled later if desired.
samcm
force-pushed
the
feat/canonical-validators-non-nullable
branch
from
92e7411 to
4a4cf42
Compare
samcm
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The route's
setOptionalEpochmapped both 0 andFAR_FUTURE_EPOCH(2^64-1) to NULL and nulled a zero balance — discarding values the beacon API returns verbatim. The Validator schema marks these fieldsrequiredand documents 'FAR_FUTURE_EPOCH if not exited/activated', so NULL was conflating real data with absence and making the column ambiguous.No schema migration.
canonical_beacon_validatorsis ~350B rows (≈2.3M validators × every epoch); aNullable→UInt64type-change mutation would rewrite the whole table for no benefit. Instead the columns stayNullable(UInt64)and the flatten always writes the exact uint64 the API reports as a set value (balance 0, genesis epoch 0, and FAR_FUTURE_EPOCH all preserved). New rows are unambiguous immediately; historical NULLs remain losslessly recoverable bystatus(pending → FAR_FUTURE, else genesis 0) and can be backfilled later with a scopedALTER UPDATEif wanted.Also makes the validators routes strict (builds on #862's halt mechanism): halt on a missing index/status/data/slashed, and on empty pubkey/withdrawal_credentials in the fanout tables, instead of silently skipping or empty-filling.