canonical: strict validate + halt-on-invalid (never drop a backfilled row)#862
Merged
Conversation
Canonical (cannon-backfilled) data is authoritative — a dropped row is a permanent gap in history. For canonical_* tables, route.ErrInvalidEvent now returns a non-permanent invalidEventError so the caller halts and retries (cannon: no checkpoint advance; consumoor: NAK/redeliver) rather than dropping. Non-canonical (live/sentry) invalid events still drop — they are point-in-time and cannot be re-fetched. isCanonical() keys off the canonical_ table prefix.
…bsent on required fields)
…bsent on required fields)
…nt on required fields)
…r-if-absent on required fields)
…nt on required fields)
…if-absent on required fields)
…sent on required fields)
…-absent on required fields)
…x aliasing test fixture Apply the audit's missing guards onto the existing (#851) validate()s rather than the workflow agents' from-scratch versions (their pre-#851 base conflicted): consolidation gains block_root/block_version/meta_network_name; attestation_reward gains epoch_start_date_time/meta_network_name. Enrich the elaborated-attestation aliasing test fixture to satisfy the new strict validate (AttestationData + block identity + meta).
…_insert) b826b35 added distributed_foreground_insert=1 as a canonical_ table default but left TestTableConfigForAppliesCanonicalDefaults / MergesInsertSettings asserting the pre-feature settings, leaving master red. Update expectations to the intended behaviour.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes canonical (cannon-backfilled) data strict: a dropped row is a permanent gap in history, so an invalid canonical event now HALTS the writer (cannon doesn't advance its checkpoint; consumoor NAKs for redelivery) instead of being silently dropped or zero-filled. Non-canonical (live/sentry) events are unchanged — they're point-in-time and still drop. The halt keys off the
canonical_table prefix via a non-permanentinvalidEventError.Adds/extends route
validate()on 16 canonical tables so every spec-required field is presence-checked (error-if-absent), driven by a field-by-field audit of each table against the beacon-API OpenAPIrequired:arrays and the consensus-specs SSZ containers. Empty-valid fields (validator index 0, genesis slot/epoch 0, zero/negative rewards, pre-finality ZERO_HASH checkpoint roots, empty lists) are deliberately left untouched so strictness never stalls on legitimate data. Supersedes #860 (this is the spec-verified version of that cleanup).Also fixes two stale consumoor tests left red on master by b826b35 (canonical
distributed_foreground_insertdefault). The validators table's lossy Nullable→NOT NULL flip + re-derive is handled in a separate follow-up PR.