feat: add configurable include set to gate captured tables#193
Merged
Conversation
…d tables
By default the producer hook captures every Create/Drop/Insert event
that reaches `T::EventCapture::capture_events`. This adds a sudo-only
extrinsic, `ProverDbIndexer::set_include_rules`, that replaces an
on-chain rule list against which every captured event is matched. An
empty rule list (the default) preserves the current "index everything"
behavior; a non-empty list gates the producer so only events for the
listed namespaces or fully-qualified tables reach the offchain DB —
saving offchain storage and OCW work for chains that index a strict
subset of their tables.
Shape:
- `sxt-core::prover_db_indexer::IncludeRule` (`Namespace(TableNamespace)`
| `Table(TableIdentifier)`), stored as
`BoundedVec<IncludeRule, ConstU32<MAX_INCLUDE_RULES = 256>>`.
- Pure helper `table_matches_rules(&TableIdentifier, &[IncludeRule])`
with the "empty = match all" semantics.
- `pallet_prover_db_indexer::IncludeSet` value-query storage, with a
matching `IncludeRulesSet { count }` event.
- `Pallet::capture_events` reads the rule set once per call and skips
the offchain-DB write entirely when nothing passes the filter, so
there's no per-event storage cost in the default case.
Tests cover the extrinsic (root succeeds + event deposited; non-root
rejected; storage unchanged on reject) and the producer side via
`offchain_index::set` → `persist_offchain_overlay()` →
`ext.offchain_db()`: empty set passes everything, namespace rules pass
only that namespace, table rules pass only the exact identifier, and a
non-matching set writes neither the payload nor the high-water-mark.
1.69.0Bug Fixes
Features
|
tlovell-sxt
reviewed
Jun 1, 2026
…de CLI config
Per review feedback: the include set is a node-local indexing policy
("which tables does *this* node's prb-service receive?"), not a
chain-wide one. Different indexer nodes can subscribe to different
subsets. Capture stays unfiltered on every validator so the offchain
queue mirrors the full block; the filter applies only when the OCW
forwards events.
Removed (on-chain):
- `IncludeSet` storage value
- `set_include_rules` extrinsic
- `IncludeRulesSet` event
- Producer-side filter in `EventCapture::capture_events`
- `IncludeRules` BoundedVec alias, `MAX_INCLUDE_RULES`,
`MaxEncodedLen` / `TypeInfo` derives on `IncludeRule`
Added (per-node):
- `PROVER_DB_INCLUDE_KEY` offchain-storage key in `sxt-core`
- `--prover-db-include-file <path>` flag on the node binary, plus a
`PROVER_DB_INCLUDE_FILE` env var. JSON schema:
[ { "kind": "namespace", "value": "ETHEREUM_MT" },
{ "kind": "table", "namespace": "OTHER_NS", "name": "SPECIAL" } ]
Identifiers get uppercased before storage so the byte-match against
the on-chain canonical form works without per-operator vigilance.
- Service-side `configure_prover_db_include` parses the file and
SCALE-encodes a `Vec<IncludeRule>` into offchain local storage at
startup, before the first block. Mirrors the existing
`configure_prover_db_url` shape exactly. Rejects the
include-file-without-url misconfiguration up front.
- Consumer-side filter: `run_consumer` reads the rules once per OCW
round and threads them through `forward_block` → `forward_events`,
where each `BlockEvent` is matched against `table_matches_rules` and
skipped (Drops included, uniformly) when it doesn't pass.
Tests dropped the four producer-filter tests and the two extrinsic
tests; added `ocw_forwards_only_events_matching_include_set` (mixed
events, only matchers POST) and `ocw_with_empty_include_set_forwards_everything`
(absent key ⇒ match-all). Kept the existing `capture_events_writes_all_events_unconditionally`
regression. 12/12 pass.
tlovell-sxt
reviewed
Jun 3, 2026
…nfig; drop the file flag for a wildcard CLI flag
Two review points addressed together:
1. Make invalid states unrepresentable. The consumer used to live across
two offchain keys — PROVER_DB_URL_KEY (`Vec<u8>`) and
PROVER_DB_INCLUDE_KEY (`Vec<IncludeRule>`) — which let an operator
accidentally have a stale include set with no URL. Replaced both
with a single PROVER_DB_CONFIG_KEY holding a SCALE-encoded
`ProverDbConsumerConfig { url: String, include: Vec<IncludeRule> }`
in sxt-core. URL is non-optional in the type: writing the config
*is* the act of enabling the consumer.
2. Replace `--prover-db-include-file` with `--prover-db-include`. A file
path complicates deployment (volume mounts, ConfigMap + mountPath
coordination) for what's now a tiny string list. The new flag is a
clap `value_delimiter = ','` Vec<String> with env-var fallback
(`PROVER_DB_INCLUDE`), and the grammar moves to a wildcard form:
NAMESPACE.NAME — exact table
NAMESPACE.* — every table in NAMESPACE
*.* — match everything (degenerate; same as omitting)
*.NAME — rejected (runtime rule type can't express it)
In K8s this is one ConfigMap key referenced by `envFrom` from every
pod in the prover stack instead of a mounted file.
CLI shape uses clap::Args + #[clap(flatten)]: a new ProverDbConsumerCli
group holds both fields. The service-side resolver in
`configure_prover_db_consumer` is the single seam where the optional CLI
shape is validated into the non-optional `ProverDbConsumerConfig` —
fails loud if include patterns are set without a URL, or if any pattern
is malformed.
Pallet OCW: one StorageValueRef read replaces the previous two; URL
encoding error variant dropped (String can't be invalid UTF-8). Tests
seed the unified key via a new `setup_with_config(include)` helper;
existing `setup_with_url()` becomes a thin alias around an empty include
set. 12/12 pallet tests pass.
End-to-end: the harness orchestrator passes
`--prover-db-include <comma-joined patterns>` instead of rendering a
JSON file. All 9 example scenarios (89 actions) green under
`--auto-start --all`.
…arse_include_pattern - cli.rs: blank-line separator before the '`*.NAME` is rejected' paragraph so it stops being parsed as a list-item continuation (doc_lazy_continuation). - service.rs: dedent the wrap line under the '`"*.*"`' bullet from 23 spaces to 2 (doc_overindented_list_items). - service.rs: `parse_include_pattern` returns `Result<_, ServiceError>`, which clippy now flags at 176 bytes (result_large_err). `ServiceError` is a substrate type we don't own, so apply the same `#[expect(..., reason = ...)]` already used on `configure_prover_db_consumer`.
tlovell-sxt
reviewed
Jun 3, 2026
…> parsed via FromStr
Per review: split each side of the dot into IdentFilter
{Wildcard, Ident(String)} and combine into TableIdentifierFilter, both
with matches() + FromStr. Drops IncludeRule, table_matches_rules, and
service-side parse_include_pattern; clap parses --prover-db-include
values directly. *.NAME now allowed.
Per review: surface the match-all default in --help and let the include vec stay non-empty. Cross-flag guard on URL-absent dropped since the default now makes that branch unreachable; URL absence remains the single dormancy signal.
tlovell-sxt
reviewed
Jun 8, 2026
…, and table_matches_filters 13 tests covering wildcard/literal matching, FromStr happy + error paths (empty side, over-length, missing dot, multi-dot), and the match-all-on-empty behaviour of table_matches_filters.
Per review: split "no filters configured" from "match-all configured" so they're distinct, addressable states. table_matches_filters now returns false on an empty list; callers wanting match-all must pass an explicit *.* filter (the CLI default already does this). Pallet tests: setup_with_url seeds *.* instead of an empty vec; a new ocw_with_empty_include_set_forwards_nothing asserts the new empty-list-rejects-everything behaviour.
tlovell-sxt
approved these changes
Jun 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a per-node include set the OCW consumer uses to gate which captured
events get forwarded to its prover-db indexer. The chain stays
unfiltered — every validator captures every Create/Drop/Insert event
into the offchain queue, exactly as before. Filtering happens only when
this node's OCW forwards to its indexer, so two indexer nodes can
subscribe to different subsets without affecting each other or the
chain.
Operator surface
Two flags on
sxt-node, both env-var-friendly via clap:--prover-db-url <URL>— toggles the consumer on; required.--prover-db-include <patterns>— comma-separated wildcard list(also via
PROVER_DB_INCLUDEor repeated flags). Grammar:NAMESPACE.NAME— exact tableNAMESPACE.*— every table in that namespace*.*— match everything (degenerate; same as omitting)*.NAME— rejectedIdentifiers are uppercased before storage so matches are byte-exact
against the on-chain canonical form. Empty list ⇒ "forward
everything" (the default).
Misconfigurations fail at startup:
--prover-db-urlwithout--enable-offchain-indexing— refused.--prover-db-includewithout--prover-db-url— refused.Implementation
sxt-core: one new offchain keyPROVER_DB_CONFIG_KEYholding aSCALE-encoded
ProverDbConsumerConfig { url: String, include: Vec<IncludeRule> }.Single key so "consumer on" and "consumer off with stale filter" are
unrepresentable.
run_consumer): oneStorageValueRefdecode perround, then the existing forward path with the include set threaded
into
forward_events. EachBlockEvent's table runs throughtable_matches_rulesbefore the HTTP POST — Drops included,uniformly.
capture_events): unchanged. Every event still hitsthe offchain queue regardless of any node-local filter.
clap::ArgsflattenedProverDbConsumerCligroup;validated into the non-optional
ProverDbConsumerConfigat serviceinit by
configure_prover_db_consumer.Tests
filter:
ocw_forwards_only_events_matching_include_set(mixedmatching/non-matching events → only matching ones POSTed) and
ocw_with_empty_include_set_forwards_everything(absence ⇒ match-all).
sxt-integration-harness --auto-start:9/9 scenarios, 89 actions green. The
include_filter.tomlscenariocreates
INCL_F_GOOD.T1andINCL_F_BAD.T2on chain, submits toboth, and asserts only
INCL_F_GOOD.T1appears on prb-service.Test plan
cargo test -p pallet-prover-db-indexercargo check --workspace--prover-db-url http://… --prover-db-include "ETHEREUM_MT.*"and confirm only that namespace's events POST to prb-service.