feat(scaffold): driving-port + runtime composition (WIP — shapes evolving fast)#1418
Draft
nachog00 wants to merge 146 commits into
Draft
feat(scaffold): driving-port + runtime composition (WIP — shapes evolving fast)#1418nachog00 wants to merge 146 commits into
nachog00 wants to merge 146 commits into
Conversation
Scaffolds the DAG-driven sync engine with sealed marker types for the two classification axes (InputScope × CompositionType), compile-time-enforced extract/merge trait families, and stub modules for DAG, engine, backend, and provisioner. The erased layer will be replaced with a partially-erased IndexPipeline approach in the next commit.
Delta types stay inside each index's pipeline boundary — the engine only sees Ctx in and Vec<WriteOp> out. No dyn Any, no downcasting. SyncEngine is now generic over Ctx (the provisioner's block context).
Replaces raw &'static str and u64 across the crate with domain-specific newtypes for compile-time type safety.
PhaseIndex wraps u32 for DAG phase indices. BlockHeight documents the overlap with zaino-state's type and notes the path toward a shared zaino-primitives crate. Provisioner Height associated type collapsed to BlockHeight.
Bridge module connects typed trait hierarchy to IndexPipeline<Ctx> for runtime dispatch. Three BlockLocal bridges (Append, Monoidal, Fold) validate that the scope × composition trait bounds compose. DAG builder implements Kahn's algorithm for topological sort, cycle detection, phase assignment, and firing rule derivation. Pipeline trait refactored from extract_block+merge_batch to process_batch (MVP decision documented — true north is split extract_one+merge_batch with opaque DeltaToken intermediates).
Engine implements sync_range: splits blocks into batches, processes each batch phase-by-phase, commits write ops to backend, flushes. Testing module provides InMemoryBackend (HashMap-based), MockProvisioner (generates TestBlockContext with predictable values), and three toy indexes covering all three composition types (Append, Monoidal, Fold). Two end-to-end tests validate the full pipeline: provisioner → engine → bridge dispatch → extract → merge → commit → backend assertions.
Sealed BridgeDispatch trait maps (Scope, Composition) marker pairs to bridge constructors, enabling a blanket IntoIndexPipeline impl — index authors never write bridge wiring by hand. IndexSet collects indexes declaratively and SyncEngine::from_index_set builds the DAG internally.
IndexDef::Context renamed to IndexDef::BlockContext — the subset of block data each index needs. ProvideContext<T> trait with identity blanket lets the set-wide context project down to each index's view. Bridges insert .context() before extraction, keeping indexes reusable across different index sets with different provisioner contexts.
ProvideContext returns owned T (identity blanket clones). Each toy index gets its own module under testing/toy_indexes/ with a local Context type. ContextRequirements trait added to descriptor.rs for deriving provisioner needs from context types (not yet wired in).
ProvideContext is the single source of truth for provisioner needs. The compiler enforces that the set-wide context can project into each index's BlockContext at IndexSet::with time. No runtime flags needed — unused context fields are dead code, signalling unnecessary RPC calls. Drops bitflags dependency.
Scheduler combines the static DependencyDag with runtime state to answer what work is ready. Tracks per-index extraction progress, pending merges, and committed batches. Downstream indexes unblock only when all dependency firing rules are satisfied. Six tests cover phase-zero readiness, dependency blocking/unblocking, batch advancement, and independent index parallelism.
BatchHandle<State> with FullyExtracted and Merged markers enforces valid transition ordering at compile time. extraction_done returns Option<BatchHandle<FullyExtracted>>, merge_done consumes it and returns BatchHandle<Merged>, batch_committed consumes that. The engine cannot skip or reorder steps — invalid sequences are type errors.
Bridges become stateful: Mutex<Vec<Delta>> buffer for extraction, Mutex<Vec<WriteOp>> for merged output. process_batch kept as default method for backward compat with batch-loop engine. Engine and tests unchanged — they call process_batch which delegates to the three phases internally.
Three near-identical bridge structs replaced by one generic struct parameterized by a MergeStrategy marker (AppendStrategy, MonoidalStrategy, FoldStrategy). Merge now stores domain-typed state (Vec<Delta>, Accumulator, FoldState) — persist is the serialization boundary where domain state becomes WriteOps.
Engine is now a thin driver loop over the Scheduler. For each block, it asks the scheduler for ready extraction jobs, calls extract_one on pipelines, and reports back. At batch boundaries the scheduler emits BatchHandle<FullyExtracted> which the engine chains through merge → persist → commit via phantom-typed handles. Partial final batches supported via effective_batch_size. Old monolithic process_batch loop removed.
IndexDef gains NAME, DEPENDENCIES, SOURCE_ACCESS as associated constants with defaults. descriptor() becomes a provided method that derives scope and composition from the marker types automatically. Index implementors no longer construct Descriptor manually — they declare axes and name, the trait does the rest.
Delta is now Entry { height: BlockHeight, value: u32 } instead of
Vec<(Vec<u8>, Vec<u8>)>. Extract produces pure domain data — no
serialization. to_write_ops is the single serialization point,
isolated from extraction logic.
Schema<M> declares key/value types and maps merge results to entries. Encode handles byte serialization on the types themselves. The bridge does the mechanical plumbing — no index code touches bytes. Using a generic type parameter M (not an associated type) avoids the cycle error that arises when the merged type references the index's own associated types (e.g. Vec<Self::Delta>).
BlockValue, BlockCount, RunningSum replace raw u32/u64 in Schema impls. Each newtype carries its own Encode impl.
The scheduler now tracks blocks_available — a watermark updated by the engine as the provisioner supplies blocks. Extractions are only emitted for blocks below this watermark. New Task enum unifies ExtractJob and CompleteBatch into a single ready_work() output. The engine can dispatch all returned tasks concurrently — the scheduler guarantees safety. Also adds provisioner_done() for signaling end-of-stream, separating "how many blocks exist" from "how many are available right now."
BlockBuffer is a sliding-window store for provisioned block contexts. Contexts are Arc-wrapped for async worker access. Eviction is per-batch — blocks drop when the slowest index commits past them. BlockOffset newtype distinguishes global sync-range positions from raw u32. ExtractJob now carries global_offset so the engine can look up blocks in the buffer directly.
The engine now owns a BlockBuffer and uses scheduler.ready_work() for task dispatch instead of manually iterating over a block slice. Blocks are loaded into the buffer, the scheduler gates on availability, and completed batches are evicted once all indexes commit past them.
Verifies the buffer is fully evicted after multi-batch sync — proves the supply/demand lifecycle from load through eviction works end to end.
sync_streaming pulls blocks incrementally from an IntoIterator, interleaving supply (batch_size blocks per pull) with demand (task dispatch). sync_range is now a thin wrapper. Streaming test verifies incremental arrival produces equivalent results with full eviction.
DRYs the task dispatch logic into a shared method so the upcoming async sync_channel can reuse it without duplicating the match arms.
The provisioner runs as a separate task and sends blocks through an mpsc channel. The engine drains available blocks (drain_channel), processes ready tasks (dispatch_tasks), and awaits more blocks when idle (await_block). Channel close signals provisioner completion. Adds tokio dependency (sync feature for library, macros+rt for tests).
Enables sharing pipeline references with spawned extraction tasks. No behavior change — Arc<dyn IndexPipeline> derefs identically to Box.
sync_channel now uses dispatch_tasks_async which spawns each extraction on the blocking thread pool, waits for all to complete, then reports completions sequentially. The scheduler guarantees at most one extraction per index per ready_work() call, so spawned tasks are fully independent.
Both sync and async dispatch now share flush_batch_completions and report_extractions. The async path adds run_extractions_parallel in between. Each method does one thing.
Points zebra patch to nachog00/zebra feat/compact-block-read branch. Adds get_compact_block() to ZebraReadStateAdapter using the new ReadRequest::CompactBlock which skips proof/signature deserialization. Adds compact_block mode to provision-bench for A/B comparison against the full raw_block path. Expected speedup in sandblast era where proofs dominate deserialization cost.
Each mode now uses the lightest ReadRequest that provides the data its index set needs: headers_only → ReadRequest::BlockHeader (no tx deser) headers_spends → ReadRequest::CompactBlock (outpoints, no proofs) current_zaino → ReadRequest::CompactBlock (all compact fields) raw_block → ReadRequest::Block (full deser, baseline) compact_block → ReadRequest::CompactBlock (compact baseline) Adds get_block_header() to ZebraReadStateAdapter.
…c path Primitives: CompactBlock, CompactTransaction, CompactTransparentInput, CompactTransparentOutput, CompactSaplingOutput, CompactOrchardAction — our own types, no zebra dependency. Source: GetCompactBlock trait in zaino-source, one method returning our domain CompactBlock. Adapter: ZebraReadStateAdapter implements GetCompactBlock via ReadRequest::CompactBlock (nachog00/zebra fork). Conversion from zebra's compact types to ours happens inside the adapter via zaino-convert-zebra::compact_block_from_zebra. Indexes: context_from_compact_block builds CurrentZainoContext from our CompactBlock — same output as context_from_block. Bench: sync-bench ReadState path now uses get_compact_block by default. provision-bench modes use optimal ReadRequest per index set. Provisioner throughput: 33,800 blocks/s compact vs ~460 blocks/s full block — 70x improvement across the full 3.41M block chain.
Domain-to-domain conversion via From trait — CompactBlock can be built from a full Block by extracting only indexer-relevant fields. RPC adapter implements GetCompactBlock by fetching the full block (getblock RPC returns full bytes regardless) then converting via From. The network transfer cost is the same but avoids the full domain Block intermediate when callers only need compact data. TODO: once compact_deserialize supports streaming (Reader trait), the RPC path can skip zebra's full Transaction deserialization too.
PreIndexCompactBlock is what source adapters return — block data with proofs/sigs stripped, no indexed state. CompactBlock is the proto- adherent serving format with required ChainMetadata (tree sizes). Eliminates duplicate Compact* sub-types — PreIndexCompactTx reuses TransparentInput, TransparentOutput, SaplingOutput, OrchardAction from the existing transaction module. Renames throughout: GetPreIndexCompactBlock trait, adapters, convert functions, bench binaries. From<&Block> for PreIndexCompactBlock enables the RPC adapter to convert full blocks to the pre-index format without zebra-chain dep.
Adds BridgeDispatch impl for (SelfCumulative, Append), completing the bridge wiring for all non-CrossIndex scope x composition pairs. New toy indexes: - CumulativeLogIndex (SC x Append): appends labeled entries to a running log, prefix depends on prior log length (even/odd). - CumulativeMaxIndex (SC x Fold): tracks running maximum with halving above a threshold. 5 new tests covering single-batch, cross-batch determinism, and state threading for both new indexes. All 39 tests pass.
The MergeStrategy::merge_deltas is a sequential fold for all composition types. For MonoidalStrategy, the associativity guarantee enables tree-shaped parallel reduction via rayon, reducing O(n) to O(log n) depth. Not implemented because current indexes have cheap combine operations where rayon overhead exceeds the gain. The trait boundary already guarantees correctness when implemented.
CumulativeBridge::load_state scans all entries to reconstruct the accumulator. For SC indexes that persist per-height snapshots, only the latest entry is needed. Notes the path to O(1) restart via get_last or range scan, and that Schema design (persist minimal state like a tree frontier) is the key lever.
Ironwood (NU6.3) is a distinct shielded pool that shares Orchard's commitment-tree node type. The inner driving surface's subtree-roots capability and the full-wallet consumer (zallet's get_ironwood_subtree_roots) both need to name it. The sole consumer of this primitive is zaino-source (pass-through, no exhaustive match); legacy zaino-state matches a separate ShieldedPool type, so nothing else breaks.
Trait-algebra stubs for the hexagonal driving side, per the design thread. - zaino-core: pure domain vocabulary over zaino-primitives (refs, locator, events, status, upgrades, serviceability). No async, no runtime. - zaino-service: the inner driving surface — segregated capability traits (Block/Transaction/Treestate/Address/Spend reads + ForkReconcile + engine controls) grouped by backing index, with Snapshot/IndexerService bundles and per-capability errors. RPITIT + BoxStream, no async-trait. - zaino-wallet: published full-wallet client (bytes/id types) + builder — an adapter, not a port. Method bodies are todo!(). - zallet-fit: compile-time proof the client services zallet's Chain/ChainView (faithful shape mirror; flip the commented git dep for the exact check). Registered as workspace members; kept out of default-members (unwired).
A concrete IndexerService over Mutex<Arc<MockChain>> that exemplifies the ADR-0003 pin: snapshot() clones the Arc, mutate() swaps in a new one while live snapshots keep the old. Wiring-complete over all twelve capability traits; most reads return empty / NotServiceable for now (rich data fabrication follows as tests need it). Includes a passing test proving a snapshot pins its tip across a scripted mutation — the executable half of the snapshot contract.
Design-direction note synthesising the NFS survey, Hahn's zaino-store (PR zingolabs#1378) and its review, and ADR-0003. Runtime = a reorg-safe block spine (zaino-store-shaped: Chain + Freezer + Lean-verified sync_step, hands out the snapshot) composed with a typed-index layer (zaino-sync) built only over frozen blocks; NFS typed queries re-derive from the in-memory Chain. Split by layer, not by FS/NFS tier. Open questions recorded: first-class snapshot (contra Hahn's framing, per reviewers), side-branch retention, FS integrity/versioning, the canonical finalized store, and adopt-vs-reimplement scope.
Pressure-tested against zaino-store code (chain.rs, chain_stream.rs, state.rs). Both resolve as additive to Hahn's design: - Q1: ChainState is RwLock<Arc<Chain>>, Chain an immutable Arc<im::Vector> — a first-class client-held snapshot() is a thin add; he only exposed per-call/range reads. Adopt structure, reject "snapshots unnecessary". - Q2: retain side branches (decided) — serve getchaintips/orphaned-fork tx-status/fork-point-by-hash from memory via a companion branch map pinned by the Snapshot. Reorg resolution itself re-fetches, so needs none.
Verified against legacy serving code: no client endpoint reads a body index (txids/sapling/orchard/transparent) at height to serve — those height-keyed reads occur only in reassembly (ephemeral backend) and the internal get_tx_out_set_info accumulator. Single-tx serving indexes into the height list. The reverse indices (hash_to_height, txid_location, transparent_spends) are the only ones queried individually. So the spine stores whole CompactBlocks (ChainMetadata included); the index layer keeps only the individually-queried auxiliary reverse indexes (+ lean headers for fork-walk, future address-history). The 8-way body decomposition is dropped. Full-block/raw-tx remains orthogonal.
…sthrough) Q5: sync_step/find_trim_index/BlockStoreSync are generic over BlockFetcher + ChainState and never touch LMDB/serialization directly, so the Lean-mirrored algorithm ports near-verbatim. Re-skin 3 seams (BlockFetcher -> zaino-source; Block ids -> zaino-primitives; storage -> zaino-persistence). Placement: a zaino-nfs crate (brings tokio + lmdb), not a zaino-core module. Full blocks/raw-tx: served by validator passthrough, not stored (legacy get_fullblock_bytes_from_node pattern). Full-block cache is a future optimization only if whole-chain full-block streaming becomes hot.
…indow Key insight: the finalized block store is just one index in the sync engine (compact_block: height -> CompactBlock). The 8-way body split is coincidental; collapse it. So the sync engine IS the FS store, and Hahn narrows to the reorg-prone window only (Chain + find_trim_index + snapshot) — no Freezer, forward-fill, or sync_step. Split by operational state: bulk catch-up (sync engine parallel build) vs tip-follow (Hahn Chain), meeting at the freeze horizon. Supersedes the block-spine + typed-index framing. Reframes Q4/Q5 accordingly.
Capability algebra (trait stubs) for the two delegated components: - zaino-fs: FinalisedState — finalised-state semantics (compact_block + aux lookups + watermark; bulk_build_to + freeze). Hides the sync engine. - zaino-nfs: NonFinalisedState + NfsSnapshot — the reorg window (snapshot, subscribe_tip, frozen(), follow(); snapshot reads + chain_tips for Q2). - zaino-service: add CompactBlockRead (compact from FS index or NFS Chain); BlockRead(full) becomes passthrough. Mock + bundle updated. Doc: placement = thin runtime + three delegated component crates (zaino-fs/nfs/mempool). Builds; mock snapshot pin test passes.
Runtime<F,N> composes a FinalisedState + NonFinalisedState.
- Compose seam: Runtime::snapshot() pins RuntimeSnapshot { fs, nfs
snapshot, watermark }; RuntimeSnapshot impls CompactBlockRead by
routing FS (<= watermark) vs NFS (> watermark) at the height. The rest
of the Snapshot bundle follows the same shape.
- Loop-wiring seam: RuntimeBuilder::init sketches bulk-build -> tip-follow
-> freeze-forward (needs executor + zaino-source bound).
Builds.
Replace the single FsError with per-operation types (HeightReadError, LookupError, AddressReadError, BuildError, FreezeError) in a zaino-fs error module — each method carries only the failures it can actually produce (above-watermark, not-enabled, source, commit...). Applying the same principle to zaino-nfs surfaces an asymmetry: NFS snapshot reads are in-memory over the pinned Chain, so they are infallible (miss = None, sync, no Result). Only follow() can fail (FollowError incl. ReorgTooDeep). Runtime snapshot routing updated: FS above-watermark -> NotServiceable, backend -> Fatal; NFS -> Ok.
Locality of correctness: snapshot() returns NfsView::{Ready(S),
Syncing{finalised}} so a consumer cannot read a not-yet-synced window
— there is no S to call reads on. RuntimeSnapshot holds Option<S>; recent
reads route to NFS when ready, else NotServiceable. Closes the catch-up
false-None bug at the type level instead of via a runtime check.
MockFinalisedState + MockNfsSnapshot/MockNonFinalisedState with a shared call recorder. Two passing tests exercise the compose seam end-to-end: - routes_finalised_to_fs_and_recent_to_nfs: reads route FS (<= watermark) vs NFS (> watermark), asserted via the recorder (fs:50, nfs:120). - recent_reads_are_not_serviceable_while_nfs_syncs: a recent read on a Syncing NFS -> NotServiceable, and NFS is never consulted (the type-encoded readiness closes the false-None at runtime, verified).
Living user-stories doc layered by actor (full-wallet lib, lightwallet serving, explorer, operator) + cross-cutting (snapshot coherence, serviceability, reorg safety). Each story traced to a zaino-service capability and the component that serves it, with status. US-1.3 (address unspent across the finalised/recent boundary) drives the next work.
Derived from user story US-1.3. Implementing the merge surfaced an incompatibility: NfsSnapshot had no address re-derivation, so added address_unspent (infallible, in-memory). RuntimeSnapshot::AddressRead now merges finalised UTXOs (FS index) with recent ones (NFS) — a merge, not a route. Test asserts both tiers are consulted. Remaining semantic flagged TODO(US-1.3): drop finalised UTXOs spent within the recent window (needs recent spends-by-address from NFS).
Separate the supervisor from read-composition, and split lib.rs into focused modules (runtime/snapshot/resolve/config/error). - resolve: the composition policy in one place — Strategy per capability (route/merge/passthrough), tier_of (the FS/NFS boundary), the passthrough decision (per-capability strategy + config gate; sync-state/coherence stay on the snapshot), and merge_unspent. Unit-tested. - snapshot: thin capability impls that consult resolve + do only the type-specific work; error mapping at the FS->service boundary. - runtime: supervisor only — owns components, holds config, produces the read-context. No query logic. Config threads through to the snapshot. 6 tests pass (3 resolve unit + 3 routing/merge integration).
Data Zaino doesn't store — full `Block`s, raw transactions — is answered by the validator directly, keyed by hash so the read stays reorg-coherent. Model this as a `Passthrough` trait and thread it through as a third generic `P` on `Runtime<F, N, P>` / `RuntimeSnapshot<F, S, P>`, matching the other component seams: static dispatch, native `async fn`, no boxing. - passthrough.rs: `Passthrough` (full_block / raw_transaction) + PassthroughError. - runtime/snapshot: carry `Arc<P>`; `init` takes the source by value. - snapshot: BlockRead::block and TransactionRead::transaction route to the validator, gated by `resolve::passthrough_allowed` (config gate). - routing tests: MockPassthrough; US-1.7 cases — by-hash passthrough consults neither tier, and a config-disabled passthrough is NotServiceable without hitting the validator.
Passthrough isn't a bespoke seam — it's the same data from the same origin. The local index is a cache built over the validator; passing through is reading that same source directly when the cache can't (or doesn't store the data). Model the indexer's read capabilities by provenance: direct (both backend and validator), synthetic (indexing output — backend only), and not-locally-stored (backend declines to cache — validator only). The local backend's read surface is thus a superset of the validator's; passthrough serves exactly the validator-serviceable subset from the validator instead of the backend. - Replace the hand-rolled `Passthrough` trait with `PassthroughSource`: a named subset over the existing `zaino-source` driven ports (GetBlockByHash, GetTransaction), with a blanket impl so any validator adapter qualifies. A synthetic capability has no port here, so a validator fallback for it won't compile — NotServiceable-when-local-can't is true by construction. - Collapse the third generic to one `Src`: the dependency fs builds from, nfs follows, and the read path passes through. `Runtime<F, N, Src>`. - Snapshot block/tx reads route to the source ports; source errors map to the service read errors (Domain::NotFound -> None, Fetch -> Transient). The raw-bytes -> parsed `Transaction` conversion is left as a marked seam. - Extract the component mocks into tests/support/mocks.rs (shared via #[path]); routing.rs now drives them through a build_runtime helper.
The passthrough surface was bound one layer too low — straight onto the
zaino-source RPC ports, whose transport shape (TransactionResponse{bytes},
QueryError) leaked upward: the snapshot was left holding a raw->parsed tx
conversion that isn't its concern (a dangling todo!).
A driven port is domain-shaped by definition (dependency inversion: the domain
owns the abstraction, infrastructure conforms). So PassthroughSource becomes a
repository-style read port in domain terms — block_by_hash -> Option<Block>,
transaction -> Option<Transaction>, a domain PassthroughError. The validator
adapter (a separate crate, over zaino-source) is the anti-corruption layer that
translates transport results/errors and parses raw bytes; the runtime never
sees the transport surface.
Consequences:
- The runtime no longer depends on zaino-source; it composes over zaino-core
domain ports only. Transport coupling moves to the (future) adapter crate.
- The raw->parsed tx seam dissolves: transaction() returns a domain type; the
adapter owns parsing.
- PassthroughError documents the responsibility split: transport mechanism
(retry/backoff/classification) lives below the port in the resilience layer;
only a domain-classified "unavailable" residue crosses it; the runtime
projects that onto its serving contract (Transient/NotServiceable) as serving
policy, never handling transport itself. The diagnostic string is opaque.
- MockSource implements the domain port directly.
Sweep the application-state axes a read resolves against — NFS ready/syncing, passthrough enabled/disabled, height tier — and for every cell assert both the outcome (serviceable vs NotServiceable) and exactly which providers were consulted, so a degraded state can never silently misroute: - route: a recent read while syncing is NotServiceable with neither tier hit (no fall-through to FS for a height it doesn't own); independent of passthrough config. - merge: syncing -> NotServiceable, nothing consulted, even with passthrough enabled — a synthetic capability has no validator fallback, by construction. - passthrough: keyed by hash, so independent of NFS sync; gated only by config.
Deployment variants (zainod vs lib; RPC subsets) reduce to *which index sets FS builds*. Model that in the type surface: split the monolithic `FinalisedState` into a `FinalisedSpine` (always present — the block store, intrinsic height/hash/treestate derivations, and ingest) plus addon reverse indexes, each in its own module: - `indexes::TxLocationIndex` — txid -> location - `indexes::SpendIndex` — spent-outpoint status - `indexes::AddressIndex` — transparent-address history (privacy-sensitive; belongs behind a non-default feature) `FinalisedState` remains as a convenience bundle (supertrait + blanket impl) of the whole set. Consumers bound on exactly the traits they use, so omitting an index is a compile-time-smaller capability set, not a runtime miss — not building one also means the sync engine never extracts its delta (less work per block). The runtime demonstrates the payoff: `RuntimeSnapshot`'s `AddressRead` impl now requires `F: AddressIndex`, so a minimal FS makes address history *unrepresentable* rather than NotServiceable. Other read impls bound on `FinalisedSpine`; the supervisor only needs the spine. (Send-future capture still forces the untouched generics to `Send + Sync`.)
Mirror the FS split on the non-finalised side, for modelling clarity and to make the FS/NFS symmetry legible: split the monolithic `NfsSnapshot` into an `NfsSpine` (window essentials + reorg/side-branch queries) plus re-derived facet traits, each in its own module: - `facts::NfsSpendFacts` — recent spend status - `facts::NfsAddressFacts` — recent address unspent `NfsSnapshot` stays as the convenience bundle (supertrait + blanket impl). Consumers bound on exactly the spine/facet traits they use, so the same feature that omits an FS index can omit its NFS facet — a disabled capability is off on *both* tiers by construction. The idiom stays NFS's, deliberately *not* unified with FS: reads are sync + infallible (in-memory over the pinned Chain), and the facets are re-derived on demand from the ~100 retained blocks — no persistent index (the window is tiny and reorg-churny, so there's nothing worth materializing or rebuilding). Same questions as the FS indexes, opposite storage strategy — dictated by the ranges. Runtime read impls now bound per-facet (`AddressRead` needs `S: NfsAddressFacts`, routed reads need `S: NfsSpine`), matching the FS-side per-capability bounds.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1402 (modular sync engine). Its branch isn't on the upstream
yet, so this PR opens against
devand its diff therefore includes #1402'scommits. The scaffold delta on top of the sync engine — what's actually new here
(+~3.5k lines, all additive) — is this compare:
nachog00/zaino@feat/zaino-primitives...feat/driving-port-scaffold
Once #1402 merges, a rebase drops those commits and this becomes just the scaffold.
What this is
The consumer-facing (driving) side of the hexagon, and the runtime that
composes the sync engine's output into a served indexer. Where the sync-engine
PR builds the driven side (sources, indexes, persistence), this scaffolds:
zaino-core— pure domain vocabulary (re-exports primitives + refs,locators, events, status, serviceability).
zaino-service— the inner driving surface: per-capability read/controltraits (
BlockRead,CompactBlockRead,AddressRead, …) +IndexerServicebundle, with a mock behind a
testingfeature.zaino-fs—FinalisedState: domain semantics over the finalised half.Pins a concrete index set and hides
zaino-sync/zaino-indexes/persistence —"consumers see finalised state, not indices."
zaino-nfs—NonFinalisedState: the reorg window (adopts hhanh00'sChain/find_trim_indexnarrowly), publishing pinned snapshots.zaino-runtime— the supervisor: owns the components, produces a pinnedcross-tier
RuntimeSnapshot, and delegates read-composition to aresolvepolicy module.
zaino-wallet,live-tests/zallet-fit— outer client + a shape-onlymirror of zallet's
Chaintrait, to pressure-test the surface against a realembedder.
Runtime composition (the part worth looking at)
Reads resolve by one of three strategies, kept in one tested policy module
(
resolve) so neither the supervisor nor the per-capability code re-derives it:Passthrough is modelled as a domain-shaped driven port (
PassthroughSource),not the validator's transport surface: it's the validator-serviceable subset of
the indexer's read capabilities, expressed in domain terms. Capability provenance
— direct (both tiers), synthetic (indexing output, local only), not-locally-
stored (validator only) — determines who can answer, and a synthetic capability
has no passthrough method by construction, so a bad fallback won't compile.
What actually runs vs. what's a sketch
(
NfsView::{Ready, Syncing}→ recent readsNotServiceablewhile syncing,never a false miss); the US-1.3 address merge; US-1.7 full-block/raw-tx
passthrough; and an anti-misrouting scenario matrix over
(NFS ready/syncing × passthrough on/off × height tier) asserting both the
outcome and which providers were (not) consulted.
todo!: mostIndexerServicereads, the loop wiring ininit(bulk-build → tip-follow → freeze-forward), by-hash→height resolution,the concrete
Fs<Src, Bk>and its index-set wiring, the validator adapter.Verify:
cargo test -p zaino-runtime(and-p zaino-service,-p zaino-fs,-p zaino-nfs,-p zaino-core).Docs
docs/runtime/composition.md— the composition anchor ("FS index engine ⊕ NFSreorg window").
docs/runtime/user-stories.md— living user stories; every capability tracesto one. Shareable mirror: https://hackmd.io/@3751jKAyQcWy0V4Syt4dig/rytNS-ZBfx
Actively-evolving threads (expect churn)
ChainHead/ChainIndexproposal (docs: propose the Zaino driving port design #1414) — thedecompositions are convergent; open decision is sync topology (freeze-handoff
vs independent-sync) and cross-boundary snapshot coherence.
FinalisedStateinto a spine + feature/config-selected addonindex traits, so deployment variants (zainod vs lib; RPC subsets) fall out of
which index sets are built.
zaino-sourceto expose{Domain | Unavailable}upward (transportmechanism stays below the port); writing the validator adapter (the ACL).