-
Notifications
You must be signed in to change notification settings - Fork 38
docs: propose the Zaino driving port design #1414
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: dev
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| # Decouple the mempool from chain state in Zaino's driving port | ||
|
|
||
| Zallet today learns of a new block through a side effect: Zaino's mempool | ||
| stream ends when the chain tip changes, and the sync loop treats that closure | ||
| as its only new-block signal (zcash/zallet, | ||
| `zallet-core/src/components/sync.rs:136`). We decided that the driving port | ||
| Zallet will consume from Zaino must not carry this coupling. The port serves | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is confusing to me, the mempool requires this functionality in some form, even if not for zallet then for the light-client interface. #1416 separates this functionality from the core mempool service so that users that dont want it are not forced to use it. If we are separating the mempool from the ChainIndex / ChainView as discussed then why would we tie zallet into any specific configuration? It should be up to the zallet devs to decide whether they want just the MempoolService or the CoherentMempoolService. |
||
| the mempool through an abstraction that stands apart from chain state, because | ||
| the protocol itself imposes no structural relation between them: a transaction | ||
| commits to no block hash. Consensus binds a transaction only to chain | ||
| *state* — shielded anchors, nullifier non-membership, transparent prevouts, | ||
| expiry height, and the consensus branch id — so a mempool view is coherent | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think there is a bit of confusion about the issue / bug that lead to the greater tip tracking we have, it is not that a transaction must be tied to a particular block hash, the issue is that the mempool must be correct for the chain being served. The first problem is, if a reorg occurs and the mempool and non-finalised state are out of sync, transactions could be missing completely (not shown in mempool or chain), or shown twice (both in mempool and chain), the fix that #1416 adds for the coherentmempool (freeze until the two re-agree) fixes this race condition. The second is that clients that use the |
||
| only relative to the tip it was validated against, and the port expresses | ||
| exactly that. | ||
|
|
||
| The port therefore provides an explicit chain tip-change subscription, and its | ||
| mempool capability is an independent stream whose view is tagged with the tip | ||
| it was validated against. Zallet composes the two: on a tip event it resyncs | ||
| and resubscribes to the mempool. This costs Zallet a sync-loop rewrite during | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. See above, this is not just a Zallet API, it is also a part of the lightclient interface. |
||
| adoption. | ||
|
|
||
| ## Considered Options | ||
|
|
||
| We rejected canonizing the closure idiom (keeping "stream ends on tip change" | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No we have not. we must keep this behaviour, just separate from core mempool functionality. |
||
| as the contract), because it hides a chain-state dependency in the mempool | ||
| abstraction's fine print and forces every future engine to implement the | ||
| entanglement. We rejected tip polling by Zallet, because it adds latency and | ||
| busy-work to every implementation's hot path. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # The driving port hands consumers raw consensus-serialized bytes | ||
|
|
||
| Zaino's driving port traffics in consensus-serialized bytes: blocks, | ||
| transactions, and treestate frontiers cross the boundary as bytes. Only | ||
| identifiers and locators — heights, block hashes, txids, consensus branch | ||
| ids — are typed, and those types come from `zaino-primitives` | ||
| (zingolabs/zaino#1402), a zero-dependency crate of checked domain types. We | ||
| chose byte payloads over typed domain payloads because the port's consumers | ||
| already parse consensus bytes into the types they actually want: Zallet reads | ||
| blocks and transactions into `zcash_primitives` types (zcash/zallet, | ||
| `backends/zaino/src/chain.rs`), so a typed payload would only add a | ||
| conversion detour. | ||
|
|
||
| This deliberately deviates from the typed philosophy of Zaino's driven ports | ||
| (zingolabs/zaino#1402: "source traits return typed `Block`, not `Vec<u8>`"). | ||
| The two boundaries face opposite directions: a driven port feeds Zaino's own | ||
| core, which needs structured data, while the driving port feeds external | ||
| consumers that own their parsing. | ||
|
|
||
| ## Considered Options | ||
|
|
||
| We rejected typing the payloads with `zaino-primitives`, because consumers | ||
| would deserialize Zaino's types only to convert them into their own; the | ||
| identifier types are the useful subset, and the port takes exactly those. We | ||
| rejected typing the payloads with `zcash_primitives`, because Zaino's engines | ||
| would then take on librustzcash version-lockstep with the wallet stack — the | ||
| coupling the Z3 stack keeps working to shed. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| # Snapshot pinning is unconditional | ||
|
|
||
| The driving port's snapshot carries its strongest guarantee: every read | ||
| through a snapshot observes the chain as of the pinned tip, and that data | ||
| stays readable while any clone of the snapshot lives — across reorgs, | ||
| without exception. We decided the guarantee is unconditional. An engine | ||
| that cannot retain the pinned view for as long as any clone lives is not | ||
| an implementation of the port. | ||
|
|
||
| We chose this because the alternative — letting a snapshot lapse — would | ||
| export the engine's storage policy into every driver's sync loop. A lapse | ||
| answer turns "retake and resync" into a code path every driver must write | ||
| and test for an event that a correctly built engine never produces, and it | ||
| demotes the port's strongest promise to a conditional one. The burden | ||
| lands instead on the adapter, which must retain the pinned view — | ||
| reference-count it, copy it, or hold the branch it was answered from — | ||
| while any clone lives. Hash-keyed reads over append-only finalised state | ||
| pin for free; the tip-relative surfaces (unspent outpoints at an address, | ||
| outpoint spend status) are what the adapter must keep answerable as of the | ||
| pinned tip after the chain moves on. The mock exemplifies the shape: | ||
| snapshots hold an `Arc` of the chain as of their creation, and a scripted | ||
| mutation swaps the `Arc` while live snapshots keep the old one. | ||
|
|
||
| One reorg race remains, and it lives outside the snapshot: taking a | ||
| snapshot can race the engine's view swap, and that failure crosses the | ||
| port as a transient backend failure. Reads through a snapshot never race | ||
| a reorg. | ||
|
|
||
| ## Considered Options | ||
|
|
||
| We rejected pinning-with-lapse (a `Lapsed` answer any pinned read may | ||
| return once the engine has discarded the pinned view), because it forces | ||
| every driver to carry a lapse branch, weakens the guarantee that gives | ||
| snapshots their meaning, and shapes the port around one engine's retention | ||
| limits. We rejected leaving the guarantee unstated, because an | ||
| implementation would then satisfy the letter of the trait while serving | ||
| mixed-chain reads during reorgs — the exact torn-read hazard the snapshot | ||
| exists to exclude. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| # The Zallet driving port | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Are we sure we want a single bespoke driving port for zallet? what are the benefits / disadvantages vs providing generic, functionality based driving ports? Eg. providing MempoolService, ChainView, BlockChainSource, etc and letting Zallet use what they want how they want, rather than tying them into a set configuration. |
||
|
|
||
| This context covers the contract through which Zallet (and later other | ||
| consumers) drives Zaino. The language below was settled in a design review | ||
| held in the Zallet repository, where the consumer side of this contract lives. | ||
|
|
||
| ## Language | ||
|
|
||
| **Mempool**: | ||
| The set of transactions awaiting mining. It is chain-validity-dependent but | ||
| chain-structure-independent: consensus binds its transactions to chain state | ||
| (anchors, nullifiers, prevouts, expiry, branch id), yet its representation and | ||
| service abstraction stand apart from chain state. A mempool view is coherent | ||
| only relative to the chain tip it was validated against. | ||
|
|
||
| **Driving port**: | ||
| The contract through which consumers drive Zaino. It spans chain reads, | ||
| mempool streaming, and transaction broadcast. Zallet is its first driver, and | ||
| zainod-for-lightclients is its expected second; zingolib is not a consumer. | ||
| _Avoid_: primary port, wallet API, indexer interface | ||
|
|
||
| **Snapshot**: | ||
| A pinned view of the best chain, taken through the port. Every read through | ||
| a snapshot observes the chain as of the tip it was pinned to, and that data | ||
| stays readable while any clone of the snapshot lives — across reorgs. The | ||
| guarantee is unconditional: an engine must retain the pinned view for as | ||
| long as any clone lives, and an engine that cannot is not an implementation | ||
| of the port. The pinned tip is a property of the snapshot, not a query | ||
| against the engine. | ||
| _Avoid_: view (Zallet's consumer-side name for the same idea) | ||
|
|
||
| **Conformance kit**: | ||
| The executable half of the port contract: a suite of invariant cases that | ||
| every implementation of the port must pass, shipped with the port itself. | ||
| An engine adapter passing the kit is what qualifies it as an implementation. | ||
|
|
||
| **Block locator**: | ||
| A descending-by-height sample of blocks a driver believes are on the chain, | ||
| offered to the port to locate the fork point between the driver's view and | ||
| the best chain. A block's identity is its hash; its height is position, so | ||
| presence is always judged by hash. | ||
|
|
||
| **Reported upgrades**: | ||
| The network-upgrade schedule as the validator reports it, ascending by | ||
| activation height, each upgrade active or pending relative to the current | ||
| tip. The port passes the validator's schedule through — activation heights | ||
| come from the validator, never from constants of the port's own — and | ||
| drivers feed it into their node-compatibility checks. | ||
|
|
||
| **Broadcast**: | ||
| Submission of a transaction to the network through the port — there is no | ||
| side channel. Acceptance returns the txid and means the engine admitted the | ||
| transaction to its mempool, where the mempool stream makes it observable. | ||
| Rejection is a domain answer (malformed bytes, or a validation rejection | ||
| with the engine's reason), distinct from a backend failure. | ||
|
|
||
| **Tip event**: | ||
| The port's explicit signal that the best chain moved, carrying the new tip. | ||
| A fresh subscription delivers the current tip first; events may coalesce | ||
| under load, but the latest tip always arrives; a reorg is a tip change like | ||
| any other, and the new tip may sit at or below the old height. This is how | ||
| drivers learn of new blocks — never through a side effect of another stream. | ||
|
|
||
| **Fork point**: | ||
| The block a driver's view and the pinned best chain last share: the locator | ||
| entry whose block sits highest on the pinned chain, judged by hash and never | ||
| by the heights the locator claims. Everything the driver holds above the | ||
| fork point is not on the pinned chain. Views sharing no block have no fork | ||
| point. | ||
|
|
||
| **Treestate**: | ||
| The note commitment tree state of every shielded pool — Sapling, Orchard, | ||
| and Ironwood — as of one block of the pinned chain. Every in-view height has | ||
| a treestate; what varies per pool is whether its frontier is present, and an | ||
| absent frontier means an empty tree, never an error (zcash/zallet#455). | ||
|
|
||
| **Transaction status**: | ||
| Where a snapshot places a transaction: mined in the pinned best chain, | ||
| orphaned onto a non-best branch, or unknown. The status speaks only of chain | ||
| state — mempool presence is deliberately not a status, because the mempool | ||
| stands apart from chain state and is observed through its own stream. | ||
|
|
||
| **Spend status**: | ||
| Whether the pinned view considers a transparent output spent. Spentness is | ||
| authoritative — it comes from the engine's UTXO set — while naming the | ||
| spending transaction may need a per-outpoint spend index the engine does not | ||
| maintain; an engine that knows the output is spent but not by whom says so | ||
| explicitly, and the driver retries rather than concluding the output is | ||
| unspent (ZcashFoundation/zebra#10806). An outpoint no in-view transaction | ||
| created has no spend status at all. | ||
|
|
||
| **Mempool view**: | ||
| The driver-side accumulation of one mempool subscription: the engine's | ||
| mempool as of the tagged tip, plus possibly transactions the engine has | ||
| since evicted. The stream signals arrivals only, so the view is a | ||
| superset of the engine's mempool, trued up by resubscribing; mempool | ||
| presence is a hint, never authoritative. | ||
|
|
||
| **Transient failure**: | ||
| A backend failure likely to resolve on retry, such as taking a snapshot | ||
| while the engine's view is mid-swap. Reads through a snapshot never race a | ||
| reorg — pinning is unconditional — so the reorg window touches only the | ||
| unpinned surface. Every backend failure crossing the port is classified as | ||
| transient or fatal, and drivers decide retry from that classification | ||
| alone. A domain rejection is an answer, not a failure, and is never | ||
| transient. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not a side effect, it is an API that was specifically designed and agreed upon in both zallet API meetings and the lightclient working group calls. This is misleading.