Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions docs/driving-port/0001-decouple-mempool-from-chain-state.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Decouple the mempool from chain state in Zaino's driving port

Zallet today learns of a new block through a side effect: Zaino's mempool

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a side effect, it is an API that was specifically designed and agreed upon in both zallet API meetings and the lightclient working group calls. This is misleading.

stream ends when the chain tip changes, and the sync loop treats that closure
as its only new-block signal (zcash/zallet,
`zallet-core/src/components/sync.rs:136`). We decided that the driving port
Zallet will consume from Zaino must not carry this coupling. The port serves

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is confusing to me, the mempool requires this functionality in some form, even if not for zallet then for the light-client interface. #1416 separates this functionality from the core mempool service so that users that dont want it are not forced to use it.

If we are separating the mempool from the ChainIndex / ChainView as discussed then why would we tie zallet into any specific configuration? It should be up to the zallet devs to decide whether they want just the MempoolService or the CoherentMempoolService.

the mempool through an abstraction that stands apart from chain state, because
the protocol itself imposes no structural relation between them: a transaction
commits to no block hash. Consensus binds a transaction only to chain
*state* — shielded anchors, nullifier non-membership, transparent prevouts,
expiry height, and the consensus branch id — so a mempool view is coherent

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there is a bit of confusion about the issue / bug that lead to the greater tip tracking we have, it is not that a transaction must be tied to a particular block hash, the issue is that the mempool must be correct for the chain being served.

The first problem is, if a reorg occurs and the mempool and non-finalised state are out of sync, transactions could be missing completely (not shown in mempool or chain), or shown twice (both in mempool and chain), the fix that #1416 adds for the coherentmempool (freeze until the two re-agree) fixes this race condition.

The second is that clients that use the get_mempool_stream method as intended see a separate race condition: non-finalised state serves the chain_tip to clients, they then call get_mempool_stream with the given chain tip, and if out of sync with the mempool, return none, also fixed in #1416.

only relative to the tip it was validated against, and the port expresses
exactly that.

The port therefore provides an explicit chain tip-change subscription, and its
mempool capability is an independent stream whose view is tagged with the tip
it was validated against. Zallet composes the two: on a tip event it resyncs
and resubscribes to the mempool. This costs Zallet a sync-loop rewrite during

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above, this is not just a Zallet API, it is also a part of the lightclient interface.

adoption.

## Considered Options

We rejected canonizing the closure idiom (keeping "stream ends on tip change"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No we have not. we must keep this behaviour, just separate from core mempool functionality.

as the contract), because it hides a chain-state dependency in the mempool
abstraction's fine print and forces every future engine to implement the
entanglement. We rejected tip polling by Zallet, because it adds latency and
busy-work to every implementation's hot path.
27 changes: 27 additions & 0 deletions docs/driving-port/0002-raw-bytes-at-the-driving-port.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# The driving port hands consumers raw consensus-serialized bytes

Zaino's driving port traffics in consensus-serialized bytes: blocks,
transactions, and treestate frontiers cross the boundary as bytes. Only
identifiers and locators — heights, block hashes, txids, consensus branch
ids — are typed, and those types come from `zaino-primitives`
(zingolabs/zaino#1402), a zero-dependency crate of checked domain types. We
chose byte payloads over typed domain payloads because the port's consumers
already parse consensus bytes into the types they actually want: Zallet reads
blocks and transactions into `zcash_primitives` types (zcash/zallet,
`backends/zaino/src/chain.rs`), so a typed payload would only add a
conversion detour.

This deliberately deviates from the typed philosophy of Zaino's driven ports
(zingolabs/zaino#1402: "source traits return typed `Block`, not `Vec<u8>`").
The two boundaries face opposite directions: a driven port feeds Zaino's own
core, which needs structured data, while the driving port feeds external
consumers that own their parsing.

## Considered Options

We rejected typing the payloads with `zaino-primitives`, because consumers
would deserialize Zaino's types only to convert them into their own; the
identifier types are the useful subset, and the port takes exactly those. We
rejected typing the payloads with `zcash_primitives`, because Zaino's engines
would then take on librustzcash version-lockstep with the wallet stack — the
coupling the Z3 stack keeps working to shed.
38 changes: 38 additions & 0 deletions docs/driving-port/0003-snapshot-pinning-is-unconditional.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Snapshot pinning is unconditional

The driving port's snapshot carries its strongest guarantee: every read
through a snapshot observes the chain as of the pinned tip, and that data
stays readable while any clone of the snapshot lives — across reorgs,
without exception. We decided the guarantee is unconditional. An engine
that cannot retain the pinned view for as long as any clone lives is not
an implementation of the port.

We chose this because the alternative — letting a snapshot lapse — would
export the engine's storage policy into every driver's sync loop. A lapse
answer turns "retake and resync" into a code path every driver must write
and test for an event that a correctly built engine never produces, and it
demotes the port's strongest promise to a conditional one. The burden
lands instead on the adapter, which must retain the pinned view —
reference-count it, copy it, or hold the branch it was answered from —
while any clone lives. Hash-keyed reads over append-only finalised state
pin for free; the tip-relative surfaces (unspent outpoints at an address,
outpoint spend status) are what the adapter must keep answerable as of the
pinned tip after the chain moves on. The mock exemplifies the shape:
snapshots hold an `Arc` of the chain as of their creation, and a scripted
mutation swaps the `Arc` while live snapshots keep the old one.

One reorg race remains, and it lives outside the snapshot: taking a
snapshot can race the engine's view swap, and that failure crosses the
port as a transient backend failure. Reads through a snapshot never race
a reorg.

## Considered Options

We rejected pinning-with-lapse (a `Lapsed` answer any pinned read may
return once the engine has discarded the pinned view), because it forces
every driver to carry a lapse branch, weakens the guarantee that gives
snapshots their meaning, and shapes the port around one engine's retention
limits. We rejected leaving the guarantee unstated, because an
implementation would then satisfy the letter of the trait while serving
mixed-chain reads during reorgs — the exact torn-read hazard the snapshot
exists to exclude.
106 changes: 106 additions & 0 deletions docs/driving-port/CONTEXT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# The Zallet driving port

@idky137 idky137 Jul 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we sure we want a single bespoke driving port for zallet? what are the benefits / disadvantages vs providing generic, functionality based driving ports?

Eg. providing MempoolService, ChainView, BlockChainSource, etc and letting Zallet use what they want how they want, rather than tying them into a set configuration.


This context covers the contract through which Zallet (and later other
consumers) drives Zaino. The language below was settled in a design review
held in the Zallet repository, where the consumer side of this contract lives.

## Language

**Mempool**:
The set of transactions awaiting mining. It is chain-validity-dependent but
chain-structure-independent: consensus binds its transactions to chain state
(anchors, nullifiers, prevouts, expiry, branch id), yet its representation and
service abstraction stand apart from chain state. A mempool view is coherent
only relative to the chain tip it was validated against.

**Driving port**:
The contract through which consumers drive Zaino. It spans chain reads,
mempool streaming, and transaction broadcast. Zallet is its first driver, and
zainod-for-lightclients is its expected second; zingolib is not a consumer.
_Avoid_: primary port, wallet API, indexer interface

**Snapshot**:
A pinned view of the best chain, taken through the port. Every read through
a snapshot observes the chain as of the tip it was pinned to, and that data
stays readable while any clone of the snapshot lives — across reorgs. The
guarantee is unconditional: an engine must retain the pinned view for as
long as any clone lives, and an engine that cannot is not an implementation
of the port. The pinned tip is a property of the snapshot, not a query
against the engine.
_Avoid_: view (Zallet's consumer-side name for the same idea)

**Conformance kit**:
The executable half of the port contract: a suite of invariant cases that
every implementation of the port must pass, shipped with the port itself.
An engine adapter passing the kit is what qualifies it as an implementation.

**Block locator**:
A descending-by-height sample of blocks a driver believes are on the chain,
offered to the port to locate the fork point between the driver's view and
the best chain. A block's identity is its hash; its height is position, so
presence is always judged by hash.

**Reported upgrades**:
The network-upgrade schedule as the validator reports it, ascending by
activation height, each upgrade active or pending relative to the current
tip. The port passes the validator's schedule through — activation heights
come from the validator, never from constants of the port's own — and
drivers feed it into their node-compatibility checks.

**Broadcast**:
Submission of a transaction to the network through the port — there is no
side channel. Acceptance returns the txid and means the engine admitted the
transaction to its mempool, where the mempool stream makes it observable.
Rejection is a domain answer (malformed bytes, or a validation rejection
with the engine's reason), distinct from a backend failure.

**Tip event**:
The port's explicit signal that the best chain moved, carrying the new tip.
A fresh subscription delivers the current tip first; events may coalesce
under load, but the latest tip always arrives; a reorg is a tip change like
any other, and the new tip may sit at or below the old height. This is how
drivers learn of new blocks — never through a side effect of another stream.

**Fork point**:
The block a driver's view and the pinned best chain last share: the locator
entry whose block sits highest on the pinned chain, judged by hash and never
by the heights the locator claims. Everything the driver holds above the
fork point is not on the pinned chain. Views sharing no block have no fork
point.

**Treestate**:
The note commitment tree state of every shielded pool — Sapling, Orchard,
and Ironwood — as of one block of the pinned chain. Every in-view height has
a treestate; what varies per pool is whether its frontier is present, and an
absent frontier means an empty tree, never an error (zcash/zallet#455).

**Transaction status**:
Where a snapshot places a transaction: mined in the pinned best chain,
orphaned onto a non-best branch, or unknown. The status speaks only of chain
state — mempool presence is deliberately not a status, because the mempool
stands apart from chain state and is observed through its own stream.

**Spend status**:
Whether the pinned view considers a transparent output spent. Spentness is
authoritative — it comes from the engine's UTXO set — while naming the
spending transaction may need a per-outpoint spend index the engine does not
maintain; an engine that knows the output is spent but not by whom says so
explicitly, and the driver retries rather than concluding the output is
unspent (ZcashFoundation/zebra#10806). An outpoint no in-view transaction
created has no spend status at all.

**Mempool view**:
The driver-side accumulation of one mempool subscription: the engine's
mempool as of the tagged tip, plus possibly transactions the engine has
since evicted. The stream signals arrivals only, so the view is a
superset of the engine's mempool, trued up by resubscribing; mempool
presence is a hint, never authoritative.

**Transient failure**:
A backend failure likely to resolve on retry, such as taking a snapshot
while the engine's view is mid-swap. Reads through a snapshot never race a
reorg — pinning is unconditional — so the reorg window touches only the
unpinned surface. Every backend failure crossing the port is classified as
transient or fatal, and drivers decide retry from that classification
alone. A domain rejection is an answer, not a failure, and is never
transient.
Loading