PoC: replace the mempool mirror with Bitcoin Core's TxCollection#599
Draft
Sjors wants to merge 19 commits into
Draft
PoC: replace the mempool mirror with Bitcoin Core's TxCollection#599Sjors wants to merge 19 commits into
TxCollection#599Sjors wants to merge 19 commits into
Conversation
Bitcoin Core submitBlock requires a fully assembled block, so JDS must retain non-coinbase transaction bodies from DeclareMiningJob validation. As a prerequisite for upcoming PushSolution handling with v32 IPC support, JdResponse::Success now carries txdata and no longer returns redundant txid_list. JDS derives txids directly from txdata when validating merkle-root/merkle-path consistency for SetCustomMiningJob, without storing an extra txid cache in DeclaredCustomJob. This addresses an earlier design blindspot and does not change v30/v31 validation behavior.
Add a full v32x unix-capnp runtime for both Job Declaration Protocol and Template Distribution Protocol, and wire V32X through common enum dispatch and version selection. Include the v32 bitcoin-capnp-types dependency pin, lockfile updates, and docs/examples updates so version 32 is discoverable and selectable. Also align shared JDP interfaces across versions by switching JdRequest::PushSolution to carry a fully assembled Block. v30/v31 handlers are updated to the new request shape for consistency, while remaining functionally unchanged (PushSolution stays unimplemented there because those IPC schemas do not expose submitBlock).
Reconstruct a full block at JDS when receiving PushSolution and submit it to Bitcoin Core via submitBlock. Follow the clarified Job Declaration semantics from sv2-spec. Reference: stratum-mining/sv2-spec#188 Reference: stratum-mining/sv2-spec#189 Design choice (KISS): JDS keeps only the latest declared custom job per downstream connection and only attempts PushSolution propagation against that entry. It does not attempt propagation for previously declared jobs.
Sjors
changed the title
TxCollectionTxCollection
This was referenced Jul 7, 2026
Contributor
Author
|
Ironically we still need a limited transaction cache on the JDS side, because the current Instead for now the new commit holds on to any transaction the JDS requested, which avoids round-trips back to the miner. It should be much simpler than the original |
Point bitcoin_capnp_types_v32 at Sjors/bitcoin-capnp-types 2026/07/tx-collection (2140-dev/bitcoin-capnp-types#29), which adds the capnp bindings for the TxCollection interface introduced by bitcoin/bitcoin#35671. Revert to the 2140-dev repository (or a crates.io release) once that PR is merged.
The v32 integration test harness previously fell back to a hardcoded local build path. Since the TxCollection work depends on a Bitcoin Core build of bitcoin/bitcoin#35671, require the BITCOIN_CORE_V32_BINARY environment variable instead and explain how to produce the binary. Remove once an official Bitcoin Core release ships TxCollection and v32 follows the standard release-binary download flow.
Instead of stubbing the v32 binary, build Bitcoin Core from the bitcoin/bitcoin#35671 (TxCollection) PR branch in the Integration Tests workflow, with ccache to keep warm-cache runs fast. This follows the approach used by 2140-dev/bitcoin-capnp-types#29. Remove once an official Bitcoin Core release ships TxCollection.
Reject declared jobs whose wtxid list contains duplicates (invalid-job) and silently drop ProvideMissingTransactions.Success transactions that are not part of the declared job; any wtxid left uncovered is simply reported as missing again. Both are protocol violations that the mirror-based validators happened to tolerate. Enforcing them at the jd-server layer establishes an invariant the upcoming TxCollection-based v32.x backend depends on: Bitcoin Core's collectTxs rejects duplicate wtxids and addMissingTxs rejects out-of-set transactions with RPC-level errors, which would otherwise let a misbehaving downstream tear down the shared IPC connection.
DeclareMiningJob stale classification at the jd-server layer previously compared the full validation context (prev_hash, nbits, min_ntime), which can misclassify errors: min_ntime can increase without a tip change, and nbits comparison is unnecessary for stale-tip detection. JdResponse::Error and JdResponse::MissingTransactions now carry only the chain tip the validator operated against. Error carries an Option<BlockHash> so internal failures (where no tip was established) are never misclassified as stale-chain-tip. On the jd-server side, DeclaredCustomJob now stores the declared-against prev_hash directly plus an Option<ValidatedJobData> (nbits + txdata), replacing the validation_context/txdata/validated field triple whose consistency was only enforced by convention. This is a partial implementation of stratum-mining#597 (the version-specific handlers still use their internal validation-context drift heuristics) and prepares for the v32.x TxCollection backend, which has no template to take nbits/min_ntime from when transactions are missing.
Bitcoin Core's TxCollection interface (bitcoin/bitcoin#35671) lets the JDS validate an external block template directly against the node's mempool, so the v32.x JDP backend no longer needs to keep an optimistic local mempool mirror in sync via a waitNext monitor loop. This removes the mirror, the monitor task, the startup createNewBlock call, and the force-refresh retry logic, along with their staleness races (stratum-mining#268). DeclareMiningJob validation now: 1. fetches the current tip via getTip 2. calls collectTxs with the declared wtxids 3. completes the collection via addMissingTxs when the client provided missing transactions (ProvideMissingTransactions.Success) 4. asks unknownTxPos which transactions are still unknown and, if any, responds with MissingTransactions 5. calls makeTemplate with the declared coinbase (zeroed extranonce, which no contextual check depends on), so Bitcoin Core reconstructs and fully validates the block including the coinbase (BIP34 height, output value, sigops, weight, witness commitment) and returns a BlockTemplate The Success response parameters (prev_hash, nbits, min_ntime) come from the validated template's header (getBlockHeader) and the full transaction list from getBlock, preserving the response contract with jd-server for SetCustomMiningJob validation and PushSolution block reconstruction. The collection and template are destroyed once the response is assembled. Stale-tip classification compares the tip before and after validation, plus a JDS-side check of the declared BIP34 height against the expected next height so a stale job is reported as stale-chain-tip rather than the generic invalid-job that bad-cb-height would map to. Behavioral notes: - readiness now waits for IBD to finish by polling isInitialBlockDownload instead of relying on createNewBlock blocking during IBD. - a future optimization could retain the BlockTemplate and use its submitSolution method for PushSolution instead of reconstructing the full block in jd-server, which would also remove txdata from the Success response.
Instead of destroying the BlockTemplate returned by TxCollection::makeTemplate, the v32.x backend retains it keyed by (downstream_id, request_id). A PushSolution now sends only the header fields and the solved coinbase to Bitcoin Core, which reconstructs the block from the retained template via BlockTemplate::submitSolution -- the block itself never crosses the IPC boundary in either direction. This removes the full-block transfer (getBlock) at declaration time and the block reconstruction in jd-server, along with the per-job txdata copy jd-server used to hold. SetCustomMiningJob merkle-path validation now uses the coinbase merkle branch reported by the validator: getCoinbaseMerklePath on v32.x (the branch is independent of the template's dummy coinbase), and a branch computed from the mirror's txids on v30.x/v31.x. Since the backend now holds per-job node-side state, jd-server mirrors its job lifecycle over the request channel so retained templates are freed when they can no longer receive a solution: - ReleaseDeclaredJob on declaration errors, expired tokens (janitor), failed SetCustomMiningJob validation, and active-job replacement - CleanupDownstream on downstream disconnect - a re-declared request id replaces (and destroys) the old template - PushSolution consumes the template v30.x/v31.x ignore the new lifecycle requests; their PushSolution handling remains unimplemented (their IPC interface has no way to submit blocks).
…se validation Two gaps in the tp -> jdc -> jds end-to-end coverage: - The ProvideMissingTransactions round was only tested on failure paths: jds_ask_for_missing_transactions runs the JDC and JDS against nodes with incompatible chains, so the declaration is expected to fail after the missing transactions are provided, and no existing test mined a block containing a non-coinbase transaction. propagated_from_jds_to_tp_with_missing_transactions runs the JDC and JDS against separate nodes sharing the same chain (via a new sync_chain_from test helper that copies blocks over RPC), with a transaction only in the JDC node's mempool. The declaration must complete a ProvideMissingTransactions round, and the solved block -- which reaches the JDS node exclusively through the JDS, as the JDC -> TP SubmitSolution path is blocked -- must contain that transaction. This exercises collectTxs/addMissingTxs/makeTemplate and the retained-template submitSolution reconstruction end to end. - Nothing checked that a consensus-invalid declared coinbase is rejected at declaration time. The new overpaying-coinbase scenario in bitcoin_core_ipc_jdp_io asserts that a coinbase paying more than the block subsidy is rejected as invalid-job on all supported Bitcoin Core versions (via checkBlock on v30.x/v31.x and via makeTemplate's coinbase validation on v32.x).
Transactions supplied through ProvideMissingTransactions.Success are typically ones the JDS node will never learn over P2P (prioritized or out-of-band transactions that don't meet its mempool policy). Without a cache, every downstream declaring a template with such a transaction -- and every retry -- pays its own ProvideMissingTransactions round. Remember them in a bounded (32 MB), least-recently-used cache keyed by wtxid, consulted only after unknownTxPos reports a transaction missing. Unlike the retired mempool mirror this involves no background synchronization and is never the source of truth: Bitcoin Core decides what is missing, and the cache merely supplies it without asking the client again. The wtxid key commits to the full serialized transaction, so one downstream cannot misrepresent a transaction requested by another; the worst a malicious downstream can do is churn the cache, which degrades to today's behavior. Confirmed transactions stop being looked up and age out on their own. Bitcoin Core itself remembering TxCollection-provided transactions (which would additionally improve relay and compact-block propagation of the solved block) is a possible future replacement; see the discussion in bitcoin/bitcoin#35671. The new bitcoin_core_ipc_jdp_io scenario asserts that a transaction provided once is remembered: a second downstream declaring it succeeds without a missing-transactions round. It runs against all backends, since the v30.x/v31.x mirror already behaves this way (previously untested). The test-harness wallet helpers build a witness-free transaction from an explicitly selected legacy coinbase output so the minimal test coinbase needs no witness commitment, and mine funding blocks in batches to stay under the RPC client timeout when several test nodes mine concurrently.
Sjors
force-pushed
the
2026/07/tx-collection
branch
from
354f23a to
31d3c51
Compare
Contributor
Author
|
Updated to the latest version of 2140-dev/bitcoin-capnp-types#29 which bumped |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
100% vibe coded proof of concept for #598.
Instead of keeping a mempool mirror in sync by polling
waitNext/getBlockevery second (the successor to the originalgetrawmempoolRPC dump), JDS backend Bitcoin Core's node validates declared jobs directly against its mempool using the newTxCollectionIPC interface.Additionally, he JDS submits solutions against the retained (node-side) template (
submitSolution()instead ofsubmitBlock()). The mempool mirror, its monitor loop, and all full-block transfers are gone: a block never crosses the IPC boundary in either direction. Also, the declared coinbase is fully consensus-checked at declaration time instead of when a solution is submitted.The v30.x/v31.x backends are unchanged and keep the mirror.
Depends on
DownstreamState#592 — JDS downstream state isolationhandle_push_solutiononjd_server_sv2+bitcoin_core_sv2#593 —handle_push_solution; this branch is stacked on top of itTxCollectioninterface (draft; an alternative to bitcoin/bitcoin#34020, which would have kept the mirror but fed it more efficiently)TxCollectionAlso
Partially implements
bitcoin_core_sv2: JDP stale-chain-tip detection should useprev_hashonly (avoidmin_ntime/nbits/bip34drift) #597 — prev_hash-only stale classificationTxCollectionis unlikely to actually land in Bitcoin Core v32, but this PR pretends it does so we don't need to add plumbing for yet another version module.Commits
TxCollectionbindings only exist on the Add TxCollection 2140-dev/bitcoin-capnp-types#29 branchTxCollection; tests need a locally built mining: add TxCollection to bandwidth-efficiently validate external block templates bitcoin/bitcoin#35671 binarynbits/min_ntimedrift can misclassify errors (bitcoin_core_sv2: JDP stale-chain-tip detection should useprev_hashonly (avoidmin_ntime/nbits/bip34drift) #597), and the TxCollection backend has no template to take those fields from when transactions are missingcollectTxs/addMissingTxs/unknownTxPos/makeTemplate(with the declared coinbase) replace the mirror andcheckBlocksubmitSolution, removing block reconstruction (and per-jobtxdata) from jd-serverProvideMissingTransactionsround was only tested on failure paths, and no test mined a block containing a transaction or rejected a consensus-invalid coinbase