mining: Optionally make extra pool persistent#313
Open
pdath wants to merge 3 commits into
Open
Conversation
When rejecttokens=1 tokens are stored in the extra pool, rather than the mempool. When Bitcoin Knots is restarted, the extra pool is lost. The issue this creates is that compact block reconstructions for tokens becomes very slow after a restart, because the extra pool is empty. The causes Bitcoin Knots to have to fetch thousands of transactions from peers. Examples are (with "debug=cmpctblock"): 2026-06-10T06:22:16.634294Z [cmpctblock] Successfully reconstructed block 00000000000000000000aa2798940f3de06a3b6b8bf845bb872a1322a8ce6688 with 1 txn prefilled, 2181 txn from mempool (incl at least 804 from extra pool) and 2297 txn requested 2026-06-10T06:47:45.132446Z [cmpctblock] Successfully reconstructed block 000000000000000000009b2bb14d59eb073cb3aa0a1a370236b5822b99663340 with 1 txn prefilled, 2847 txn from mempool (incl at least 435 from extra pool) and 959 txn requested 2026-06-10T06:55:41.588544Z [cmpctblock] Successfully reconstructed block 00000000000000000000a51c141257c439152ffb45c6a3bded318ffaef161b78 with 1 txn prefilled, 2523 txn from mempool (incl at least 521 from extra pool) and 1800 txn requested To resolve this issue I propose treating the extra pool like the mempool, and saving/loading the state to extrapool.dat, but only when rejecttokens=1. The issue does not happen when rejecttokens=0, because tokens then get stored in mempool.dat, and persisted across restarts.
Retropex
reviewed
Jul 2, 2026
kwsantiago
reviewed
Jul 3, 2026
kwsantiago
reviewed
Jul 3, 2026
kwsantiago
reviewed
Jul 3, 2026
pdath
changed the title
pdath
force-pushed
the
feature-persist-extra-pool
branch
from
ff8b3c8 to
266e83b
Compare
…pool Implement reviewer feedback for extra pool persistence: - Introduce -persistextrapool option (default: 0) to independently control whether the extra pool is persisted to disk, decoupling from -rejecttokens - Add memory DoS protection during LoadExtraPool: enforce per-TX size limit (BLOCK_RECONSTRUCTION_EXTRA_TXN_PER_TXN_SIZE_LIMIT) and cumulative memory limit (max_mem_bytes) to prevent malicious extrapool.dat from exhausting memory - Fix ring buffer position inconsistency: derive vExtraTxnForCompactIt from pool size on load instead of persisting position to disk, maintaining the invariant that TXs are at [0, pool.size()) with next write at pool.size() % max_count - Simplify interface: GetExtraPoolForDump() returns vector only (no position), SetExtraPool() takes pool and memusage only (position derived from pool size) - Add Doxygen comments to all public functions in extrapool_persist - Update file format: version(8) + count(8) + transactions (24 bytes removed) - Tighten functional test assertion: require len(requested) == 0 instead of len(requested) < num_token_txs to ensure all persisted transactions are actually reloaded - Update copyright headers to 2026 The Bitcoin Knots
pdath
force-pushed
the
feature-persist-extra-pool
branch
from
266e83b to
1447153
Compare
…error handling. - Replace pre-allocation with push_back() to eliminate null entries when skipping oversized transactions during LoadExtraPool() - Guard against division by zero: check max_count > 0 before modulo operation in pool_pos calculation - Add test case for zero-capacity (max_count=0) handling This ensures the extra pool vector only contains valid transactions, matching runtime behavior where oversized TXs are rejected before insertion.
Author
|
I have added an extra commit to resolve a UBSAN failure in the unit test. I have also further improved some of the error handling. Highlights:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When
rejecttokens=1, tokens are stored in the extra pool rather than the mempool. When Bitcoin Knots is restarted, the extra pool is lost.The issue this creates is that compact block reconstructions for tokens become very slow after a restart because the extra pool is empty. The causes Bitcoin Knots to have to fetch thousands of transactions from peers. Examples are (with "debug=cmpctblock"):
2026-06-10T06:22:16.634294Z [cmpctblock] Successfully reconstructed block 00000000000000000000aa2798940f3de06a3b6b8bf845bb872a1322a8ce6688 with 1 txn prefilled, 2181 txn from mempool (incl at least 804 from extra pool) and 2297 txn requested 2026-06-10T06:47:45.132446Z [cmpctblock] Successfully reconstructed block 000000000000000000009b2bb14d59eb073cb3aa0a1a370236b5822b99663340 with 1 txn prefilled, 2847 txn from mempool (incl at least 435 from extra pool) and 959 txn requested 2026-06-10T06:55:41.588544Z [cmpctblock] Successfully reconstructed block 00000000000000000000a51c141257c439152ffb45c6a3bded318ffaef161b78 with 1 txn prefilled, 2523 txn from mempool (incl at least 521 from extra pool) and 1800 txn requestedThe issue does not happen when
rejecttokens=0, because tokens then get stored in "mempool.dat", and persisted across restarts.To resolve this issue, I propose treating the extra pool like the mempool and saving/loading the state to "extrapool.dat", controlled by a new configuration setting
persistextrapool, that works similarly topersistmempool.persistextrapoolwill default to 0 (off).