Add independent pacing controls for bucket garbage collection
Problem
VShard garbage-collects the source copy of a transferred bucket by repeatedly deleting tuples in transactions until the bucket is empty. BUCKET_CHUNK_SIZE bounds the size of each transaction and creates yield points, but garbage collection remains work-conserving: after committing one batch, it immediately starts the next one.
This limits the duration of an individual transaction but does not limit sustained garbage-collection pressure on Tarantool's transaction thread. On large buckets or busy storage nodes, source cleanup can consume most of the available transaction-thread capacity even though bucket transfer concurrency is already constrained.
The cost is especially visible for Vinyl spaces and spaces with secondary indexes, where each logical tuple deletion can require additional index and engine work. Smaller chunks improve scheduling fairness, but they do not reduce the sustained deletion rate.
VShard has controls for bucket movement concurrency, but it has no equivalent independent control for pacing removal of transferred bucket data. Reusing movement controls would also be undesirable because operators may need fast bucket transfer while deliberately spreading source cleanup over a longer period.
Existing behavior
Bucket garbage collection deletes up to BUCKET_CHUNK_SIZE tuples in one transaction, commits, checks the bucket state, and continues immediately. The same constant is also used by bucket sending and receiving, so changing it for garbage collection also changes movement behavior.
collect_bucket_garbage_interval does not pace deletion batches, and ERRINJ_BUCKET_GC_PAUSE is an all-or-nothing testing mechanism rather than an operational rate control.
Proposal
Add independent, dynamically reconfigurable bucket-GC settings:
bucket_gc_batch_size = 1000
bucket_gc_batch_delay = 0
bucket_gc_batch_size defines the maximum number of tuples deleted in one bucket-GC transaction.bucket_gc_batch_delay defines the delay in seconds between non-empty bucket-GC deletion transactions.- The defaults preserve existing behavior: transactions contain at most 1000 deletes and no pacing delay is applied.
- Any delay must happen after
box.commit(), never while a transaction is open.
- Pacing should apply to both automatic garbage collection and
vshard.storage.bucket_delete_garbage().
- Pacing should continue across space and bucket boundaries so a sequence of small spaces or buckets cannot bypass the configured delay.
- Bucket state should be checked again after a pacing yield before more data is deleted.
- No trailing delay should be added after the final deletion transaction when there is no more data to remove.
Rationale
A GC-specific batch size controls the longest uninterrupted deletion burst without changing bucket transfer transaction sizing. A GC-specific delay bounds sustained cleanup pressure independently from transfer admission and concurrency, allowing transferred buckets to move quickly while source cleanup proceeds at a rate appropriate for the storage workload.
A delay-based control is intentionally simple and backward compatible. More elaborate adaptive or deletes-per-second controls could be considered later, but a fixed inter-batch delay provides a predictable operational mechanism without requiring engine-specific load measurements.
Tradeoffs
Pacing increases the time that transferred bucket data remains on the source node and can temporarily increase disk usage. Smaller batches also create more transactions and invoke the bucket-GC event trigger more frequently. These tradeoffs are opt-in because the default delay is zero and the default batch size matches current behavior.
Related work
Add independent pacing controls for bucket garbage collection
Problem
VShard garbage-collects the source copy of a transferred bucket by repeatedly deleting tuples in transactions until the bucket is empty.
BUCKET_CHUNK_SIZEbounds the size of each transaction and creates yield points, but garbage collection remains work-conserving: after committing one batch, it immediately starts the next one.This limits the duration of an individual transaction but does not limit sustained garbage-collection pressure on Tarantool's transaction thread. On large buckets or busy storage nodes, source cleanup can consume most of the available transaction-thread capacity even though bucket transfer concurrency is already constrained.
The cost is especially visible for Vinyl spaces and spaces with secondary indexes, where each logical tuple deletion can require additional index and engine work. Smaller chunks improve scheduling fairness, but they do not reduce the sustained deletion rate.
VShard has controls for bucket movement concurrency, but it has no equivalent independent control for pacing removal of transferred bucket data. Reusing movement controls would also be undesirable because operators may need fast bucket transfer while deliberately spreading source cleanup over a longer period.
Existing behavior
Bucket garbage collection deletes up to
BUCKET_CHUNK_SIZEtuples in one transaction, commits, checks the bucket state, and continues immediately. The same constant is also used by bucket sending and receiving, so changing it for garbage collection also changes movement behavior.collect_bucket_garbage_intervaldoes not pace deletion batches, andERRINJ_BUCKET_GC_PAUSEis an all-or-nothing testing mechanism rather than an operational rate control.Proposal
Add independent, dynamically reconfigurable bucket-GC settings:
bucket_gc_batch_sizedefines the maximum number of tuples deleted in one bucket-GC transaction.bucket_gc_batch_delaydefines the delay in seconds between non-empty bucket-GC deletion transactions.box.commit(), never while a transaction is open.vshard.storage.bucket_delete_garbage().Rationale
A GC-specific batch size controls the longest uninterrupted deletion burst without changing bucket transfer transaction sizing. A GC-specific delay bounds sustained cleanup pressure independently from transfer admission and concurrency, allowing transferred buckets to move quickly while source cleanup proceeds at a rate appropriate for the storage workload.
A delay-based control is intentionally simple and backward compatible. More elaborate adaptive or deletes-per-second controls could be considered later, but a fixed inter-batch delay provides a predictable operational mechanism without requiring engine-specific load measurements.
Tradeoffs
Pacing increases the time that transferred bucket data remains on the source node and can temporarily increase disk usage. Smaller batches also create more transactions and invoke the bucket-GC event trigger more frequently. These tradeoffs are opt-in because the default delay is zero and the default batch size matches current behavior.
Related work