Add streaming compression for replication full sync - #20
Open
roshkhatri wants to merge 23 commits into
Open
Conversation
Adds replication wire compression on top of valkey-io#3531 with lz4 as the first supported codec for the incremental replication stream. The replication stream from primary to replica is wrapped in a VKCS envelope (using STREAM_KIND_REPL) and compressed as a single long-lived frame at the per-replica buffer layer. Default behavior is unchanged with 'replcompression no'; existing replicas without the new capability stay uncompressed. Negotiation is per-replica via the existing PSYNC handshake; a new REPLICA_CAPA_COMPRESSION capability lets each side opt in independently. Compression runs inline on the IO thread that owns the replica's write job; no dedicated compression thread, no IPC, no reordering. Optional sticky thread affinity (lazy ownership + event-driven rebalance) keeps the long-lived LZ4 frame state on a single IO thread for cache locality. Configs: replcompression bool, default no repl-compression-thread-affinity bool, default yes Internal constants: REPLICA_CAPA_COMPRESSION (1 << 4) REPL_COMPRESSION_ALGO ALGO_LZ4 REPL_COMPRESSION_LEVEL 0 (LZ4 fast mode) REPL_COMPRESSION_BATCH_LIMIT 1 MB raw input per dispatch REPL_STREAM_DECODER_OUTPUT_MAX 256 MB INFO replication per-replica fields: compression=lz4, compressed_bytes, uncompressed_bytes, compression_ratio, compression_errors, compression_cpu_usec, debug_compression_pending_drains, debug_thread_switches INFO replication server-level (replica side): repl_decompression_errors, repl_decompression_cpu_usec, repl_decompressed_bytes_total, repl_apply_cpu_usec, repl_apply_batches CI adds a test-replication-compression job that runs the replication-tagged integration tests with replcompression=yes to exercise compression across the broader replication test surface. Tests: 18 streamReader push-mode unit tests + 3 replCompression unit tests + 27 integration tests. Performance (BlockMesh tweets, 3M keys x ~315 byte JSON values, 1,073 MB uncompressed per replica, 30 clients, pipeline 50, 2 cross-region replicas): LZ4 level 0 (default): 0.48 ratio, 52% bandwidth saved, 2.5s compression CPU per replica, <1% throughput overhead vs uncompressed baseline. Affinity ON vs OFF: throughput unchanged (118.6K vs 118.1K keys/s) but thread switches drop from ~800K to ~30 per replica. ZSTD support follows in valkey-io#3798. Related to valkey-io#3531. Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
# Conflicts: # src/io_threads.c # src/networking.c # src/replication.c
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
The compression CI job (--config replcompression yes) ran replication-buffer.tcl and the dual-channel buffer-memory tests, which assert exact replication buffer/backlog memory and byte volumes. Compression legitimately changes those (per-replica codec buffers add ~1MB scratch; fewer wire bytes let replicas keep up), so the assertions fail under compression even though replication is correct. Drop the repl-compression tag from replication-buffer.tcl and the two dual-channel blocks holding the memory tests; they still run uncompressed in the regular job. Functional dual-channel coverage stays in the compression job. Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
…thakaggarwal97/valkey into replication-streaming-compression-pr # Conflicts: # src/compression_stream.c # src/compression_stream.h # src/config.c # src/rdb.c # src/server.h # src/unit/test_compression.cpp # tests/integration/rdb-compression.tcl # valkey.conf
…tate and plaintext passthrough Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
…val (fix macOS -Werror) Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
…ion-streaming-compression-pr
roshkhatri
force-pushed
the
repl-streaming-comression-fullsync-pr
branch
from
June 24, 2026 23:49
25ab585 to
acabef9
Compare
|
tests are failing :P |
roshkhatri
force-pushed
the
repl-streaming-comression-fullsync-pr
branch
3 times, most recently
from
June 27, 2026 02:38
f018f82 to
0a5e7e5
Compare
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
…ion-streaming-compression-pr
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
roshkhatri
force-pushed
the
repl-streaming-comression-fullsync-pr
branch
from
June 30, 2026 21:56
203fbec to
4e269f4
Compare
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
roshkhatri
force-pushed
the
repl-streaming-comression-fullsync-pr
branch
from
June 30, 2026 22:53
4e269f4 to
efc525d
Compare
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
Compress the full-sync RDB payload for replicas that negotiate the compression capability, covering disk-based, diskless, and dual-channel sync. Reuses the streaming RDB codec (VCS/LZ4) and the existing repl-compression config (no | lz4-stream); default off. The payload is compressed only when every replica in the attaching cohort advertised REPLICA_CAPA_COMPRESSION (cohort-AND); a mixed group falls back to plaintext for all. Disk: decided in startBgsaveForReplication via RDBFLAGS_REPL_COMPRESSED_SYNC and gated by repl-compression in rdbSaveInternal. Diskless: decided in rdbSaveToReplicasSockets; the $EOF framing stays plaintext and the frame checksum replaces the RDB CRC64. The replica decompresses inline socket loads (both $EOF-mark and size-framed) and via the existing rdbLoad auto-detect for disk-receive; the dual-channel handshake advertises the capability. Adds streamReaderValidateFrameEnd and STREAM_READER_ERROR_TRUNCATED so a link dropped mid-frame is a recoverable resync rather than codec corruption. Tests: disk grouping (all/mixed/none/off), diskless, dual-channel, disk-receive load, checksum on/off, full->incremental handoff, teardown+resync, mid-transfer truncation recovery, and an in-frame corruption unit test. Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
roshkhatri
force-pushed
the
repl-streaming-comression-fullsync-pr
branch
from
June 30, 2026 23:09
efc525d to
ef02d03
Compare
roshkhatri
force-pushed
the
replication-streaming-compression-pr
branch
from
July 30, 2026 08:31
850bd66 to
dbe7581
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds streaming compression of the full-sync RDB payload for replicas that negotiate the compression capability, covering disk-based sync, diskless sync, and dual-channel sync. Builds on the streaming RDB codec (VCS/LZ4) and reuses the existing
repl-compressionconfig (no|lz4-stream); default off. Stacked onreplication-streaming-compression-pr(the incremental-stream compression work).Negotiation and the cohort rule
A replica advertises
REPLCONF capa compression(REPLICA_CAPA_COMPRESSION) whenrepl-compressionis enabled; the primary compresses only when it also hasrepl-compressionenabled. Because a single full sync (a shared disk file, or one connset/pipe) serves a whole cohort of replicas, the decision is the AND of the capability over every replica in the attaching cohort: if any replica is not capable, the payload is sent plaintext so all of them can load it. This mirrors the existingskip_rdb_checksumcohort-AND.Disk-based
The cohort AND is computed in
startBgsaveForReplication(the genericrdbSaveBackgroundis not replication-aware) and passed viaRDBFLAGS_REPL_COMPRESSED_SYNC;rdbSaveInternalapplies therepl-compressiongate, decoupled from therdbcompressionpersistence setting.Diskless
The cohort AND is computed inside
rdbSaveToReplicasSocketsalongsideskip_rdb_checksum. Only the RDB body is wrapped as a VCS frame; the$EOFframing stays plaintext and the frame's own checksum replaces the RDB CRC64. The replica decompresses inline when it loads from the socket (repl-diskless-loadenabled), or via the existingrdbLoadauto-detect when it receives the stream to disk first (the default, and dual-channel). The dual-channel handshake now advertises the compression capability.streamReaderValidateFrameEndis added so the replica can validate a closed compressed frame without consuming the caller-owned trailing EOF mark.Testing
Disk grouping (all/mixed/none capable), diskless and dual-channel sync, default disk-receive load, checksum on/off, the compressed-full-sync to compressed-incremental handoff,
REPLICAOF NO ONEteardown and resync, and the AOF-baseBGREWRITEAOFfallback (a compression-capable replica's compressed disk sync RDB cannot be reused as the AOF base).valkey.confdocuments the expandedrepl-compressioncoverage.Follow-up
A
src/unitgtest for thestreamReaderValidateFrameEndsplit is tracked as a follow-up (currently covered by the integration tests).