Skip to content

feat(replication): primary → replica data sync stream - #187

Merged
kacy merged 3 commits into
mainfrom
feat/replication-stream
Feb 19, 2026
Merged

feat(replication): primary → replica data sync stream#187
kacy merged 3 commits into
mainfrom
feat/replication-stream

Conversation

@kacy

@kacy kacy commented Feb 19, 2026

Copy link
Copy Markdown
Owner

summary

implements the replication stream between primary and replica nodes (phase 6, pr 2). when a node runs CLUSTER REPLICATE <id>, it connects to the primary's replication port and loads a full-state snapshot, then receives all subsequent mutations as a live stream.

replication.rs — new module with two types:

  • ReplicationServer: accepts TCP connections from replicas. sends a per-shard binary snapshot (using the existing snapshot format in-memory) followed by an unbounded stream of AofRecord frames. on broadcast channel lag, sends MSG_RESYNC and closes; the replica reconnects and resync's
  • ReplicationClient: connects to the primary, loads the snapshot into the local engine via key-based routing, replays incremental records, and reconnects with exponential backoff (500ms → 30s) on any disconnect

wire protocol (little-endian framing):

replica → primary: [version: 1B][num_shards: 2B]
primary → replica: [version: 1B][num_shards: 2B][id_len: 1B][id: N bytes][status: 1B]
per shard:         [MSG_SHARD_SYNC][shard_id: 2B][len: 4B][snapshot bytes]
                   [MSG_SHARD_OFFSET][shard_id: 2B][offset: 8B]
incremental:       [MSG_RECORD][shard_id: 2B][offset: 8B][len: 4B][AofRecord bytes]
on lag:            [MSG_RESYNC]  → replica reconnects

cluster.rs — adds set_engine, start_replication_server, start_replication_client, and replication_info. cluster_replicate now connects to the primary after topology update. replication port = data_port + gossip_offset + 2.

server.rs / main.rs — broadcast channel (65536 capacity) is created when cluster mode is enabled, injected into EngineConfig, and the engine is set on the coordinator after startup so the replication server can use it.

connection.rsINFO replication section with role, connected_replicas (primary) or master_host/port/link_status (replica).

what was tested

  • cargo test --workspace: all unit tests pass. cluster integration tests show the same pre-existing failures as main (12, down from 14 — two tests now pass that didn't before)
  • cargo clippy --workspace -- -D warnings: clean
  • cargo build --workspace: clean

design considerations

the broadcast channel is the bridge between shard mutations and the replication stream. each shard publishes ReplicationEvent { shard_id, offset, record } after every successful write. the server subscribes a new receiver per replica connection, so each replica gets an independent stream starting from the current position. snapshots are serialized in-memory using io::Cursor<Vec<u8>> — avoids disk I/O for full-sync, which matters on fast reconnects.

replica routing uses the same FNV-1a key-hash as the primary (via engine.route(key, request)), so records always land on the correct shard without a separate shard-ID lookup.

kacy added 3 commits February 18, 2026 19:48
…hard

- make AofRecord::to_bytes/from_bytes pub for wire protocol use
- add write_snapshot_bytes/read_snapshot_from_bytes to snapshot.rs for
  in-memory snapshot serialization without filesystem I/O
- add ReplicationEvent struct: shard_id, offset, record
- add ShardRequest::SerializeSnapshot and ShardResponse::SnapshotData
- thread optional broadcast::Sender<ReplicationEvent> through spawn_shard,
  run_shard, and process_message; publishes after each mutation
- add Engine::subscribe_replication() to create broadcast receivers
- thread replication_tx through EngineConfig for wiring in main.rs
adds a dedicated TCP replication channel between primary and replica nodes:

- replication.rs: new module with ReplicationServer and ReplicationClient
  - primary side: per-shard snapshot handshake + incremental AofRecord stream
  - replica side: snapshot load + record replay with exponential backoff reconnect
  - wire protocol: little-endian framing with version, shard sync, and resync msgs
  - aof_record_to_shard_request: maps all 20+ record types to ShardRequest

- cluster.rs: set_engine, start_replication_server, start_replication_client
  - replication port = data_port + gossip_offset + 2 (default: 16381)
  - cluster_replicate now starts the client after updating topology
  - replication_info for INFO replication output

- server.rs: wire engine into coordinator + start replication server at startup
- main.rs: create broadcast channel, inject into EngineConfig when cluster enabled
- config.rs: include replication_tx in EngineConfig initializer
- connection.rs: INFO replication section (role, replicas, master addr)
@kacy
kacy merged commit 10b7cb2 into main Feb 19, 2026
4 of 7 checks passed
@kacy
kacy deleted the feat/replication-stream branch February 19, 2026 01:24
kacy added a commit that referenced this pull request Feb 19, 2026
* feat(core): add ReplicationEvent broadcast and SerializeSnapshot to shard

- make AofRecord::to_bytes/from_bytes pub for wire protocol use
- add write_snapshot_bytes/read_snapshot_from_bytes to snapshot.rs for
  in-memory snapshot serialization without filesystem I/O
- add ReplicationEvent struct: shard_id, offset, record
- add ShardRequest::SerializeSnapshot and ShardResponse::SnapshotData
- thread optional broadcast::Sender<ReplicationEvent> through spawn_shard,
  run_shard, and process_message; publishes after each mutation
- add Engine::subscribe_replication() to create broadcast receivers
- thread replication_tx through EngineConfig for wiring in main.rs

* fix(core): update shard test calls for replication_tx param

* feat(server): implement replication stream (primary → replica sync)

adds a dedicated TCP replication channel between primary and replica nodes:

- replication.rs: new module with ReplicationServer and ReplicationClient
  - primary side: per-shard snapshot handshake + incremental AofRecord stream
  - replica side: snapshot load + record replay with exponential backoff reconnect
  - wire protocol: little-endian framing with version, shard sync, and resync msgs
  - aof_record_to_shard_request: maps all 20+ record types to ShardRequest

- cluster.rs: set_engine, start_replication_server, start_replication_client
  - replication port = data_port + gossip_offset + 2 (default: 16381)
  - cluster_replicate now starts the client after updating topology
  - replication_info for INFO replication output

- server.rs: wire engine into coordinator + start replication server at startup
- main.rs: create broadcast channel, inject into EngineConfig when cluster enabled
- config.rs: include replication_tx in EngineConfig initializer
- connection.rs: INFO replication section (role, replicas, master addr)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant