Skip to content

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#48

Draft
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-777c
Draft

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#48
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-777c

Conversation

@cursor

@cursor cursor Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Bug and impact

After snapshot restore, restore_from_file() assigned fresh HLC timestamps instead of the values stored in the snapshot. Post-restore causal writes with legitimate successor HLCs were rejected as TimestampRegression, causing silent data loss in disaster-recovery and multi-node gossip scenarios.

Concrete trigger: Node writes k=v1 with hlc=(5000, 10, nodeA), snapshots, crashes, restores. A causal successor write k=v2 with hlc=(5000, 11, nodeA) is rejected because restore assigned a fresh wall-clock HLC far in the future.

Gossip had the same class of bug: receive() ignored HLC from peer events, allowing stale overwrites or rejecting valid updates.

Root cause

restore_from_file() created MemoryNode without passing hlc, so the default factory assigned _hlc.now(). The global _hlc was never advanced from restored nodes. bulk_write() and gossip.receive() also omitted HLC passthrough.

Fix

  • Add _parse_hlc() for snapshot/wire HLC normalisation (with legacy fallback for pre-v0.6.0 snapshots)
  • Restore HLC on snapshot load and advance global _hlc
  • Pass HLC through bulk_write and gossip.receive()
  • Reject stale gossip updates; skip HLC-less updates on existing keys
  • Hold lock during restore_from_file and gc_expired

Validation

  • Reproduced pre-fix: successor write rejected after restore
  • New regression tests: test_restore_preserves_hlc_for_replication, gossip HLC ordering tests
  • Full suite: 192 passed, 8 skipped
Open in Web View Automation 

After snapshot restore, nodes were assigned fresh HLC timestamps instead
of the stored values from the snapshot. Post-restore causal writes with
legitimate successor HLCs were rejected as TimestampRegression, causing
silent data loss in disaster-recovery and multi-node gossip scenarios.

- Add _parse_hlc() for snapshot/wire HLC normalisation
- Restore HLC on snapshot load and advance global _hlc
- Pass HLC through bulk_write and gossip receive
- Reject stale gossip updates; skip HLC-less updates on existing keys
- Add regression tests for restore round-trip and gossip ordering

Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant