Skip to content

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#51

Draft
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-211a
Draft

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#51
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-211a

Conversation

@cursor

@cursor cursor Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Bug and impact

After snapshot restore, restore_from_file() assigned fresh HLC timestamps to every node instead of restoring the stored values from the snapshot. This caused silent data loss in disaster-recovery and multi-node gossip scenarios:

  1. Post-restore writes rejected: Legitimate causal successor writes (e.g. hlc=(5000, 11, 'nodeA') after a node stored at (5000, 10, 'nodeA')) raised TimestampRegression because restored nodes got a fresh wall-clock HLC.
  2. Gossip replay dropped: After a node restarted from snapshot, gossip events carrying pre-crash HLC timestamps were silently rejected, losing replicated memory updates.

Root cause

restore_from_file() created MemoryNode without passing the hlc field from the snapshot JSON. gossip.receive() also omitted HLC when calling remember(), and bulk_write() did not pass HLC through.

Fix

  • Add _parse_hlc() to normalise HLC from snapshot/wire form (list or tuple)
  • Restore HLC on snapshot load and advance the global _hlc clock
  • Pass HLC through bulk_write and gossip.receive
  • Reject stale gossip updates; skip HLC-less updates on existing keys
  • Hold _lock during restore to prevent concurrent mutation

Validation

  • Reproduced: restore assigned fresh HLC instead of snapshot (5000, 10, 'nodeA'); causal successor write raised TimestampRegression
  • New regression tests: test_restore_preserves_hlc_for_replication, gossip HLC ordering tests
  • Full suite: 192 passed, 8 skipped
Open in Web View Automation 

After snapshot restore, nodes were assigned fresh HLC timestamps instead
of the stored values from the snapshot. Post-restore causal writes with
legitimate successor HLCs were rejected as TimestampRegression, causing
silent data loss in disaster-recovery and multi-node gossip scenarios.

- Add _parse_hlc() for snapshot/wire HLC normalisation
- Restore HLC on snapshot load and advance global _hlc
- Pass HLC through bulk_write and gossip receive
- Reject stale gossip updates; skip HLC-less updates on existing keys
- Add regression tests for restore round-trip and gossip ordering

Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant