Skip to content

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#50

Draft
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-f941
Draft

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#50
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-f941

Conversation

@cursor

@cursor cursor Bot commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Bug and impact

restore_from_file() and gossip.receive() did not preserve or apply Hybrid Logical Clock (HLC) timestamps introduced in v0.6.0. After a snapshot restore, nodes were assigned fresh HLCs instead of the stored values. Legitimate causal successor writes (e.g. from the original writer or a peer node) were rejected as TimestampRegression, causing silent data loss in disaster-recovery and multi-node gossip scenarios.

Concrete trigger:

  1. Node writes remember("k", "v1", hlc=(5000, 10, "nodeA"))
  2. Snapshot taken and restored on a fresh brain
  3. Restored node gets a fresh HLC (wall, 0, uuid) instead of (5000, 10, "nodeA")
  4. Causal successor remember("k", "v2", hlc=(5000, 11, "nodeA")) is rejected

Root cause

restore_from_file() constructed MemoryNode without passing the snapshot's hlc field, so the dataclass default factory generated a new HLC. gossip.receive() similarly omitted HLC passthrough. bulk_write() also did not forward HLC from row data.

Fix

  • Add _parse_hlc() for snapshot/wire HLC normalisation (with legacy fallback from ts_ns)
  • Restore HLC on snapshot load and advance global _hlc via _hlc.update()
  • Pass HLC through bulk_write and gossip.receive()
  • Reject stale gossip updates; skip HLC-less updates on existing keys
  • Hold lock during restore to prevent concurrent mutation

Validation

  • Reproduced bug before fix (successor write rejected after restore)
  • Added regression tests in test_enterprise_backup.py and test_gossip.py
  • Full suite: 192 passed, 8 skipped
Open in Web View Automation 

After snapshot restore, nodes were assigned fresh HLC timestamps instead
of the stored values from the snapshot. Post-restore causal writes with
legitimate successor HLCs were rejected as TimestampRegression, causing
silent data loss in disaster-recovery and multi-node gossip scenarios.

- Add _parse_hlc() for snapshot/wire HLC normalisation
- Restore HLC on snapshot load and advance global _hlc
- Pass HLC through bulk_write and gossip receive
- Reject stale gossip updates; skip HLC-less updates on existing keys
- Add regression tests for restore round-trip and gossip ordering

Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant