fix: keep lastLog cache in sync after append truncation failures#4
Draft
anyasabo wants to merge 1 commit into
Draft
fix: keep lastLog cache in sync after append truncation failures#4anyasabo wants to merge 1 commit into
anyasabo wants to merge 1 commit into
Conversation
appendEntries can truncate a conflicting suffix, fail to store replacement entries, and leave cached lastLog metadata stale versus durable storage. Recompute lastLog from the log store on StoreLogs failure and add a regression test that reproduces the stale-cache path. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
appendEntriescan leave cachedlastLogmetadata stale when this sequence occurs:StoreLogserror),lastLogfrom durable store.This creates cache/store divergence (
r.getLastLog()vsLogStore).How this can happen in concrete terms
A realistic failure chain in production:
DeleteRangesucceeds (durable suffix removed),lastLogindex/term from memory cache.Existing mitigations and gaps
Mitigations that reduce probability:
Gap prior to this PR: even when storage returned an explicit append failure, cache was not reconciled.
Impact
Consistency of local bookkeeping degrades under storage fault:
How we would notice in production
failed to append to logsshortly after conflict truncation.last_log_*stats/debug output.Provenance / Preconditions
lastLogrisk in this path.fdb8145f, 2016), with logging additions later.StoreLogsfailure.What this PR changes
lastLogfrom durable store onStoreLogsfailure.Reviewer reproduction (live in-process Raft node)
Reproduce pre-fix behavior
git checkout 0d1b34c^git checkout fix/append-truncation-store-failure-lastlog -- raft_test.gogo test -run "TestRaft_AppendEntriesStoreLogsFailureRefreshesLastLogAfterTruncate" -count=1 .lastLog(cache index > durable store index).Verify fixed behavior
fix/append-truncation-store-failure-lastlog).go test -run "TestRaft_AppendEntriesStoreLogsFailureRefreshesLastLogAfterTruncate" -count=1 .Test plan
go test -run "TestRaft_AppendEntriesStoreLogsFailureRefreshesLastLogAfterTruncate" -count=1 .go test -run "TestRaft_AppendEntry$|TestRaft_AppendEntriesStoreLogsFailureRefreshesLastLogAfterTruncate" -count=1 .