fix(change): survive LogPos wraparound and hard-fail on unknown rows-event subtypes#1039
Draft
morgo wants to merge 2 commits into
Draft
fix(change): survive LogPos wraparound and hard-fail on unknown rows-event subtypes#1039morgo wants to merge 2 commits into
morgo wants to merge 2 commits into
Conversation
ev.Header.LogPos is a 4-byte wire field, and a binlog file only rotates on a transaction boundary: one transaction writing more than 4GiB of row images past the max_binlog_size point pushes end positions past 2^32, where they wrap to small values. setBufferedPos's monotonic guard then freezes bufferedPos just under the wrap point, and every subsequent RowsEvent in the file compared <= bufferedPos — silently discarded as a post-reconnect replay. Lost changes were only caught by the checksum (where enabled). Make the skip decision wraparound-aware and fail safe (re-apply instead of skip): outside a post-recreateStreamer replay window, a same-file event more than 2^31 below bufferedPos is classified as wraparound, logged once per file, and delivered. Re-delivery is idempotent (REPLACE/DELETE); skipping is not. The replay window is a local flag in readStream, set on a successful streamer recreation and cleared when the first rows event is delivered again, so normal reconnect-replay deduplication is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ping rows
parseEventType does not recognize e.g. PARTIAL_UPDATE_ROWS_EVENT, which
go-mysql delivers as a regular *replication.RowsEvent. It reached
processRowsEvent (both the binlog and GTID clients), fell into the
default branch, and its rows were skipped with only
logger.Error("unknown event type") — silent data loss, asymmetric with
the minimal-row-image case which hard-fails. Preflight requires
binlog_row_value_options to be empty, but the global can change after
preflight, at which point JSON partial updates stream and are silently
not applied.
Return an error naming the event type (string form and header value)
and the table from processRowsEvent in both clients, so it flows
through the existing fatalError path exactly like the minimal-row-image
case. The insert/delete loop's default branch becomes a hard error too,
so a future eventType addition cannot reintroduce a silent drop.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two robustness fixes in
pkg/changebinlog event handling, both in the "silent data loss" class: events that arrive on the stream but never make it into the buffer.Bug A: uint32 LogPos wraparound makes the replay-skip guard discard live row events
Problem.
ev.Header.LogPosis a 4-byte wire field, and a binlog file only rotates on a transaction boundary. One transaction writing more than 4GiB of row images past themax_binlog_sizepoint pushes end positions past 2^32, where they wrap to small values.setBufferedPos's monotonic guard then freezesbufferedPosjust under the wrap point, and every subsequentRowsEventin that file compares<= bufferedPos—shouldSkipReplayedEventclassified them all as post-reconnect replays and silently discarded them. The lost changes were only caught by the checksum, where enabled.Fix. The skip decision (
shouldSkipRowsEvent) is now wraparound-aware and fails safe by re-applying instead of skipping: outside a post-recreateStreamerreplay window, a same-file event more than 2^31 belowbufferedPosis classified as wraparound, logged with a warning once per file, and delivered. Re-delivery is idempotent (REPLACE/DELETEapplication); skipping is not.Design. Considered tracking a 64-bit logical position (file + wrap count) instead, but rejected it as the more invasive option for no additional safety in practice:
mysql.Positionis 32-bit throughout (checkpoint serialization,StartSync, flushed/buffered plumbing), so a 64-bit shadow position would still need a heuristic to detect each wrap increment, and could not be expressed in a resume position anyway. The chosen design is local to the skip decision plus onereplayingflag inreadStream(set on successful streamer recreation, cleared when the first rows event is delivered again), so normal reconnect-replay deduplication is unchanged. The half-space threshold (2^31) cleanly separates the two cases: right after a wrap the gap is nearly 2^32, while a replay's gap only spans positions actually reached within the file. Residual limits, documented in code: a reconnect whilebufferedPosis already frozen at a pre-wrap value is genuinely ambiguous with 32-bit positions (keeps the old skip behavior until rotation), andbufferedPos/flushedPosstill cannot advance past a wrap until the next rotation — the fix targets not discarding live events, not making >4GiB positions checkpointable.Testing.
TestShouldSkipRowsEventWraparound— pure-function matrix: wrap gap delivered + flagged, exact-2^31 boundary still skipped, earlier-file gap not misclassified, replay window still skips (including far-below positions), replay catch-up delivers.TestReadStreamDeliversWrappedLogPosEvents— drives the realreadStreamloop through an injectedreplication.BinlogStreamerwithbufferedPosfrozen just under 4GiB: post-wrap events are buffered (verified red pre-fix: they were discarded and the test times out), a slightly-below stale event is still skipped, the warning fires exactly once per file, andbufferedPosstays frozen.TestRecreateStreamerSkipsFlushedReplay(existing E2E for the real recreate path) and the existingTestShouldSkipReplayedEvent*tests pass unchanged.Bug B: unknown rows-event subtypes were dropped with only an error log
Problem.
parseEventTypedoesn't recognize e.g.PARTIAL_UPDATE_ROWS_EVENT, which go-mysql delivers as a regular*replication.RowsEvent. It reachedprocessRowsEvent(both the binlog and GTID clients), fell into thedefault:branch, and its rows were skipped with onlylogger.Error("unknown event type")— nofatalError(). That is asymmetric with the minimal-row-image case, which hard-fails. Preflight requiresbinlog_row_value_optionsto be empty, but the global can change after preflight, at which point JSON partial updates stream and are silently not applied.Fix. An unrecognized rows-event subtype for a subscribed table is now a hard error from
processRowsEventin both clients, naming the event type (string form and header value, e.g.PartialUpdateRowsEvent (0x27)) and the table. It flows through the existing fatal path exactly like the minimal-row-image case. The insert/delete loop'sdefault:branch is also converted to a hard error so a futureeventTypeaddition cannot reintroduce a silent drop. Events for unsubscribed tables (e.g._newtables) are still ignored, as before.Testing.
TestProcessRowsEventUnknownSubtypeFails(binlog client) andTestGTIDProcessRowsEventUnknownSubtypeFails(GTID client) — syntheticPARTIAL_UPDATE_ROWS_EVENTthroughprocessRowsEventreturns an error containing the type name, header value, and table, and buffers nothing (pre-fix: returned nil and dropped the rows); a recognized subtype still buffers; unsubscribed tables stay ignored.Verification
go build,gofmt,go vet,golangci-lint run ./pkg/change/(0 issues). Targeted suite against MySQL 8.0.45:TestReplClient*,TestRecreateStreamer*,TestShouldSkip*,TestReadStream*,TestProcessRows*,TestGTID*,TestSetBufferedPos*,TestFlushedPos*,TestAllChangesFlushed,TestBlockWait*,TestFlushUnderTableLock*— all pass; new tests also pass under-race.Note: open PRs #1033 (
processRowsEvent) and #1025 (buildSyncerConfig) touchpkg/changein nearby regions; minor conflicts possible whichever lands second.🤖 Generated with Claude Code