Skip to content

fix(change): survive LogPos wraparound and hard-fail on unknown rows-event subtypes#1039

Draft
morgo wants to merge 2 commits into
block:mainfrom
morgo:fix/binlog-event-robustness
Draft

fix(change): survive LogPos wraparound and hard-fail on unknown rows-event subtypes#1039
morgo wants to merge 2 commits into
block:mainfrom
morgo:fix/binlog-event-robustness

Conversation

@morgo

@morgo morgo commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Two robustness fixes in pkg/change binlog event handling, both in the "silent data loss" class: events that arrive on the stream but never make it into the buffer.

Bug A: uint32 LogPos wraparound makes the replay-skip guard discard live row events

Problem. ev.Header.LogPos is a 4-byte wire field, and a binlog file only rotates on a transaction boundary. One transaction writing more than 4GiB of row images past the max_binlog_size point pushes end positions past 2^32, where they wrap to small values. setBufferedPos's monotonic guard then freezes bufferedPos just under the wrap point, and every subsequent RowsEvent in that file compares <= bufferedPosshouldSkipReplayedEvent classified them all as post-reconnect replays and silently discarded them. The lost changes were only caught by the checksum, where enabled.

Fix. The skip decision (shouldSkipRowsEvent) is now wraparound-aware and fails safe by re-applying instead of skipping: outside a post-recreateStreamer replay window, a same-file event more than 2^31 below bufferedPos is classified as wraparound, logged with a warning once per file, and delivered. Re-delivery is idempotent (REPLACE/DELETE application); skipping is not.

Design. Considered tracking a 64-bit logical position (file + wrap count) instead, but rejected it as the more invasive option for no additional safety in practice: mysql.Position is 32-bit throughout (checkpoint serialization, StartSync, flushed/buffered plumbing), so a 64-bit shadow position would still need a heuristic to detect each wrap increment, and could not be expressed in a resume position anyway. The chosen design is local to the skip decision plus one replaying flag in readStream (set on successful streamer recreation, cleared when the first rows event is delivered again), so normal reconnect-replay deduplication is unchanged. The half-space threshold (2^31) cleanly separates the two cases: right after a wrap the gap is nearly 2^32, while a replay's gap only spans positions actually reached within the file. Residual limits, documented in code: a reconnect while bufferedPos is already frozen at a pre-wrap value is genuinely ambiguous with 32-bit positions (keeps the old skip behavior until rotation), and bufferedPos/flushedPos still cannot advance past a wrap until the next rotation — the fix targets not discarding live events, not making >4GiB positions checkpointable.

Testing.

  • TestShouldSkipRowsEventWraparound — pure-function matrix: wrap gap delivered + flagged, exact-2^31 boundary still skipped, earlier-file gap not misclassified, replay window still skips (including far-below positions), replay catch-up delivers.
  • TestReadStreamDeliversWrappedLogPosEvents — drives the real readStream loop through an injected replication.BinlogStreamer with bufferedPos frozen just under 4GiB: post-wrap events are buffered (verified red pre-fix: they were discarded and the test times out), a slightly-below stale event is still skipped, the warning fires exactly once per file, and bufferedPos stays frozen.
  • TestRecreateStreamerSkipsFlushedReplay (existing E2E for the real recreate path) and the existing TestShouldSkipReplayedEvent* tests pass unchanged.

Bug B: unknown rows-event subtypes were dropped with only an error log

Problem. parseEventType doesn't recognize e.g. PARTIAL_UPDATE_ROWS_EVENT, which go-mysql delivers as a regular *replication.RowsEvent. It reached processRowsEvent (both the binlog and GTID clients), fell into the default: branch, and its rows were skipped with only logger.Error("unknown event type") — no fatalError(). That is asymmetric with the minimal-row-image case, which hard-fails. Preflight requires binlog_row_value_options to be empty, but the global can change after preflight, at which point JSON partial updates stream and are silently not applied.

Fix. An unrecognized rows-event subtype for a subscribed table is now a hard error from processRowsEvent in both clients, naming the event type (string form and header value, e.g. PartialUpdateRowsEvent (0x27)) and the table. It flows through the existing fatal path exactly like the minimal-row-image case. The insert/delete loop's default: branch is also converted to a hard error so a future eventType addition cannot reintroduce a silent drop. Events for unsubscribed tables (e.g. _new tables) are still ignored, as before.

Testing.

  • TestProcessRowsEventUnknownSubtypeFails (binlog client) and TestGTIDProcessRowsEventUnknownSubtypeFails (GTID client) — synthetic PARTIAL_UPDATE_ROWS_EVENT through processRowsEvent returns an error containing the type name, header value, and table, and buffers nothing (pre-fix: returned nil and dropped the rows); a recognized subtype still buffers; unsubscribed tables stay ignored.

Verification

go build, gofmt, go vet, golangci-lint run ./pkg/change/ (0 issues). Targeted suite against MySQL 8.0.45: TestReplClient*, TestRecreateStreamer*, TestShouldSkip*, TestReadStream*, TestProcessRows*, TestGTID*, TestSetBufferedPos*, TestFlushedPos*, TestAllChangesFlushed, TestBlockWait*, TestFlushUnderTableLock* — all pass; new tests also pass under -race.

Note: open PRs #1033 (processRowsEvent) and #1025 (buildSyncerConfig) touch pkg/change in nearby regions; minor conflicts possible whichever lands second.

🤖 Generated with Claude Code

morgo and others added 2 commits July 2, 2026 06:44
ev.Header.LogPos is a 4-byte wire field, and a binlog file only rotates
on a transaction boundary: one transaction writing more than 4GiB of row
images past the max_binlog_size point pushes end positions past 2^32,
where they wrap to small values. setBufferedPos's monotonic guard then
freezes bufferedPos just under the wrap point, and every subsequent
RowsEvent in the file compared <= bufferedPos — silently discarded as a
post-reconnect replay. Lost changes were only caught by the checksum
(where enabled).

Make the skip decision wraparound-aware and fail safe (re-apply instead
of skip): outside a post-recreateStreamer replay window, a same-file
event more than 2^31 below bufferedPos is classified as wraparound,
logged once per file, and delivered. Re-delivery is idempotent
(REPLACE/DELETE); skipping is not. The replay window is a local flag in
readStream, set on a successful streamer recreation and cleared when the
first rows event is delivered again, so normal reconnect-replay
deduplication is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ping rows

parseEventType does not recognize e.g. PARTIAL_UPDATE_ROWS_EVENT, which
go-mysql delivers as a regular *replication.RowsEvent. It reached
processRowsEvent (both the binlog and GTID clients), fell into the
default branch, and its rows were skipped with only
logger.Error("unknown event type") — silent data loss, asymmetric with
the minimal-row-image case which hard-fails. Preflight requires
binlog_row_value_options to be empty, but the global can change after
preflight, at which point JSON partial updates stream and are silently
not applied.

Return an error naming the event type (string form and header value)
and the table from processRowsEvent in both clients, so it flows
through the existing fatalError path exactly like the minimal-row-image
case. The insert/delete loop's default branch becomes a hard error too,
so a future eventType addition cannot reintroduce a silent drop.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant