Skip to content

vfs: fix Windows WAL corruption by mapping the wal-index into wasm memory#405

Open
franchb wants to merge 2 commits into
ncruces:mainfrom
franchb:windows-wal-shm-mmap
Open

vfs: fix Windows WAL corruption by mapping the wal-index into wasm memory#405
franchb wants to merge 2 commits into
ncruces:mainfrom
franchb:windows-wal-shm-mmap

Conversation

@franchb

@franchb franchb commented Jul 5, 2026

Copy link
Copy Markdown

Fixes #404.

Problem

On Windows (and in the sqlite3_dotlk build), concurrent WAL writes across pooled connections — a single *sql.DB with SetMaxOpenConns(N>1) and ordinary BEGIN DEFERRED writers — corrupt the database, and produce SQLITE_PROTOCOL / database is locked error storms under load.

The root cause is the WAL-index copy scheme (vfs/shm_copy.go). The WAL-index is kept in a per-connection private copy and synchronized with the shared -shm file only at xShmLock boundaries. But wal.c reads and writes the WAL-index between lock calls and relies on seeing other connections' updates live:

  • readers re-read the two header copies around a barrier and retry on a dirty read (walIndexTryHdr);
  • walTryBeginRead re-checks aReadMark[] and the header after taking a read lock and retries on any change;
  • the checkpointer, when its exclusive probe of a reader slot loses to a live reader, caps mxSafeFrame on the aReadMark value it just read (walCheckpoint).

Copy-at-lock-boundary is too coarse for those patterns, so a connection acts on a stale view. Under concurrent writers this lets a checkpointer backfill stale page versions into the database and truncate frames that were never backfilled — corrupting the file. I confirmed this with word-level instrumentation of the private/shared/shadow copies and a content trace of the data plane (the checkpointer demonstrably writes older committed page versions to the main DB).

This is Windows-only among the VFS shapes (the unix build maps -shm into wasm memory directly; native C shares one mapping), and it reproduces on every released version back to when the Windows shared memory was introduced.

Fix

Stop copying. On Windows 10 1803+ / Server 2019+, map the -shm regions directly into the wasm linear memory using address-space placeholders (VirtualAlloc2(MEM_RESERVE_PLACEHOLDERS) + MapViewOfFile3(MEM_REPLACE_PLACEHOLDER)) — the analogue of the MAP_FIXED mapping the unix build already uses. SQLite then works on genuinely shared, always-current memory; xShmBarrier becomes a plain fence, and the private/shadow copies and their synchronization points disappear. This removes the entire staleness class by construction — both the corruption and the SQLITE_PROTOCOL storms.

The wasm heap is committed at 64K (allocation-granularity) so a 64K-aligned WAL-index view never straddles two allocations, which a placeholder split cannot cross.

On older Windows the copy scheme remains as a fallback, now with process-wide copy serialization and exact per-page comparison (instead of the header-equality fast path, which is unsound: SQLite changes hash-table words while leaving the 136-byte header byte-identical, e.g. walCleanupHash after a rollback). This makes the fallback less bad, though it cannot be made fully correct — see the caveat below.

What changed

  • internal/sqlite3_wrap/placeholder_windows.go (new): thin bindings for VirtualAlloc2 / MapViewOfFile3 / UnmapViewOfFile2 and a capability probe.
  • internal/sqlite3_wrap/mem_windows.go: reserve the wasm memory with placeholders; MapFileRegion / UnmapFileRegion map file views into carved holes.
  • vfs/shm_windows.go: dual path — direct-map on 1803+, copy scheme as fallback.
  • vfs/shm_copy.go, vfs/shm_dotlk.go: copy-scheme hardening for the fallback path.
  • vfs/tests/wal_stress_test.go (new): a heavy concurrent-WAL-writers regression test (short-mode skipped; excluded from sqlite3_dotlk).

Validation

  • The standalone reproducer from Windows: concurrent WAL writes across pooled connections corrupt the database #404 is clean 10/10 on Windows Server 2022 (stock corrupts on ~every run); at 2× concurrency it degrades only to a genuine busy-timeout with the database intact.
  • The regression test passes on Windows and Linux.
  • Cross-process interop: the fixed writer and a native sqlite3.exe (stock 3.53.2) reading the same WAL database concurrently — PRAGMA integrity_check = ok, foreign_key_check clean, 128000 rows consistent, no read errors. Lock offsets and the on-disk format are unchanged, so native tools stay compatible.
  • -race is clean with the fix.

Caveats / open questions

  • Full correctness requires Windows 10 1803+ / Server 2019+ (for the placeholder APIs). Older Windows uses the improved-but-imperfect copy fallback; I kept it because I did not want to drop support, but I'd understand if you'd rather gate WAL differently there.
  • The direct-map path serializes WAL-index copies only within a process (there are none — it's genuinely shared). But if you consider the copy scheme fundamentally, its cross-process Go↔Go story has the same class of issue; the direct map avoids it entirely on supported Windows.
  • Happy to adjust the structure (e.g. how the Memory placeholder logic is factored, or the capability gate) to your taste.

@ncruces

ncruces commented Jul 6, 2026

Copy link
Copy Markdown
Owner

I've made this a bit harder to merge, because I committed my non fixes (because they cleaned up some deprecations, mostly). Sorry about that.

If it's not too much trouble to fix the conflict, as you drop the old Windows fallback, I'd appreciate it.

Otherwise, if it's too much of a bother, I can always merge this to a branch and take over.

franchb added 2 commits July 7, 2026 01:46
…mory

Fixes ncruces#404.

On Windows the WAL-index was kept in a per-connection private copy and
synchronized with the shared -shm file only at xShmLock boundaries. But
wal.c reads and writes the WAL-index *between* lock calls and relies on
seeing other connections' updates live: dirty-header re-reads, aReadMark
probes when a checkpointer's exclusive slot probe loses to a live reader,
and post-lock header re-checks. The copy-on-lock-boundary scheme served
stale views on those paths, so under concurrent WAL writes a checkpointer
could backfill stale page versions into the database and truncate frames
that were never backfilled, corrupting the file (and producing
SQLITE_PROTOCOL error storms).

On Windows 10 1803+ / Server 2019+, map the -shm regions directly into
the wasm linear memory using address-space placeholders (VirtualAlloc2 +
MapViewOfFile3), the analogue of the MAP_FIXED mapping the unix build
already uses. SQLite then works on genuinely shared, always-current
memory; xShmBarrier becomes a plain fence and the private/shadow copies
and their sync points disappear.

The heap is committed at 64K granularity so a wal-index view (64K
aligned, 64K) never straddles two allocations, which a placeholder split
cannot cross.

Below Windows 10 1803 the placeholder APIs are unavailable. Rather than
keep a copy-on-lock-boundary fallback that cannot make wal.c's
between-lock reads coherent, shmMap refuses shared memory there
(_IOERR_SHMMAP): such a build must use WAL with EXCLUSIVE locking mode,
like platforms without shared-memory support. This drops the old Windows
copy path entirely and the now-unused mmap_windows.go.

The copy scheme (shm_copy.go) is retained for the sqlite3_dotlk build,
which has no file mapping and genuinely needs it; it keeps the
per-file copy serialization and exact per-page comparison that make it
correct within a single process.
A repeat-until-budget stress test that drives many concurrent WAL writers
through one *sql.DB, aggressively checkpoints (TRUNCATE), then cold-reopens
the file and runs PRAGMA integrity_check. It repeats that cycle until it
either observes the defect or a time budget elapses.

On the pre-fix copy-on-lock-boundary Windows scheme this reliably corrupts
the database within the first round (database disk image malformed / file
is not a database / SQLITE_PROTOCOL). On the fix — and on every platform
whose VFS maps the -shm directly (unix, native) — every round stays clean.
The test therefore goes red only on an unfixed Windows build and green
everywhere else, so it can be watched to flip red→green.

It is opt-in: several minutes of heavy I/O, gated behind
SQLITE3_TEST_WAL_STRESS so it never taxes the normal test matrix. The
wal-repro workflow (workflow_dispatch) flips the gate and runs it on
Windows and Linux on demand.

BUSY, IOERR and FULL are tolerated during the storm: heavy concurrency and
a full or slow scratch filesystem are environment limits, not the defect,
and the defect surfaces only as SQLITE_PROTOCOL / CORRUPT / NOTADB. The
cold-reopen integrity_check is the definitive corruption gate regardless.

The workload (64 writers, 2000 iters, 8K rows, checkpoint every 25) is
sized to open the race window even on the slow temp disk of hosted CI
runners; lighter workloads under-expose it and can false-green.
@franchb franchb force-pushed the windows-wal-shm-mmap branch from 43a7fcd to 8645d97 Compare July 7, 2026 06:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Windows: concurrent WAL writes across pooled connections corrupt the database

2 participants