Skip to content

fix(ingest): consistent, verified snapshots under live writers (FIX-02)#38

Merged
ChelseaKR merged 2 commits into
mainfrom
fix/wal-consistent-snapshots
Jul 12, 2026
Merged

fix(ingest): consistent, verified snapshots under live writers (FIX-02)#38
ChelseaKR merged 2 commits into
mainfrom
fix/wal-consistent-snapshots

Conversation

@ChelseaKR

Copy link
Copy Markdown
Owner

What

Closes the WAL-torn-snapshot correctness hole (ideation FIX-02, roadmap BF-2). This is the finished work that sat uncommitted in the checkout since 2026-07-05, rescued via rescue/uncommitted-2026-07-09 (content byte-identical to rescue/uncommitted-2026-07-06) and rebuilt clean off current main.

  • Bug closed: a plain byte-copy of a mid-write metadata.db/statistics.sqlite omits committed rows still in the -wal sidecar; immutable=1 then suppresses SQLite recovery, so readers silently saw stale/inconsistent state. It could never corrupt sources (that guarantee already held) but could silently mis-ingest.
  • Fix: when -wal/-shm/-journal sidecars are present, ingest.snapshot.snapshot() copies via SQLite's online backup API from a strictly read-only (mode=ro) source handle — one transactionally-consistent snapshot, WAL folded in, written as a standalone sidecar-free file. Every snapshot (both paths) is verified with PRAGMA integrity_check; a failed plain copy escalates to the backup path once, then fails loudly with the new SnapshotIntegrityError instead of handing a reader a bad copy.
  • Read-only invariant preserved: the source is never opened writable; the new guardrail test snapshots a live WAL-mode DB with an open writer and asserts the source main file's SHA-256 is unchanged.
  • docs/audits/coverage.xml regenerated in the same PR (resolves the deliberate leftover noted in the roadmap).

Tests

4 new tests in tests/test_snapshot_readonly.py — including one that first demonstrates the torn-copy bug (naive copy of a WAL-mode DB drops committed rows) before proving the fixed path returns all rows with integrity_check = ok and no sidecars.

Verification (CI is billing-blocked — local gate run is the review gate)

make verify green end-to-end on this branch: ruff format+check + marker-hygiene, mypy --strict, 203 tests passed, 96.77% branch coverage (gate ≥85%), pip-audit + gitleaks (0 leaks) + osv-scanner (0 vulns vs uv.lock), static a11y checker + pa11y/axe (0 violations), offline eval (content beats popularity). GitHub Actions cannot run on this private repo until the account billing issue is resolved (⛔USER-1); per the CI-dead protocol, this local run is the merge evidence.

Follow-up (not this PR)

Before trusting as a daily driver, validate once against a live, busy Calibre library (roadmap §7 item 8, real-data gate).

🤖 Generated with Claude Code

A plain byte-copy of a mid-write metadata.db/statistics.sqlite is torn:
committed rows still living in the -wal sidecar are silently dropped, and
immutable=1 then suppresses SQLite's recovery, so the reader sees stale
state with no error. Now:

- when -wal/-shm/-journal sidecars are present, the snapshot is taken via
  SQLite's online backup API from a strictly read-only (mode=ro) source
  handle — one transactionally-consistent snapshot, WAL folded in, written
  as a standalone sidecar-free file;
- every snapshot (both paths) is verified with PRAGMA integrity_check
  before a reader sees it; a failed plain copy escalates to the backup
  path once, then fails loudly with the new SnapshotIntegrityError;
- the source is never opened writable anywhere (guardrail test asserts
  the source main file's hash is unchanged after snapshotting a live
  WAL-mode DB).

Tests: 4 new (torn-copy bug demonstrated + consistency, source-unmutated
hash check, missing source, corrupt-source fail-loud). 203 total pass,
96.77% branch coverage; docs/audits/coverage.xml regenerated in-sync.

Verified locally (CI is billing-blocked): make verify green end-to-end
(ruff, mypy --strict, pytest, pip-audit + gitleaks + osv-scanner, static
a11y + pa11y/axe, offline eval).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ChelseaKR

Copy link
Copy Markdown
Owner Author

Review verdict — 2026-07-09: READY TO MERGE (verified locally: make verify green — ruff+marker-hygiene, mypy --strict, 203 tests @ 96.77% cov, pip-audit/gitleaks/osv-scanner clean, static a11y + pa11y 0 violations, eval green; CI billing-blocked, local run is the review gate). Adversarial self-review notes: backup path opens source strictly mode=ro (never writable); hot-journal sources a read-only handle cannot recover raise and land in the fail-loud SnapshotIntegrityError path (correct: no silent bad read); source-hash-unchanged guardrail test covers a live WAL writer. Merge order note: merge before #7 is fine either way — no overlapping files.

@ChelseaKR

Copy link
Copy Markdown
Owner Author

Drain round 2026-07-11: re-ran the full local gate on this branch today — green (ruff+marker-hygiene, mypy --strict, 203 tests @ 96.77% branch cov, pip-audit/gitleaks/osv-scanner clean, pa11y 0, eval green; the one gitleaks hit in a full-ref scan is a false positive on PR #43's branch content, not this PR — scoped scans of this branch and main are clean).

Merge attempted and hard-blocked by infrastructure: gh pr merge --squash rejected by the live protect-main ruleset (required checks standards/verify); re-ran both checks today — they still die at job start with the account-billing error, 0 steps executed; --admin was then attempted under the 2026-07-11 authorization and GitHub rejected it too because the ruleset has an empty bypass-actors list (nobody, including the owner, can bypass). Nothing can merge to main until a human either fixes Actions billing (⛔USER-1) or adds a bypass actor to / temporarily disables the protect-main ruleset. This PR is otherwise ready to land first, per roadmap order.

@ChelseaKR
ChelseaKR merged commit 01af2d2 into main Jul 12, 2026
5 checks passed
@ChelseaKR
ChelseaKR deleted the fix/wal-consistent-snapshots branch July 12, 2026 06:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant