fix(ingest): consistent, verified snapshots under live writers (FIX-02)#38
Conversation
A plain byte-copy of a mid-write metadata.db/statistics.sqlite is torn: committed rows still living in the -wal sidecar are silently dropped, and immutable=1 then suppresses SQLite's recovery, so the reader sees stale state with no error. Now: - when -wal/-shm/-journal sidecars are present, the snapshot is taken via SQLite's online backup API from a strictly read-only (mode=ro) source handle — one transactionally-consistent snapshot, WAL folded in, written as a standalone sidecar-free file; - every snapshot (both paths) is verified with PRAGMA integrity_check before a reader sees it; a failed plain copy escalates to the backup path once, then fails loudly with the new SnapshotIntegrityError; - the source is never opened writable anywhere (guardrail test asserts the source main file's hash is unchanged after snapshotting a live WAL-mode DB). Tests: 4 new (torn-copy bug demonstrated + consistency, source-unmutated hash check, missing source, corrupt-source fail-loud). 203 total pass, 96.77% branch coverage; docs/audits/coverage.xml regenerated in-sync. Verified locally (CI is billing-blocked): make verify green end-to-end (ruff, mypy --strict, pytest, pip-audit + gitleaks + osv-scanner, static a11y + pa11y/axe, offline eval). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Review verdict — 2026-07-09: READY TO MERGE (verified locally: make verify green — ruff+marker-hygiene, mypy --strict, 203 tests @ 96.77% cov, pip-audit/gitleaks/osv-scanner clean, static a11y + pa11y 0 violations, eval green; CI billing-blocked, local run is the review gate). Adversarial self-review notes: backup path opens source strictly mode=ro (never writable); hot-journal sources a read-only handle cannot recover raise and land in the fail-loud SnapshotIntegrityError path (correct: no silent bad read); source-hash-unchanged guardrail test covers a live WAL writer. Merge order note: merge before #7 is fine either way — no overlapping files. |
|
Drain round 2026-07-11: re-ran the full local gate on this branch today — green (ruff+marker-hygiene, mypy --strict, 203 tests @ 96.77% branch cov, pip-audit/gitleaks/osv-scanner clean, pa11y 0, eval green; the one gitleaks hit in a full-ref scan is a false positive on PR #43's branch content, not this PR — scoped scans of this branch and main are clean). Merge attempted and hard-blocked by infrastructure: |
# Conflicts: # docs/audits/coverage.xml
What
Closes the WAL-torn-snapshot correctness hole (ideation FIX-02, roadmap BF-2). This is the finished work that sat uncommitted in the checkout since 2026-07-05, rescued via
rescue/uncommitted-2026-07-09(content byte-identical torescue/uncommitted-2026-07-06) and rebuilt clean off currentmain.metadata.db/statistics.sqliteomits committed rows still in the-walsidecar;immutable=1then suppresses SQLite recovery, so readers silently saw stale/inconsistent state. It could never corrupt sources (that guarantee already held) but could silently mis-ingest.-wal/-shm/-journalsidecars are present,ingest.snapshot.snapshot()copies via SQLite's online backup API from a strictly read-only (mode=ro) source handle — one transactionally-consistent snapshot, WAL folded in, written as a standalone sidecar-free file. Every snapshot (both paths) is verified withPRAGMA integrity_check; a failed plain copy escalates to the backup path once, then fails loudly with the newSnapshotIntegrityErrorinstead of handing a reader a bad copy.docs/audits/coverage.xmlregenerated in the same PR (resolves the deliberate leftover noted in the roadmap).Tests
4 new tests in
tests/test_snapshot_readonly.py— including one that first demonstrates the torn-copy bug (naive copy of a WAL-mode DB drops committed rows) before proving the fixed path returns all rows withintegrity_check = okand no sidecars.Verification (CI is billing-blocked — local gate run is the review gate)
make verifygreen end-to-end on this branch: ruff format+check + marker-hygiene,mypy --strict, 203 tests passed, 96.77% branch coverage (gate ≥85%), pip-audit + gitleaks (0 leaks) + osv-scanner (0 vulns vsuv.lock), static a11y checker + pa11y/axe (0 violations), offline eval (content beats popularity). GitHub Actions cannot run on this private repo until the account billing issue is resolved (⛔USER-1); per the CI-dead protocol, this local run is the merge evidence.Follow-up (not this PR)
Before trusting as a daily driver, validate once against a live, busy Calibre library (roadmap §7 item 8, real-data gate).
🤖 Generated with Claude Code