Incident evidence
The 2026-07-13 Tier-0 incident included a WriterInUseError, and long-running health/watch/drain processes were observed in macOS U (uninterruptible wait) state while the 16 GB database was under I/O pressure. The Tier-0 health check now backs off rather than repeatedly kickstarting a U/D process, but the underlying writer/I/O contention remains.
Follow-up
- Correlate writer ownership, queue depth, WAL size, checkpoint activity, and process state
- Emit bounded diagnostics when the writer lease is held unusually long
- Make
U/D backoff and escalation visible in operator telemetry
- Define thresholds that distinguish expected bursts from persistent contention
- Add a reproducible load test for writer arbitration under checkpoint/enrichment pressure
Safety
Preserve one-writer-at-a-time semantics. Do not solve this by weakening the writer lock, killing active writers, or mutating the canonical DB during diagnosis.
Incident evidence
The 2026-07-13 Tier-0 incident included a
WriterInUseError, and long-running health/watch/drain processes were observed in macOSU(uninterruptible wait) state while the 16 GB database was under I/O pressure. The Tier-0 health check now backs off rather than repeatedly kickstarting aU/Dprocess, but the underlying writer/I/O contention remains.Follow-up
U/Dbackoff and escalation visible in operator telemetrySafety
Preserve one-writer-at-a-time semantics. Do not solve this by weakening the writer lock, killing active writers, or mutating the canonical DB during diagnosis.