Incident evidence
During the 2026-07-13 Tier-0 incident, the missing-embeddings backlog was hundreds of rows and changed while enrichment drained it (477 in an older health state, then 433 in a later bounded read-only check). The health-check query itself was fixed separately; this issue is about the underlying backlog behavior.
Follow-up
- Establish backlog age, arrival rate, and drain-rate telemetry
- Distinguish transient healthy catch-up from a stuck or growing backlog
- Add an SLO/alert based on age and sustained growth, not one point-in-time count
- Identify quota, provider, poison-row, and enrichment-worker causes in the diagnostic output
- Document a safe recovery path that preserves write arbitration
Safety
Diagnosis should use read-only queries and existing telemetry. Do not bypass the single-writer policy or bulk-update the canonical DB.
Incident evidence
During the 2026-07-13 Tier-0 incident, the missing-embeddings backlog was hundreds of rows and changed while enrichment drained it (477 in an older health state, then 433 in a later bounded read-only check). The health-check query itself was fixed separately; this issue is about the underlying backlog behavior.
Follow-up
Safety
Diagnosis should use read-only queries and existing telemetry. Do not bypass the single-writer policy or bulk-update the canonical DB.