Context
On 2026-06-22 Cirdi was down: every query returned HTTP 500. Root cause was an OpenAI 429 insufficient_quota — the OpenAI account ran out of credit. R2R embeds every query with openai/text-embedding-3-small (the retrieval step), so an exhausted OpenAI quota 500s every request even though generation runs on Mistral. Health endpoint stayed ok throughout, which masked the cause.
This is a single point of failure with no early warning, and it will recur.
Actions
Notes
Context
On 2026-06-22 Cirdi was down: every query returned HTTP 500. Root cause was an OpenAI
429 insufficient_quota— the OpenAI account ran out of credit. R2R embeds every query withopenai/text-embedding-3-small(the retrieval step), so an exhausted OpenAI quota 500s every request even though generation runs on Mistral. Health endpoint stayedokthroughout, which masked the cause.This is a single point of failure with no early warning, and it will recur.
Actions
OPENAI_API_KEY, set a monthly budget limit and a usage/balance alert email (well below the hard cap).deploy/that embeddings depend on OpenAI billing, distinct from the Mistral generation account.Notes
text-embedding-3-smalland would need full re-embedding.