fix(llm): HTTP 429 treated as hard failure in streaming path — no retry, no backoff, no Retry-After

[jdmanring]

Bug

In the streaming path (stream_llm), an HTTP 429 from the upstream provider causes an immediate event: error yield and return — no retry, no delay. stream_llm_with_fallback then advances to the next candidate in the fallback chain with no sleep between candidates. A brief rate-limit burst drains the entire configured fallback chain in under 1 second instead of waiting for the rate limit window to expire.

No code path in Odysseus reads the Retry-After response header.

Reproduction

1. Configure an NVIDIA NIM endpoint with a free-tier API key. Set a primary model and 2+ fallback models.
2. Send several agent requests in quick succession to hit the per-minute rate limit.
3. The fallback indicator fires immediately on the first 429, pulling a fallback model into service rather than waiting.
4. With all fallbacks on the same endpoint, the entire chain exhausts before the rate limit window resets.

From logs, a confirmed 429 sequence for memory extraction (non-streaming path, llm_call_async) on request cc5018fb8ec3:

2026-06-16T20:31:54.693758Z  HTTP 429 Too Many Requests
2026-06-16T20:31:54.750481Z  LLM async call failed in 0.04s (attempt 1): HTTP 429 NVIDIA rate-limited
2026-06-16T20:31:55.286105Z  HTTP 429 Too Many Requests
2026-06-16T20:31:55.286522Z  LLM async call failed in 0.03s (attempt 2): HTTP 429 NVIDIA rate-limited
2026-06-16T20:31:55.834757Z  HTTP 429 Too Many Requests
2026-06-16T20:31:55.835238Z  LLM async call failed in 0.05s (attempt 3): HTTP 429 NVIDIA rate-limited
2026-06-16T20:31:55.835384Z  LLM memory extraction failed; using fallback candidates: 429

3 attempts at ~0.5s intervals (LLMConfig.RETRY_DELAY), then immediate fallback. The streaming path used for all agent and chat completions is worse — it has zero retries: a single 429 immediately yields event: error and triggers the next candidate.

Root Cause

Streaming path: no retry on 429

stream_llm (src/llm_core.py, ~line 2051), OpenAI-compatible path:

async with client.stream('POST', target_url, json=payload, headers=h, timeout=stream_timeout) as r:
    if r.status_code != 200:
        raw = (await r.aread()).decode(errors="replace")
        friendly = _format_upstream_error(r.status_code, raw, target_url)
        yield f'event: error\ndata: {json.dumps({"status": r.status_code, "text": friendly, "raw": raw[:500]})}\n\n'
        return

429, 502, 503 — all handled identically. No retry loop in the streaming path.

Fallback: no inter-candidate delay

stream_llm_with_fallback (src/llm_core.py, ~line 2296):

async for chunk in stream_llm(url, model, messages, headers=headers, **kwargs):
    if chunk.startswith("event: error"):
        if not emitted and not is_last:
            last_error = chunk
            retried = True
            break      # immediately tries next candidate — no sleep
        yield chunk
        continue

No asyncio.sleep between candidates. On a 429, candidate 1 fires within milliseconds of candidate 0 failing.

Non-streaming path: fixed 0.5s delay, still ignores Retry-After

llm_call_async (src/llm_core.py, ~line 1659):

if r.status_code in (429, 502, 503, 504) and attempt < max_retries:
    await asyncio.sleep(LLMConfig.RETRY_DELAY)   # 0.5s fixed
    continue

MAX_RETRIES=3, RETRY_DELAY=0.5. Three attempts at 0.5s intervals — total ~1 second of backoff before fallback fires. The Retry-After header is not read; a provider that sends Retry-After: 60 still gets only 1 second of patience.

Retry-After never read

No code in any path reads or acts on the Retry-After response header (confirmed: grep -r "Retry-After\|retry_after" src/ routes/ returns nothing). Rate-limited providers typically include this header to communicate exactly how long to wait.

Scale of the Problem

From two log rotation files (logs/server.log + logs/server.log.1):

• 2,241 × HTTP 429 from integrate.api.nvidia.com
• 1,585 × HTTP 200 from the same endpoint
• 58% rate-limit failure rate on the primary model

429s appear in dense bursts consistent with a per-minute quota exhausting and the full fallback chain cycling through before the window resets.

Impact

A rate limit that should result in a brief wait instead immediately burns through fallback models, consuming their quota too. When all fallbacks share the same endpoint (common with NIM's multi-model catalog), the entire chain fails in under 1 second. When fallbacks are on a different endpoint, quota there gets consumed unnecessarily. For a user with a configured fallback chain, rate limiting feels like a total outage rather than a temporary throttle.

This compounds with the context budget bug (#54): fallback models receive history already trimmed to ~5K tokens, so a rate-limited primary plus a fallback response equals no memory of the session.

Fix

1. Read Retry-After before yielding the error in stream_llm:

Before the yield f'event: error' on a non-200 status, check the response headers for Retry-After. If present and within a reasonable ceiling (e.g. 30s), sleep and retry once before falling through to the error yield.

2. Separate 429 from hard failures in stream_llm_with_fallback:

A 429 means the server is alive and will accept requests after the stated window. Connection errors and 5xx mean something is wrong with the server. These deserve different treatment:

• Connection error / 5xx: fall back immediately (current behavior)
• 429: sleep for Retry-After or a minimum floor (e.g. 2s), then retry the same candidate once before advancing

3. Exponential backoff + Retry-After in llm_call_async:

Replace asyncio.sleep(LLMConfig.RETRY_DELAY) on 429 with asyncio.sleep(retry_after or backoff(attempt)). The 0.5s fixed delay is appropriate for transient 5xx errors; it is not appropriate for rate limits with a 30–60s window.

4. Keep 502/503 behavior unchanged:

The immediate retry with 0.5s delay works correctly for transient server errors. The fix should separate 429 handling without touching the 5xx retry behavior.

Files

• src/llm_core.py — stream_llm (OpenAI path ~line 2051), stream_llm_with_fallback (~line 2296), llm_call_async (~line 1659), LLMConfig (~line 18)

[github-actions]

⚠️ Issue description is incomplete. Please update the following sections:

• Install Method — select how you installed Odysseus
• Operating System — select your OS
• Steps to Reproduce — must include at least one numbered or bulleted step
• Expected Behaviour — section is empty or too short
• Actual Behaviour — section is empty or too short

This comment is deleted automatically once all sections are complete.

[jdmanring]

Reopening: closed as COMPLETED when the fix landed on develop, but per the fork rule an upstream-candidate issue stays OPEN until its upstream PR is filed — and nothing has been filed upstream yet. The staged branch is the active tracker until the PR goes up.