Skip to content
This repository was archived by the owner on May 20, 2026. It is now read-only.

fix: rotate prompt_cache_key per call and treat 413 as retryable - #2

Open
Blue-B wants to merge 1 commit into
jvm:mainfrom
Blue-B:fix/413-rotate-prompt-cache-key
Open

fix: rotate prompt_cache_key per call and treat 413 as retryable#2
Blue-B wants to merge 1 commit into
jvm:mainfrom
Blue-B:fix/413-rotate-prompt-cache-key

Conversation

@Blue-B

@Blue-B Blue-B commented May 17, 2026

Copy link
Copy Markdown

Rotate prompt_cache_key per call and treat 413 as retryable

Fixes #1

Problem

Calling codex_generate_image repeatedly in the same Pi session eventually
fails with 413 Request exceeds the maximum size, because:

  1. prompt_cache_key is pinned to sessionId, so the Codex backend keeps
    replaying accumulated cached prefix for that key across calls.
  2. 413 is not in isRetryableStatus, so the failure is surfaced immediately
    with no recovery.
  3. Even with a retry, reusing the same prompt_cache_key would hit the same
    accumulated state and 413 again.

For image_generation calls the prompt-cache hit rate is effectively zero
(each call produces a new image), so pinning the cache key only inherits the
downside.

Change

  • Add 408 / 413 / 425 to isRetryableStatus and add request exceeds | payload too large to the message-pattern matcher.
  • New makeRequestCacheKey(sessionId) returns
    ${sessionId}:${timestamp}:${random} so every image call uses its own cache
    slot. sessionId is still part of the key, so any future per-session
    bucketing on the backend side keeps working.
  • buildRequestBody accepts an optional cacheKeyOverride so the retry loop
    can rotate the key after a 413 (${sessionId}:retry${attempt}) and rebuild
    the body inside the loop.
  • The existing exponential-backoff + jitter loop is reused unchanged.

No new dependencies. No behavior change for non-413 paths beyond the cache-key
rotation (which is a no-op for cache-miss workloads like image generation).

Repro / verification

Before: 3–5 sequential codex_generate_image calls in the same Pi session
reliably produce
Codex image generation request failed (413): Request exceeds the maximum size.

After: same workload completes successfully; on the (now rare) case a 413 still
arrives, the retry rotates the cache key and succeeds on attempt 2.

Notes

  • prompt_cache_key is documented as advisory: the server uses it as a hint
    for prefix caching, so rotating it per call is safe.
  • I considered keeping sessionId as the cache key and only rotating on 413,
    but that means every fresh session pays the 413 + retry round-trip the first
    time it hits the limit. Per-call rotation makes the failure mode disappear
    entirely without sacrificing anything image_generation actually benefits from.

Calling codex_generate_image repeatedly in the same Pi session
eventually fails with '413 Request exceeds the maximum size' because:

1. prompt_cache_key is pinned to sessionId, so the Codex Responses
   backend keeps replaying accumulated cached prefix for that key
   across calls. image_generation calls don't benefit from prefix
   caching (each call produces a fresh image) but they inherit the
   accumulation.
2. 413 is not in isRetryableStatus, so the failure is surfaced
   immediately with no recovery.
3. Even with a retry, reusing the same prompt_cache_key would hit
   the same accumulated state and 413 again.

Changes:
- Add 408/413/425 to isRetryableStatus and add 'request exceeds |
  payload too large' to the message-pattern matcher.
- New makeRequestCacheKey(sessionId) returns
  ${sessionId}:${timestamp}:${random} so every image call uses
  its own cache slot. sessionId is still part of the key, so any
  future per-session bucketing on the backend keeps working.
- buildRequestBody accepts an optional cacheKeyOverride so the
  retry loop can rotate the key after a 413
  (${sessionId}:retry${attempt}) and rebuild the body inside
  the loop.
- The existing exponential-backoff + jitter loop is reused
  unchanged.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

413 Request exceeds the maximum size after a few image generations in the same Pi session

1 participant