Skip to content

fix: chat_json reports truncation instead of "unparseable response"#5

Merged
hherb merged 3 commits into
mainfrom
fix/chat-json-truncation-fail-fast
Jul 2, 2026
Merged

fix: chat_json reports truncation instead of "unparseable response"#5
hherb merged 3 commits into
mainfrom
fix/chat-json-truncation-fail-fast

Conversation

@hherb

@hherb hherb commented Jul 2, 2026

Copy link
Copy Markdown
Owner

Symptom

When a chat_json response is cut off at the max_tokens ceiling, the JSON can't parse, so the agent retries the identical request up to 3 times (paying for each truncated generation) and finally raises a misleading ValueError: Failed after 3 attempts: unparseable response. This cost a full debugging loop in evidenceseeker (cairn-ehr/evidenceseeker#8) before the real cause — the 4096 default budget — was found.

Fix

  • chat_json now checks response.stop_reason against truncation markers ("max_tokens" for Anthropic, "length" for OpenAI-compatible) when the content doesn't parse:
    • temperature 0: raise immediately — greedy sampling provably reproduces the identical truncation, so retries only pay for it again.
    • temperature > 0: keep the normal retries (a retry may sample a shorter completion that fits — existing callers on tight budgets like study_classifier/quality_agent rely on that recovery), but the final error names truncation and the budget instead of "unparseable response".
  • Truncation is checked before the empty-content branch, so a budget consumed before any visible text (e.g. by thinking tokens) reports as truncation, not "empty response from model".
  • The truncated output is logged at ERROR, matching the unparseable branch.
  • A response that parses despite hitting the ceiling is returned as usual.
  • Ollama provider: surface done_reason="length" as stop_reason instead of hardcoding "stop" — previously truncation was invisible for every ollama-backed agent, so the new detection could never fire for them.

Verification

  • 21 tests in tests/test_agents.py (6 new: parametrized greedy fail-fast, nonzero-temperature retry-then-name-cause, recovery on shorter retry, empty+truncated, parseable-despite-ceiling) and 8 in tests/test_llm.py pass; ruff clean on touched files.

🤖 Generated with Claude Code

hherb and others added 3 commits July 3, 2026 08:14
An unparseable response with stop_reason "max_tokens" (Anthropic) or
"length" (OpenAI-compatible) hit the max_tokens ceiling: retrying
resends the identical request and pays for another truncated
generation, then misreports the failure as "unparseable response".
Raise immediately, naming the budget and stop_reason, so callers learn
to raise max_tokens instead of debugging a parse error. A response that
parses despite hitting the ceiling is still returned.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ama responses

Review follow-ups on the truncation fail-fast:
- Fail fast only at temperature 0, where greedy sampling provably
  reproduces the truncation. At temperature > 0 a retry may sample a
  shorter completion that fits, and existing callers on tight budgets
  (study_classifier, quality_agent) rely on that recovery — so keep the
  normal retries but name truncation in the final error instead of
  "unparseable response".
- Check truncation before the empty-content branch: a budget consumed
  before any visible text (e.g. by thinking tokens) is truncation, not
  "empty response from model".
- Log the truncated output at ERROR like the unparseable branch does.
- Ollama provider: surface done_reason="length" as stop_reason instead
  of hardcoding "stop", so truncation detection works for ollama models.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hherb hherb merged commit daddcae into main Jul 2, 2026
4 checks passed
@hherb hherb deleted the fix/chat-json-truncation-fail-fast branch July 2, 2026 23:08
@hherb hherb mentioned this pull request Jul 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant