fix: chat_json reports truncation instead of "unparseable response"#5
Merged
Conversation
An unparseable response with stop_reason "max_tokens" (Anthropic) or "length" (OpenAI-compatible) hit the max_tokens ceiling: retrying resends the identical request and pays for another truncated generation, then misreports the failure as "unparseable response". Raise immediately, naming the budget and stop_reason, so callers learn to raise max_tokens instead of debugging a parse error. A response that parses despite hitting the ceiling is still returned. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ama responses Review follow-ups on the truncation fail-fast: - Fail fast only at temperature 0, where greedy sampling provably reproduces the truncation. At temperature > 0 a retry may sample a shorter completion that fits, and existing callers on tight budgets (study_classifier, quality_agent) rely on that recovery — so keep the normal retries but name truncation in the final error instead of "unparseable response". - Check truncation before the empty-content branch: a budget consumed before any visible text (e.g. by thinking tokens) is truncation, not "empty response from model". - Log the truncated output at ERROR like the unparseable branch does. - Ollama provider: surface done_reason="length" as stop_reason instead of hardcoding "stop", so truncation detection works for ollama models. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Symptom
When a
chat_jsonresponse is cut off at themax_tokensceiling, the JSON can't parse, so the agent retries the identical request up to 3 times (paying for each truncated generation) and finally raises a misleadingValueError: Failed after 3 attempts: unparseable response. This cost a full debugging loop in evidenceseeker (cairn-ehr/evidenceseeker#8) before the real cause — the 4096 default budget — was found.Fix
chat_jsonnow checksresponse.stop_reasonagainst truncation markers ("max_tokens"for Anthropic,"length"for OpenAI-compatible) when the content doesn't parse:study_classifier/quality_agentrely on that recovery), but the final error names truncation and the budget instead of "unparseable response".done_reason="length"asstop_reasoninstead of hardcoding"stop"— previously truncation was invisible for every ollama-backed agent, so the new detection could never fire for them.Verification
tests/test_agents.py(6 new: parametrized greedy fail-fast, nonzero-temperature retry-then-name-cause, recovery on shorter retry, empty+truncated, parseable-despite-ceiling) and 8 intests/test_llm.pypass;ruffclean on touched files.🤖 Generated with Claude Code