Self-correct unknown tool calls via a hidden fallback tool#252
Open
yilunzhao wants to merge 3 commits into
Open
Self-correct unknown tool calls via a hidden fallback tool#252yilunzhao wants to merge 3 commits into
yilunzhao wants to merge 3 commits into
Conversation
TestOpenAIAgentsToolCorrection (and its Agents-only helpers) need the optional
`agents`/`openai` deps, which are not installed in the default CI test groups,
so they broke the default test command. Move them into a dedicated
tests/core/harness/test_openai_agents_scaffold.py guarded at module level by
pytest.importorskip("agents") / ("openai"); test_scaffold.py now keeps only the
non-Agents TestGetScaffold and no longer imports agents/openai. This is a pure
test move plus import guard, with no test logic changed.
Addresses review feedback on #239.
yilunzhao
force-pushed
the
yilun/unknown-tool-correction
branch
from
July 16, 2026 07:29
87efbec to
b8645a7
Compare
undfined
approved these changes
Jul 17, 2026
undfined
left a comment
undfined
left a comment
Collaborator
There was a problem hiding this comment.
As discussed off-line, this is fine for now but will you add a issue in the repo to upgrade openai agents sdk to the current latest as it supports this behavior natively.
Thanks!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Split out of #239 per review, so the behavior-level change can be reviewed on its own (would love @tylerm's take). Stacked on #239: the diff here shows only the self-correction commits; GitHub will retarget this PR to
mainwhen #239 merges.What it does
When the model calls a tool name that does not exist, the SDK used to raise
ModelBehaviorError, which ended the episode (with #239's accounting fix, the instance stays in the denominator and scores 0). Now the scaffold rewrites the unknown call to a hidden internal fallback tool whose result tells the model the real tool names, so it can correct itself and continue. TheModelBehaviorErrorcatch stays as a last-resort backstop.Why
This failure mode is common in practice. In an ExpertQA probe run, 3 of 5 instances died this way; even with tool names enumerated in the prompt it was still 1 of 10. Real agent deployments feed the error back rather than terminating, and the retries still consume the max_turns budget, so nothing is free.
What changes for existing evals
Applies to every task running through the
openai_agentsscaffold (litsearch, SAGE, etc.). Instances that previously scored 0 on a hallucinated tool name can now recover, so agentic numbers may shift slightly upward. The rewritten call and the corrective message are recorded in the trajectory, so the fumble stays visible.Field data from full runs with this in place: on LitSearch (500 queries), Qwen3.5-9B misspelled the tool name 3 times (e.g.
semantic_schollar_snippet_search) and recovered 3/3, one of those runs going on to find the gold paper; on a SAGE run, Olmo-3-7B had 12 hallucinated tool names rescued.Validation
Unit tests cover the rewrite, the hidden-tool wiring, a reserved-name guard, and SDK kwargs forwarding. Live checks: a 10-instance regression run showed no regressions, and a forced-hallucination probe confirmed the full loop end to end (unknown call rewritten, corrective message returned, model retried with the correct name and completed normally).
Note: #249 adds a test file with the same name (
tests/core/harness/test_openai_agents_scaffold.py); whichever lands second needs a trivial rebase.cc @donovanr