fix(providers/openaicompat): forward tool result media to LLM#241
Open
ezynda3 wants to merge 1 commit into
Open
fix(providers/openaicompat): forward tool result media to LLM#241ezynda3 wants to merge 1 commit into
ezynda3 wants to merge 1 commit into
Conversation
The openaicompat provider's ToPromptFunc only handled
ToolResultContentTypeText and ToolResultContentTypeError. Tool results
carrying ToolResultOutputContentMedia silently fell through the switch
with no matching case, producing no message at all. The downstream LLM
never received the tool result, typically causing it to hallucinate or
re-call the tool.
Mirror the openai provider's behavior:
- Emit a text tool message containing the media's accompanying text
(or a placeholder describing the media type) so the
tool_call/tool_result pairing stays valid.
- Follow up with a synthetic user message holding the image_url or
input_audio content part so vision- and audio-capable models can
actually see the media.
Supported media types: image/* (any), audio/wav, audio/mpeg, audio/mp3.
Unsupported types (e.g. video/mp4) still get the text tool message and
emit a CallWarning instead of being silently dropped.
Add table-driven coverage for png images, audio (wav, mp3), accompanying
text precedence, and the unsupported-media-type warning path.
Fixes charmbracelet#208
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #208.
The
openaicompatprovider'sToPromptFunconly handledToolResultContentTypeTextandToolResultContentTypeError. Tool results carryingToolResultOutputContentMediasilently fell through the switch with no matching case, producing no tool message at all. The downstream LLM never received the tool result, typically causing it to hallucinate or re-call the tool.This affected every OpenAI-compatible provider built on top of
openaicompat(OpenRouter, etc.).Fix
Mirror the existing behavior in the
openaiprovider (which already handles this correctly inproviders/openai/language_model_hooks.go):toolmessage containing the media's accompanyingText(or a placeholder describing the media type) so thetool_call/tool_resultpairing stays valid.usermessage holding theimage_urlorinput_audiocontent part so vision- and audio-capable models can actually see the media.Supported media types via the synthetic user message:
image/*image_urlwithdata:-URI base64audio/wavinput_audio(format:wav)audio/mpeg/audio/mp3input_audio(format:mp3)Unsupported types (e.g.
video/mp4) still get the text tool message and now emit aCallWarninginstead of being silently dropped — a strict improvement over the previous behavior.A
default:arm was also added to the switch to surface any futureToolResultContentTypeadditions as a warning rather than dropping silently.Tests
Added
providers/openaicompat/tool_result_media_test.gocovering:image/png→ synthetic user message with data-URIimage_urlTextis preferred over the placeholderaudio/wav→input_audiowith formatwavaudio/mpeg→input_audiowith formatmp3video/mp4→ text-only tool message + warning, no synthetic user messageFull suite (
go test ./... -count=1) is green.golangci-lint run ./providers/openaicompat/...is clean.gofumpt -lis clean.Notes
openaicompatsince that's what the bug report covers and what affectsopenrouter/azure(which both build on it). Thegoogleprovider has a similar gap (providers/google/google.go~line 503 only handles text/error) and is happy to be a follow-up PR if maintainers want — kept out of this one to keep the change focused and easy to review.