Skip to content

fix(providers/openaicompat): forward tool result media to LLM#241

Open
ezynda3 wants to merge 1 commit into
charmbracelet:mainfrom
mark3labs:fix/openaicompat-tool-result-media
Open

fix(providers/openaicompat): forward tool result media to LLM#241
ezynda3 wants to merge 1 commit into
charmbracelet:mainfrom
mark3labs:fix/openaicompat-tool-result-media

Conversation

@ezynda3
Copy link
Copy Markdown

@ezynda3 ezynda3 commented May 16, 2026

Summary

Fixes #208.

The openaicompat provider's ToPromptFunc only handled ToolResultContentTypeText and ToolResultContentTypeError. Tool results carrying ToolResultOutputContentMedia silently fell through the switch with no matching case, producing no tool message at all. The downstream LLM never received the tool result, typically causing it to hallucinate or re-call the tool.

This affected every OpenAI-compatible provider built on top of openaicompat (OpenRouter, etc.).

Fix

Mirror the existing behavior in the openai provider (which already handles this correctly in providers/openai/language_model_hooks.go):

  • Emit a text tool message containing the media's accompanying Text (or a placeholder describing the media type) so the tool_call / tool_result pairing stays valid.
  • Follow up with a synthetic user message holding the image_url or input_audio content part so vision- and audio-capable models can actually see the media.

Supported media types via the synthetic user message:

Media type Mapped to
image/* image_url with data:-URI base64
audio/wav input_audio (format: wav)
audio/mpeg / audio/mp3 input_audio (format: mp3)

Unsupported types (e.g. video/mp4) still get the text tool message and now emit a CallWarning instead of being silently dropped — a strict improvement over the previous behavior.

A default: arm was also added to the switch to surface any future ToolResultContentType additions as a warning rather than dropping silently.

Tests

Added providers/openaicompat/tool_result_media_test.go covering:

  • image/png → synthetic user message with data-URI image_url
  • Accompanying Text is preferred over the placeholder
  • audio/wavinput_audio with format wav
  • audio/mpeginput_audio with format mp3
  • Unsupported video/mp4 → text-only tool message + warning, no synthetic user message
$ go test ./providers/openaicompat/... -count=1
ok  	charm.land/fantasy/providers/openaicompat	0.004s

Full suite (go test ./... -count=1) is green.

golangci-lint run ./providers/openaicompat/... is clean. gofumpt -l is clean.

Notes

  • The fix is intentionally scoped to openaicompat since that's what the bug report covers and what affects openrouter / azure (which both build on it). The google provider has a similar gap (providers/google/google.go ~line 503 only handles text/error) and is happy to be a follow-up PR if maintainers want — kept out of this one to keep the change focused and easy to review.
  • No public API changes. No new dependencies.

The openaicompat provider's ToPromptFunc only handled
ToolResultContentTypeText and ToolResultContentTypeError. Tool results
carrying ToolResultOutputContentMedia silently fell through the switch
with no matching case, producing no message at all. The downstream LLM
never received the tool result, typically causing it to hallucinate or
re-call the tool.

Mirror the openai provider's behavior:

  - Emit a text tool message containing the media's accompanying text
    (or a placeholder describing the media type) so the
    tool_call/tool_result pairing stays valid.
  - Follow up with a synthetic user message holding the image_url or
    input_audio content part so vision- and audio-capable models can
    actually see the media.

Supported media types: image/* (any), audio/wav, audio/mpeg, audio/mp3.
Unsupported types (e.g. video/mp4) still get the text tool message and
emit a CallWarning instead of being silently dropped.

Add table-driven coverage for png images, audio (wav, mp3), accompanying
text precedence, and the unsupported-media-type warning path.

Fixes charmbracelet#208
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

openaicompat provider silently drops ToolResultOutputContentMedia — image/media tool results are lost

2 participants