fix: send tool image artifacts as a user message for openai-compatible providers#2
Merged
Merged
Conversation
…e providers
Scaleway/Mistral rejects image_url parts inside a tool message and also
rejects a user message directly after a tool message. formatArtifactPayload
inlined the image into the tool result, producing a 400 ("3 validation errors
for ToolMessage"). Keep the tool result text-only and, when the model is
vision-capable, append a short assistant bridge plus a user message carrying
the image(s) after the trailing tool run. This lets the vision model evaluate
the generated image and satisfies role rules across OpenAI, Scaleway/Mistral,
and Google. Images are dropped when the model has no vision.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
When an MCP/agent tool returns an image (e.g. the image-gen MCP),
formatArtifactPayloadinlined the imageimage_urlblock into thetoolmessage content before sending to OpenAI-compatible providers. Scaleway/Mistral rejects that with a 400:Why the obvious alternatives don't work on Scaleway/Mistral
Verified against the live Scaleway API with
mistral-small-3.2:toolmessageusermessage right after thetoolmessageUnexpected role 'user' after role 'tool'tooltext-only + shortassistantbridge +usermessage with the imageImages cannot live in a
toolmessage, and strict providers also forbid ausermessage directly after atoolmessage.Change
formatArtifactPayloadnow, for the trailing tool round:assistantbridge followed by ausermessage carrying the image(s).When the model has no vision, images are dropped (tool text remains), matching the downstream non-vision image strip. The call site passes
agentContext.visionand uses the returned array.Verified end-to-end:
formatArtifactPayload→_convertMessagesToOpenAIParams→ Scaleway returns 200 and the model describes the generated image. New unit tests cover the vision, non-vision, tool-content-strip, and non-tail cases.