Skip to content

fix: send tool image artifacts as a user message for openai-compatible providers#2

Merged
JumpLink merged 1 commit into
mainfrom
fix/scaleway-tool-image-vision
Jul 3, 2026
Merged

fix: send tool image artifacts as a user message for openai-compatible providers#2
JumpLink merged 1 commit into
mainfrom
fix/scaleway-tool-image-vision

Conversation

@JumpLink

@JumpLink JumpLink commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

What

When an MCP/agent tool returns an image (e.g. the image-gen MCP), formatArtifactPayload inlined the image image_url block into the tool message content before sending to OpenAI-compatible providers. Scaleway/Mistral rejects that with a 400:

3 validation errors for ToolMessage
content.str  Input should be a valid string ...
content.list[TextChunk].1.type  Input should be <ChunkTypes.text: 'text'> ...

Why the obvious alternatives don't work on Scaleway/Mistral

Verified against the live Scaleway API with mistral-small-3.2:

shape result
image in the tool message ❌ 400 (the error above)
image in a user message right after the tool message ❌ 400 Unexpected role 'user' after role 'tool'
tool text-only + short assistant bridge + user message with the image ✅ 200, and the vision model sees/evaluates the image

Images cannot live in a tool message, and strict providers also forbid a user message directly after a tool message.

Change

formatArtifactPayload now, for the trailing tool round:

  • keeps every tool result text-only (never places image parts in a tool message), and
  • when the model is vision-capable, appends a short assistant bridge followed by a user message carrying the image(s).

When the model has no vision, images are dropped (tool text remains), matching the downstream non-vision image strip. The call site passes agentContext.vision and uses the returned array.

Verified end-to-end: formatArtifactPayload_convertMessagesToOpenAIParams → Scaleway returns 200 and the model describes the generated image. New unit tests cover the vision, non-vision, tool-content-strip, and non-tail cases.

…e providers

Scaleway/Mistral rejects image_url parts inside a tool message and also
rejects a user message directly after a tool message. formatArtifactPayload
inlined the image into the tool result, producing a 400 ("3 validation errors
for ToolMessage"). Keep the tool result text-only and, when the model is
vision-capable, append a short assistant bridge plus a user message carrying
the image(s) after the trailing tool run. This lets the vision model evaluate
the generated image and satisfies role rules across OpenAI, Scaleway/Mistral,
and Google. Images are dropped when the model has no vision.
@JumpLink JumpLink merged commit 54cd296 into main Jul 3, 2026
7 of 10 checks passed
@JumpLink JumpLink deleted the fix/scaleway-tool-image-vision branch July 3, 2026 09:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant