Image content in transcript (OpenAI image_url parts) is silently dropped on the RubyLLM backend

[ashhadsheikh]

Summary

When a multimodal user message is added to the transcript using the OpenAI-style content-array format (a text part plus an image_url part), the image is silently dropped on the RubyLLM backend. The model receives text only and — under a vision prompt — confabulates an answer rather than erroring. No exception is raised, so this fails silently.

Text-only completions and MCP image responses (#29) are unaffected; this is specifically about sending an image to a vision model via transcript.

Environment

• raix 2.0.1
• ruby_llm 1.14.0
• Provider: OpenRouter (Gemini vision models), but the mechanism is provider-independent

Root cause

ChatCompletion#chat_completion → ruby_llm_request forwards the transcript content array verbatim into RubyLLM:

# lib/raix/chat_completion.rb (~line 363)
chat.add_message(role: :user, content:)   # content = [ {type:"text",...}, {type:"image_url", image_url:{url:...}} ]

RubyLLM does not interpret OpenAI-style {type: "image_url", image_url: {url: ...}} parts. Its multimodal path expects attachments via chat.ask(prompt, with: <path|url|io|ActiveStorage>), which build a RubyLLM::Content. A raw array of OpenAI content hashes passed as content: is not recognised as an image, so the image_url part never becomes an image in the outgoing request. Raix performs no translation between the two formats.

This is the same class of bug seen elsewhere when an OpenAI content array is treated as plain text (e.g. crewAIInc/crewAI#2565).

Reproduction

require "base64"
require "raix"

# 2x2 solid-RED PNG
red = "iVBORw0KGgoAAAANSUhEUgAAAAIAAAACCAIAAAD91JpzAAAAEUlEQVR4nGP8z8Dwn4EIwDiqEAAQOAQBjEZ1pgAAAABJRU5ErkJggg=="
uri = "data:image/png;base64,#{red}"

klass = Class.new { include Raix::ChatCompletion }
c = klass.new
c.model = "google/gemini-2.5-flash"   # any vision model
c.temperature = 0.0
c.transcript << { system: "What is the single dominant color of this image? Answer with one word." }
c.transcript << { user: [{ type: "image_url", image_url: { url: uri } }] }
puts c.chat_completion

Expected vs actual

• Expected: Red (the model sees the image).
• Actual: Blue / Black — a guess. The model received no image. At temperature: 0.0 the answer also varies run-to-run, which confirms there is no stable image input.

Going straight through RubyLLM's native attachment API with the same model and image returns Red correctly:

RubyLLM.chat(model: "google/gemini-2.5-flash", provider: :openrouter, assume_model_exists: true)
       .ask("What is the dominant color? One word.", with: "red.png")   # => "Red"

Suggested fix / note

Raix should translate OpenAI-style image_url (and other multimodal) content parts into RubyLLM attachments before calling add_message, or document that images must be supplied via a RubyLLM-native mechanism.

It looks like #46 ("Standalone runtime: remove RubyLLM dependency") may resolve this incidentally by replacing the RubyLLM-backed execution with a Raix-native transport that speaks the OpenAI content format directly. If so, it'd be worth a regression test covering transcript-supplied image_url content.