Skip to content

fix: translate OpenAI image_url content into RubyLLM attachments (#51)#54

Merged
obie merged 2 commits into
mainfrom
51-image-content-rubyllm
Jun 4, 2026
Merged

fix: translate OpenAI image_url content into RubyLLM attachments (#51)#54
obie merged 2 commits into
mainfrom
51-image-content-rubyllm

Conversation

@obie

@obie obie commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Fixes #51. (Supersedes #53, which GitHub auto-closed when its stacked base branch was deleted on merge of #52.)

Problem

A multimodal user message added to the transcript in OpenAI content-array form is silently dropped on the RubyLLM backend:

c.transcript << { user: [{ type: "image_url", image_url: { url: "data:image/png;base64,..." } }] }
c.chat_completion   # vision model gets TEXT ONLY → confabulates an answer, no error raised

ruby_llm_request passed the array straight to chat.add_message(role: :user, content:). RubyLLM treats a raw array of OpenAI content hashes as plain text — it never recognizes the image_url part as an image, so nothing reaches the provider.

Fix

New Raix::MultimodalContentAdapter translates OpenAI-style content arrays into a RubyLLM::Content with attachments before add_message:

  • data: URIs are base64-decoded into a binary StringIO. RubyLLM's Attachment does not recognize data: URIs — it would classify one as a filesystem path and blow up on File.binread. Decoding to IO is the same trick RubyLLM uses internally (providers/gemini/media.rb).
  • http(s) URLs pass through unchanged (Attachment fetches them).
  • Anything that isn't an array of hashes with at least one image_url part is returned untouched, so text completions and the Anthropic cache_control multipart shape are unaffected.
chat.add_message(role: :user, content: MultimodalContentAdapter.translate(content))

Tests

New spec/raix/multimodal_content_adapter_spec.rb (6 examples) verifies a data: URI decodes to a real image/png attachment (Marcel content-sniffing), http URLs become URL attachments, text survives alongside the image, string-keyed parts work, and non-image content passes through unchanged.

Full suite: 113 examples, 0 failures, 1 pending. RuboCop clean. CHANGELOG entry added under the existing [2.0.5] section.

🤖 Generated with Claude Code

obie and others added 2 commits June 4, 2026 11:13
A multimodal user message added to the transcript in OpenAI content-array
form (a `text` part plus `{ type: "image_url", image_url: { url: ... } }`)
was passed to RubyLLM's `add_message` verbatim. RubyLLM treats a raw array
of OpenAI content hashes as plain text, so the image was silently dropped —
a vision model received text only and confabulated an answer, with no error.

Add `Raix::MultimodalContentAdapter`, which translates `image_url` parts into
RubyLLM attachments before `add_message`:
- base64 `data:` URIs are decoded into a binary StringIO, because RubyLLM's
  `Attachment` does not recognize `data:` URIs and would treat one as a
  filesystem path.
- http(s) URLs are passed through as-is (Attachment fetches them).

Anything that is not an array of hashes containing at least one `image_url`
part is returned untouched, so text completions and the Anthropic
cache_control multipart shape are unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@obie obie merged commit 2d46314 into main Jun 4, 2026
1 check passed
@obie obie deleted the 51-image-content-rubyllm branch June 4, 2026 09:14

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6066d74ded

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +55 to +56
match = url.match(/\Adata:[^;,]*;base64,(.+)\z/m)
return url unless match

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Decode parameterized base64 data URIs

When an image_url uses a valid data URI with media-type parameters, e.g. data:image/png;charset=utf-8;base64,... or data:image/svg+xml;name=...;base64,..., this regex does not match because it only allows characters up to the first semicolon before ;base64. The method then returns the original data: string, which the surrounding comment notes RubyLLM treats as a filesystem path, so these images still fail or get dropped instead of becoming attachments.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Image content in transcript (OpenAI image_url parts) is silently dropped on the RubyLLM backend

1 participant