fix: translate OpenAI image_url content into RubyLLM attachments (#51)#54
Conversation
A multimodal user message added to the transcript in OpenAI content-array
form (a `text` part plus `{ type: "image_url", image_url: { url: ... } }`)
was passed to RubyLLM's `add_message` verbatim. RubyLLM treats a raw array
of OpenAI content hashes as plain text, so the image was silently dropped —
a vision model received text only and confabulated an answer, with no error.
Add `Raix::MultimodalContentAdapter`, which translates `image_url` parts into
RubyLLM attachments before `add_message`:
- base64 `data:` URIs are decoded into a binary StringIO, because RubyLLM's
`Attachment` does not recognize `data:` URIs and would treat one as a
filesystem path.
- http(s) URLs are passed through as-is (Attachment fetches them).
Anything that is not an array of hashes containing at least one `image_url`
part is returned untouched, so text completions and the Anthropic
cache_control multipart shape are unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6066d74ded
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| match = url.match(/\Adata:[^;,]*;base64,(.+)\z/m) | ||
| return url unless match |
There was a problem hiding this comment.
Decode parameterized base64 data URIs
When an image_url uses a valid data URI with media-type parameters, e.g. data:image/png;charset=utf-8;base64,... or data:image/svg+xml;name=...;base64,..., this regex does not match because it only allows characters up to the first semicolon before ;base64. The method then returns the original data: string, which the surrounding comment notes RubyLLM treats as a filesystem path, so these images still fail or get dropped instead of becoming attachments.
Useful? React with 👍 / 👎.
Fixes #51. (Supersedes #53, which GitHub auto-closed when its stacked base branch was deleted on merge of #52.)
Problem
A multimodal user message added to the
transcriptin OpenAI content-array form is silently dropped on the RubyLLM backend:ruby_llm_requestpassed the array straight tochat.add_message(role: :user, content:). RubyLLM treats a raw array of OpenAI content hashes as plain text — it never recognizes theimage_urlpart as an image, so nothing reaches the provider.Fix
New
Raix::MultimodalContentAdaptertranslates OpenAI-style content arrays into aRubyLLM::Contentwith attachments beforeadd_message:data:URIs are base64-decoded into a binaryStringIO. RubyLLM'sAttachmentdoes not recognizedata:URIs — it would classify one as a filesystem path and blow up onFile.binread. Decoding to IO is the same trick RubyLLM uses internally (providers/gemini/media.rb).image_urlpart is returned untouched, so text completions and the Anthropiccache_controlmultipart shape are unaffected.Tests
New
spec/raix/multimodal_content_adapter_spec.rb(6 examples) verifies adata:URI decodes to a realimage/pngattachment (Marcel content-sniffing), http URLs become URL attachments, text survives alongside the image, string-keyed parts work, and non-image content passes through unchanged.Full suite: 113 examples, 0 failures, 1 pending. RuboCop clean. CHANGELOG entry added under the existing
[2.0.5]section.🤖 Generated with Claude Code