feat(openai): optional vision gating to strip images for non-vision models#257
Open
JumpLink wants to merge 1 commit into
Open
feat(openai): optional vision gating to strip images for non-vision models#257JumpLink wants to merge 1 commit into
JumpLink wants to merge 1 commit into
Conversation
Add a `vision` constructor flag to ChatOpenAI, AzureChatOpenAI,
ChatDeepSeek and ChatXAI. When set to false, image_url content parts are
stripped from messages at a single choke point before the message stream
is delegated to the base class, via a new exported helper
`stripImagesFromMessages`.
This prevents hard provider errors ("model is not a multimodal model" /
"No endpoints found that support image input") when an agent routes
image content (for example image-generation artifacts fed back as
context) to a model that has no vision support. The flag defaults to true,
so existing behavior is unchanged unless a caller opts in.
Stripping an image-only message substitutes a short text placeholder so
the message still carries content.
This was referenced Jun 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When an agent feeds image content to a model that has no vision support, OpenAI-compatible providers reject the whole request:
model is not a multimodal modelNo endpoints found that support image inputThis happens in practice when a tool returns image content (for example an image-generation tool whose base64 artifact is fed back as context) and the active model is text-only. The caller has no way to tell the client "this model can't take images, drop them."
Approach
Add an opt-in
visionflag to the OpenAI-family chat classes. Whenvision: false,image_urlcontent parts are stripped from the messages at a single choke point, right before the message stream is delegated to the base class.A new exported helper
stripImagesFromMessages(messages, visionCapable):visionCapableistrue(default), so existing behavior is untouched;Wired into
ChatOpenAI,AzureChatOpenAI,ChatDeepSeekandChatXAIvia their_streamResponseChunksoverride. Because the strip runs beforesuper._streamResponseChunks(...), it does not depend on the internal message-conversion path.API
Defaults to
trueeverywhere, so this is non-breaking.Tests
src/llm/openai/utils/stripImages.test.tscovers: pass-through when vision-capable, stripping for non-vision models, placeholder substitution for image-only messages, untouched string/image-free messages, and no mutation of the input.Relation to prior work
This supersedes #48 with a smaller, tested implementation (3 files, ~170 lines vs. 12 files, ~530). #48 filtered base64 artifacts at the artifact-payload layer; this strips all
image_urlparts at the LLM-client layer, which covers HTTP image URLs too and keeps the change in one place.Related LibreChat issues: danny-avila/LibreChat#11418, danny-avila/LibreChat#11413.