Skip to content

feat(openai): optional vision gating to strip images for non-vision models#257

Open
JumpLink wants to merge 1 commit into
danny-avila:mainfrom
faktenforum:feat/vision-capability
Open

feat(openai): optional vision gating to strip images for non-vision models#257
JumpLink wants to merge 1 commit into
danny-avila:mainfrom
faktenforum:feat/vision-capability

Conversation

@JumpLink

Copy link
Copy Markdown
Contributor

Problem

When an agent feeds image content to a model that has no vision support, OpenAI-compatible providers reject the whole request:

  • model is not a multimodal model
  • No endpoints found that support image input

This happens in practice when a tool returns image content (for example an image-generation tool whose base64 artifact is fed back as context) and the active model is text-only. The caller has no way to tell the client "this model can't take images, drop them."

Approach

Add an opt-in vision flag to the OpenAI-family chat classes. When vision: false, image_url content parts are stripped from the messages at a single choke point, right before the message stream is delegated to the base class.

A new exported helper stripImagesFromMessages(messages, visionCapable):

  • returns the input unchanged when visionCapable is true (default), so existing behavior is untouched;
  • only clones the messages that actually carry images;
  • substitutes a short text placeholder when stripping would empty a message, so the message still carries content.

Wired into ChatOpenAI, AzureChatOpenAI, ChatDeepSeek and ChatXAI via their _streamResponseChunks override. Because the strip runs before super._streamResponseChunks(...), it does not depend on the internal message-conversion path.

API

new ChatOpenAI({ /* ...existing fields... */, vision: false });

Defaults to true everywhere, so this is non-breaking.

Tests

src/llm/openai/utils/stripImages.test.ts covers: pass-through when vision-capable, stripping for non-vision models, placeholder substitution for image-only messages, untouched string/image-free messages, and no mutation of the input.

Relation to prior work

This supersedes #48 with a smaller, tested implementation (3 files, ~170 lines vs. 12 files, ~530). #48 filtered base64 artifacts at the artifact-payload layer; this strips all image_url parts at the LLM-client layer, which covers HTTP image URLs too and keeps the change in one place.

Related LibreChat issues: danny-avila/LibreChat#11418, danny-avila/LibreChat#11413.

Add a `vision` constructor flag to ChatOpenAI, AzureChatOpenAI,
ChatDeepSeek and ChatXAI. When set to false, image_url content parts are
stripped from messages at a single choke point before the message stream
is delegated to the base class, via a new exported helper
`stripImagesFromMessages`.

This prevents hard provider errors ("model is not a multimodal model" /
"No endpoints found that support image input") when an agent routes
image content (for example image-generation artifacts fed back as
context) to a model that has no vision support. The flag defaults to true,
so existing behavior is unchanged unless a caller opts in.

Stripping an image-only message substitutes a short text placeholder so
the message still carries content.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant