feat: add Responses API support and OpenAI Chat image rewriting - #13
Conversation
- vision_client: VISION_API_PROTOCOL=responses sends /responses with input_image/input_text parts, max_output_tokens, and reasoning effort, enabling vision models like gpt-5.6-luna that only speak the Responses API - vision_client: send a non-default User-Agent for OpenCode Go compatibility - vision_proxy: detect and rewrite image_url blocks in OpenAI Chat Completions messages (openai_chat format) - tests: cover responses payload/parsing, UA header, image ordering, and openai_chat rewriting - docs: document VISION_API_PROTOCOL and VISION_REASONING_EFFORT in .env.example, READMEs, and AGENT_INSTALL
Anionex
left a comment
Anionex
left a comment
There was a problem hiding this comment.
Thanks for the focused contribution. The feature itself is valuable, and I confirmed that the main implementation works: the required local test suite passes, a real Responses API vision request succeeds, and an end-to-end OpenAI Chat image_url request is rewritten by the proxy and answered successfully by a text-only upstream.
I would like the following items addressed before merge:
-
Set
store: falsein Responses requests. The Responses API defaults to storing generated responses. These calls contain the source image and focus prompt, so the client should explicitly disable storage to preserve this project's data-handling expectations. Please add an assertion for this intests/test_vision_client.py. -
Make the custom User-Agent opt-in rather than unconditional. Hardcoding
agent-vision-toolkit/0.1changes every existing Chat Completions request even though the PR otherwise promises that the default path remains unchanged, and this repository keeps UA/header compatibility behavior disabled by default. AVISION_USER_AGENToverride would solve the endpoint-specific compatibility case without changing all existing users. Please test both the default and configured cases. -
Either propagate the new protocol setting to the native extensions or narrow the documentation.
README.mdsays all entry points share one configuration, but the Pi and OpenCode extensions do not readVISION_API_PROTOCOLand still always call/chat/completions. With a Responses-only endpoint, the Python client/proxy would work while those extensions would fail. Preferably add the same Responses branch to both single-file extensions and their tests; otherwise document that these variables apply only to the Python client/proxy. -
Use a host-neutral channel note for
openai_chat. It currently reuses the Codex note that tells the model to callview_image. A generic Chat Completions client may not expose that tool, so the note should explain re-querying without naming a host-specific tool.
Please also align the remaining setup text (AGENT_INSTALL.md prerequisite, .env.example capability comment, the shared-config list in the skill) and add the user-visible change to CHANGELOG.md.
Once these are addressed, I think this is worth merging. The body-shape dialect detection and reuse of the existing rewrite pipeline are good choices.
Anionex
changed the title
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Anionex
left a comment
Anionex
left a comment
There was a problem hiding this comment.
All requested changes are addressed on 931e186: store:false is covered, the UA hunk now uses main, native-extension scope is documented accurately, the OpenAI Chat note is host-neutral, supporting docs/changelog are aligned, all CI checks pass, and a real Responses API image request succeeded.
2 participants
Problem
The shared Python vision client only supported OpenAI Chat Completions, so a vision model exposed exclusively through the Responses API could not back
glance,ground,detect, or the Python proxy. Separately, the proxy did not recognize OpenAI Chat Completions request bodies containingmessages[].content[].image_url, so those images could reach a text-only upstream unchanged.Changes
VISION_API_PROTOCOL=responsesto the shared Python client:input_imageblocks followed byinput_text;max_output_tokens;VISION_REASONING_EFFORT;store: falseso image requests are not stored by default;output_textfrom Responses API output messages.image_urlblocks through the existing parallel vision-description pipeline.VISION_API_PROTOCOLandVISION_REASONING_EFFORTapply to the standalone Python CLIs/proxy; the Pi and OpenCode native extensions continue to use their own settings and/chat/completions.Review resolution
All requested changes from the maintainer review are addressed:
store: false, with a test assertion.VISION_USER_AGENTimplementation from fix: avoid Cloudflare 403 for urllib vision requests #21.view_image.AGENT_INSTALL.md,.env.example, the Skill shared-config list, bilingual README text, andCHANGELOG.mdare aligned.Verification
Passed locally after merging the latest
main:python3 -m py_compile vision_proxy.py vision_client.py ground.py detect.py bin/glance bin/trace bin/croppython3 tests/test_image_rewrite_shapes.pypython3 tests/test_focus_hint.pypython3 tests/test_anthropic_rewrite.pypython3 tests/smoke_test_proxy.pypython3 tests/smoke_test_egress_failover.pypython3 tests/test_vision_client.pypython3 tests/test_restore_ui_playbook.pynode tests/test_extensions.mjsgit diff --checkReal Responses API vision request:
/v1/responses;gpt-5.6-luna;input_image;A solid red square fills the image.This is a non-UI protocol/proxy change, so there is no meaningful browser verification path. The substitute verification covers the real Responses API request plus local end-to-end proxy and dialect fixtures.