feat: add Responses API support and OpenAI Chat image rewriting

.env.example

@@ -1,6 +1,7 @@
 # Only the vision API is configured here. DeepSeek auth is still sent by Codex and passed through by the proxy.
 VISION_API_KEY=
-# Any OpenAI-compatible endpoint that supports /chat/completions with image_url works.
+# Any OpenAI-compatible endpoint that supports /chat/completions with image_url works;
+# the Python client/proxy can also use /responses with input_image (see below).
 # Recommended options (see README):
 #   OpenRouter:        https://openrouter.ai/api/v1
 #   Aliyun DashScope:  https://dashscope.aliyuncs.com/compatible-mode/v1
@@ -9,6 +10,11 @@ VISION_BASE_URL=https://openrouter.ai/api/v1
 VISION_MODEL=google/gemini-3.6-flash
 # Vision model output language: zh=Chinese, en=English (defaults to Chinese when unset)
 LANG=zh
+# Python client/proxy protocol: chat_completions (default) or responses. Choose
+# "responses" for models exposed only through the Responses API.
+# VISION_API_PROTOCOL=chat_completions
+# Reasoning effort sent with the responses protocol (optional).
+# VISION_REASONING_EFFORT=medium
 # Optional outbound User-Agent override. The default is browser-compatible to avoid
 # gateways that block Python-urllib clients.
 # VISION_USER_AGENT=custom-vision-client/1.0

AGENT_INSTALL.md

@@ -27,7 +27,7 @@
 
 - 已接入纯文本模型并能正常对话的宿主（Codex 或 Claude Code）
 - Python 3.11+
-- 一个支持 `/chat/completions` 和 `image_url` 的 OpenAI-compatible 视觉 API
+- 一个支持 `/chat/completions` 和 `image_url` 的 OpenAI-compatible 视觉 API；Python 客户端/代理也可配置 `/responses` + `input_image`
 
 ## 1. 定位并备份现有配置
 
@@ -58,6 +58,8 @@ VISION_API_KEY=...
 VISION_BASE_URL=...
 VISION_MODEL=...
 LANG=zh  # 可选：视觉模型输出语言（zh/en），不填保持默认中文
+# VISION_API_PROTOCOL=chat_completions  # 可选：Python 客户端/代理可改用 responses 协议
+# VISION_REASONING_EFFORT=medium        # 可选：responses 协议下的推理强度
 # VISION_USER_AGENT=custom-vision-client/1.0  # 可选：覆盖默认的浏览器兼容 User-Agent
 ```
 

CHANGELOG.md

@@ -4,6 +4,11 @@ All notable user-facing changes to agent-vision-toolkit are documented in this f
 
 ## [Unreleased]
 
+### Added
+
+- Let the shared Python vision client call either Chat Completions or Responses APIs, including optional reasoning effort and explicit `store: false` data handling.
+- Rewrite OpenAI Chat Completions `image_url` blocks through the existing vision-description pipeline with a host-neutral channel note.
+
 ### Fixed
 
 - Send a browser-compatible, configurable User-Agent from the shared Python vision client so Cloudflare-backed OpenAI-compatible endpoints do not reject the default `Python-urllib` signature.

README.md

@@ -151,7 +151,7 @@ VISION_BASE_URL=https://openrouter.ai/api/v1
 VISION_MODEL=google/gemini-3.6-flash
 ```
 
-Any OpenAI-compatible endpoint that supports `/chat/completions` with `image_url` works (e.g. Aliyun DashScope: `https://dashscope.aliyuncs.com/compatible-mode/v1` + `qwen-vl-max-latest`). Add `LANG=en` for English descriptions (default is Chinese).
+Any OpenAI-compatible endpoint that supports `/chat/completions` with `image_url` works (e.g. Aliyun DashScope: `https://dashscope.aliyuncs.com/compatible-mode/v1` + `qwen-vl-max-latest`). The Python client/proxy can also use `/responses` with `input_image` by setting `VISION_API_PROTOCOL=responses`. Add `LANG=en` for English descriptions (default is Chinese).
 
 **2. Put the CLIs on your PATH:**
 
@@ -322,14 +322,16 @@ Codex -> 127.0.0.1:19100 -> your existing text-only upstream
 <details>
 <summary><b>Environment variables</b></summary>
 
-The toolkit and proxy use only these environment variables; just three are required:
+The standalone CLIs and Python proxy use these environment variables; just three are required. The native Pi and OpenCode extensions use their own settings and currently call `/chat/completions` only.
 
 | Variable | Required | Description |
 |---|---:|---|
 | `VISION_API_KEY` | Yes | API key of the multimodal model |
 | `VISION_BASE_URL` | Yes | OpenAI-compatible API base URL |
 | `VISION_MODEL` | Yes | Multimodal model name |
 | `LANG` | No | Vision model output language: `zh` (Chinese) or `en` (English); default `zh` |
+| `VISION_API_PROTOCOL` | No | Python client/proxy protocol: `chat_completions` (default) or `responses` |
+| `VISION_REASONING_EFFORT` | No | Optional provider-supported reasoning effort for the Python client/proxy when using `responses` |
 | `VISION_USER_AGENT` | No | Outbound User-Agent for the Python client/proxy; defaults to a browser-compatible value and can be overridden for provider requirements |
 
 </details>
@@ -349,7 +351,7 @@ The route whose connection (TCP/TLS handshake) succeeds is kept in memory and re
 ## Prerequisites
 
 - A coding agent already working with a model, including a text-only model such as DeepSeek V4
-- An OpenAI-compatible vision API that supports `/chat/completions` and `image_url`
+- An OpenAI-compatible vision API that supports `/chat/completions` and `image_url`; the Python client/proxy can also use `/responses` with `input_image` via `VISION_API_PROTOCOL=responses`
 - No other configuration is required
 
 ## FAQ

README_CN.md

@@ -151,7 +151,7 @@ VISION_BASE_URL=https://openrouter.ai/api/v1
 VISION_MODEL=google/gemini-3.6-flash
 ```
 
-任何支持 `/chat/completions` 与 `image_url` 的 OpenAI-compatible 端点都可以（如阿里云百炼：`https://dashscope.aliyuncs.com/compatible-mode/v1` + `qwen-vl-max-latest`）。需要英文描述时加 `LANG=en`（默认中文）。
+任何支持 `/chat/completions` 与 `image_url` 的 OpenAI-compatible 端点都可以（如阿里云百炼：`https://dashscope.aliyuncs.com/compatible-mode/v1` + `qwen-vl-max-latest`）。Python 客户端/代理也可设置 `VISION_API_PROTOCOL=responses` 使用 `/responses` + `input_image`。需要英文描述时加 `LANG=en`（默认中文）。
 
 **2. 把 CLI 放进 PATH：**
 
@@ -319,14 +319,16 @@ Codex -> 127.0.0.1:19100 -> 用户原有的纯文本模型上游
 <details>
 <summary><b>环境变量</b></summary>
 
-工具箱与代理都只需要这些环境变量，必填的只有3个：
+独立 CLI 和 Python 代理使用这些环境变量，必填的只有 3 个。Pi 与 OpenCode 原生扩展使用各自设置，目前仅调用 `/chat/completions`。
 
 | 变量 | 必需 | 说明 |
 |---|---:|---|
 | `VISION_API_KEY` | 是 | 多模态模型的 API key |
 | `VISION_BASE_URL` | 是 | OpenAI-compatible API 地址 |
 | `VISION_MODEL` | 是 | 多模态模型名 |
 | `LANG` | 否 | 视觉模型输出语言：`zh`=中文，`en`=English（默认 `zh`） |
+| `VISION_API_PROTOCOL` | 否 | Python 客户端/代理的视觉 API 协议：`chat_completions`（默认）或 `responses` |
+| `VISION_REASONING_EFFORT` | 否 | Python 客户端/代理使用 `responses` 时可选的服务商支持推理强度 |
 | `VISION_USER_AGENT` | 否 | Python 客户端/代理的出站 User-Agent；默认使用浏览器兼容值，也可按服务商要求覆盖 |
 
 </details>
@@ -346,7 +348,7 @@ Codex -> 127.0.0.1:19100 -> 用户原有的纯文本模型上游
 ## 前置条件
 
 - 已接入(纯文本)模型（如 DeepSeek V4）并可正常使用的 coding agent
-- 一个支持 `/chat/completions` 与 `image_url` 的 OpenAI-compatible 视觉 API
+- 一个支持 `/chat/completions` 与 `image_url` 的 OpenAI-compatible 视觉 API；Python 客户端/代理也可通过 `VISION_API_PROTOCOL=responses` 使用 `/responses` + `input_image`
 - 没有其他需要的配置
 
 ## 常见问题

skills/vision-tools/SKILL.md

@@ -15,7 +15,8 @@ description: >-
 
 Five local CLIs that give a text-only agent eyes. They read one shared
 vision config (`VISION_API_KEY` / `VISION_BASE_URL` / `VISION_MODEL` /
-`LANG`) — no extra credentials.
+`LANG`), plus the optional Python-client settings `VISION_API_PROTOCOL`,
+`VISION_REASONING_EFFORT`, and `VISION_USER_AGENT` — no extra credentials.
 
 Pick the tool by the question you are answering:
 

tests/test_image_rewrite_shapes.py

@@ -60,6 +60,25 @@ def test_shapes():
     print("PASS: shapes A (content) and B (output) rewritten; text-only untouched")
 
 
+def test_openai_chat_images_are_rewritten():
+    mod = _load_proxy()
+    body = {"model": "deepseek-v4-flash", "messages": [{
+        "role": "user",
+        "content": [
+            {"type": "text", "text": "请读出图片里的标题"},
+            {"type": "image_url", "image_url": {"url": "data:image/png;base64,AAAA"}},
+        ],
+    }]}
+    assert asyncio.run(mod._rewrite_image_inputs(body))
+    content = body["messages"][0]["content"]
+    texts = [block["text"] for block in content if block.get("type") == "text"]
+    note = next(text for text in texts if text.startswith("[vision proxy]"))
+    assert "view_image" not in note, note
+    assert "submit the image again" in note, note
+    assert any("TEST-DESC" in text for text in texts), content
+    print("PASS: OpenAI Chat Completions image_url rewritten")
+
+
 def test_parallel():
     mod = _load_proxy()
     real_desc = mod._image_desc_from_url
@@ -150,6 +169,7 @@ def test_channel_note_lands_once_on_the_first_image():
 
 if __name__ == "__main__":
     test_shapes()
+    test_openai_chat_images_are_rewritten()
     test_parallel()
     test_failure_is_not_forwarded()
     test_failure_reason_is_included()

tests/test_vision_client.py

@@ -22,10 +22,12 @@ class Handler(BaseHTTPRequestHandler):
     calls = 0
     last_body = b""
     last_headers = {}
+    last_path = ""
 
     def do_POST(self):
         Handler.calls += 1
         Handler.last_headers = dict(self.headers)
+        Handler.last_path = self.path
         length = int(self.headers.get("Content-Length", 0))
         Handler.last_body = self.rfile.read(length)
         status = Handler.statuses.pop(0)
@@ -67,8 +69,12 @@ def main():
     environment = dict(os.environ, VISION_API_KEY="test-key",
                        VISION_BASE_URL=f"http://127.0.0.1:{server.server_port}/v1",
                        VISION_MODEL="fixture-model")
+    environment.pop("VISION_API_PROTOCOL", None)
+    environment.pop("VISION_REASONING_EFFORT", None)
     environment.pop("VISION_USER_AGENT", None)
     saved = dict(os.environ)
+    os.environ.pop("VISION_API_PROTOCOL", None)
+    os.environ.pop("VISION_REASONING_EFFORT", None)
     os.environ.pop("VISION_USER_AGENT", None)
     os.environ.update(environment)
     try:
@@ -134,7 +140,8 @@ def fail_with_secret(*_args, **_kwargs):
             vision_client.describe_image("data:image/png;base64,AAAA")
         finally:
             os.environ.pop("LANG", None)
-        text = json.loads(Handler.last_body)["messages"][0]["content"][0]["text"]
+        parts = json.loads(Handler.last_body)["messages"][0]["content"]
+        text = next(part["text"] for part in parts if part.get("type") == "text")
         assert text.startswith("Please respond in English.")
         assert Handler.calls == 1
 
@@ -144,25 +151,70 @@ def fail_with_secret(*_args, **_kwargs):
             vision_client.describe_image("data:image/png;base64,AAAA")
         finally:
             os.environ.pop("LANG", None)
-        text = json.loads(Handler.last_body)["messages"][0]["content"][0]["text"]
+        parts = json.loads(Handler.last_body)["messages"][0]["content"]
+        text = next(part["text"] for part in parts if part.get("type") == "text")
         assert text.startswith("请使用简体中文回答。")
         assert Handler.calls == 1
 
         Handler.calls, Handler.statuses, Handler.bodies = 0, [200], []
         os.environ.pop("LANG", None)
         vision_client.describe_image("data:image/png;base64,AAAA")
-        text = json.loads(Handler.last_body)["messages"][0]["content"][0]["text"]
+        parts = json.loads(Handler.last_body)["messages"][0]["content"]
+        text = next(part["text"] for part in parts if part.get("type") == "text")
         assert "Please respond in English." not in text
         assert "请使用简体中文回答。" not in text
         assert Handler.calls == 1
 
         Handler.calls, Handler.statuses, Handler.bodies = 0, [200], []
         vision_client.describe_image(["data:image/png;base64,AAAA", "data:image/png;base64,BBBB"])
         content = json.loads(Handler.last_body)["messages"][0]["content"]
+        assert content[0].get("type") == "image_url", \
+            "vision payloads must put image parts before text for OpenCode Go MiMo compatibility"
+        assert Handler.last_headers.get("User-Agent") == vision_client.DEFAULT_USER_AGENT
         image_parts = [part for part in content if part.get("type") == "image_url"]
         assert len(image_parts) == 2, "a list of URLs must become one request with all images"
         assert Handler.calls == 1
 
+        Handler.calls, Handler.statuses, Handler.bodies = 0, [200], [json.dumps({
+            "object": "response",
+            "output": [{
+                "type": "message",
+                "content": [{"type": "output_text", "text": "responses fixture answer"}],
+            }],
+        }).encode()]
+        os.environ["VISION_API_PROTOCOL"] = "responses"
+        os.environ["VISION_REASONING_EFFORT"] = "medium"
+        try:
+            assert vision_client.describe_image(
+                ["data:image/png;base64,AAAA", "data:image/png;base64,BBBB"],
+                prompt="read both images",
+                max_tokens=123,
+            ) == "responses fixture answer"
+        finally:
+            os.environ.pop("VISION_API_PROTOCOL", None)
+            os.environ.pop("VISION_REASONING_EFFORT", None)
+        assert Handler.last_path == "/v1/responses"
+        payload = json.loads(Handler.last_body)
+        content = payload["input"][0]["content"]
+        assert [part["type"] for part in content] == ["input_image", "input_image", "input_text"]
+        assert payload["store"] is False
+        assert payload["max_output_tokens"] == 123
+        assert payload["reasoning"] == {"effort": "medium"}
+        assert Handler.calls == 1
+
+        Handler.calls, Handler.statuses, Handler.bodies = 0, [], []
+        os.environ["VISION_API_PROTOCOL"] = "unsupported"
+        try:
+            try:
+                vision_client.describe_image("data:image/png;base64,AAAA")
+            except vision_client.VisionError as exc:
+                assert "Unsupported VISION_API_PROTOCOL" in str(exc)
+            else:
+                raise AssertionError("an unsupported protocol must fail before making a request")
+        finally:
+            os.environ.pop("VISION_API_PROTOCOL", None)
+        assert Handler.calls == 0
+
         Handler.calls, Handler.statuses, Handler.bodies = 0, [200], []
         with tempfile.TemporaryDirectory() as raw:
             image = Path(raw) / "fixture.png"
@@ -174,6 +226,7 @@ def fail_with_secret(*_args, **_kwargs):
                 "VISION_API_KEY=test-key\n"
                 f"VISION_BASE_URL=http://127.0.0.1:{server.server_port}/v1\n"
                 "VISION_MODEL=fixture-model\n"
+                "VISION_API_PROTOCOL=chat_completions\n"
             )
             isolated_env = dict(environment, HOME=raw)
             glance = Path(__file__).resolve().parent.parent / "bin/glance"

vision_client.py

@@ -98,6 +98,20 @@ def _message_text(message: object) -> str:
     return ""
 
 
+def _responses_text(response: object) -> str:
+    if not isinstance(response, dict) or not isinstance(response.get("output"), list):
+        return ""
+    return "\n".join(
+        part["text"]
+        for item in response["output"]
+        if isinstance(item, dict) and item.get("type") == "message"
+        and isinstance(item.get("content"), list)
+        for part in item["content"]
+        if isinstance(part, dict) and part.get("type") == "output_text"
+        and isinstance(part.get("text"), str)
+    ).strip()
+
+
 def _redact(text: str, *secrets: str) -> str:
     for secret in secrets:
         if secret:
@@ -123,16 +137,40 @@ def describe_image(image_url: str | list[str], prompt: str | None = None, max_to
         instruction = LANG_INSTRUCTIONS.get(os.environ.get("LANG", "").strip().lower())
         if instruction:
             text = f"{instruction}\n\n{text}"
-    payload = {
-        "model": _required("VISION_MODEL"),
-        "messages": [{"role": "user", "content": [{"type": "text", "text": text}] + [
+    model = _required("VISION_MODEL")
+    protocol = os.environ.get("VISION_API_PROTOCOL", "").strip().lower() or "chat_completions"
+    if protocol == "responses":
+        payload = {
+            "model": model,
+            "store": False,
+            "input": [{"role": "user", "content": [
+                {"type": "input_image", "image_url": url} for url in urls
+            ] + [{"type": "input_text", "text": text}]}],
+        }
+        if max_tokens is not None:
+            payload["max_output_tokens"] = max_tokens
+        reasoning_effort = os.environ.get("VISION_REASONING_EFFORT", "").strip()
+        if reasoning_effort:
+            payload["reasoning"] = {"effort": reasoning_effort}
+        endpoint = "/responses"
+        extract_text = _responses_text
+    elif protocol == "chat_completions":
+        payload = {
+            "model": model,
+            "messages": [{"role": "user", "content": [
             {"type": "image_url", "image_url": {"url": url}} for url in urls
-        ]}],
-    }
-    if max_tokens is not None:
-        payload["max_tokens"] = max_tokens
+            ] + [{"type": "text", "text": text}]}],
+        }
+        if max_tokens is not None:
+            payload["max_tokens"] = max_tokens
+        endpoint = "/chat/completions"
+        extract_text = lambda data: _message_text(data["choices"][0]["message"]["content"])
+    else:
+        raise VisionError(
+            "Unsupported VISION_API_PROTOCOL; use chat_completions or responses"
+        )
     request = urllib.request.Request(
-        base_url + "/chat/completions",
+        base_url + endpoint,
         data=json.dumps(payload).encode(),
         headers={
             "Content-Type": "application/json",
@@ -147,7 +185,7 @@ def describe_image(image_url: str | list[str], prompt: str | None = None, max_to
             with urllib.request.urlopen(request, timeout=timeout) as response:
                 data = json.load(response)
             try:
-                text = _message_text(data["choices"][0]["message"]["content"])
+                text = extract_text(data)
             except (KeyError, IndexError, TypeError) as exc:
                 raise VisionError("Vision API returned an incompatible response structure") from exc
             if not text: