diff --git a/.env.example b/.env.example
index 432f372..d80f10e 100644
--- a/.env.example
+++ b/.env.example
@@ -1,6 +1,7 @@
# Only the vision API is configured here. DeepSeek auth is still sent by Codex and passed through by the proxy.
VISION_API_KEY=
-# Any OpenAI-compatible endpoint that supports /chat/completions with image_url works.
+# Any OpenAI-compatible endpoint that supports /chat/completions with image_url works;
+# the Python client/proxy can also use /responses with input_image (see below).
# Recommended options (see README):
# OpenRouter: https://openrouter.ai/api/v1
# Aliyun DashScope: https://dashscope.aliyuncs.com/compatible-mode/v1
@@ -9,6 +10,11 @@ VISION_BASE_URL=https://openrouter.ai/api/v1
VISION_MODEL=google/gemini-3.6-flash
# Vision model output language: zh=Chinese, en=English (defaults to Chinese when unset)
LANG=zh
+# Python client/proxy protocol: chat_completions (default) or responses. Choose
+# "responses" for models exposed only through the Responses API.
+# VISION_API_PROTOCOL=chat_completions
+# Reasoning effort sent with the responses protocol (optional).
+# VISION_REASONING_EFFORT=medium
# Optional outbound User-Agent override. The default is browser-compatible to avoid
# gateways that block Python-urllib clients.
# VISION_USER_AGENT=custom-vision-client/1.0
diff --git a/AGENT_INSTALL.md b/AGENT_INSTALL.md
index dd27980..1c1cea5 100644
--- a/AGENT_INSTALL.md
+++ b/AGENT_INSTALL.md
@@ -27,7 +27,7 @@
- 已接入纯文本模型并能正常对话的宿主（Codex 或 Claude Code）
- Python 3.11+
-- 一个支持 `/chat/completions` 和 `image_url` 的 OpenAI-compatible 视觉 API
+- 一个支持 `/chat/completions` 和 `image_url` 的 OpenAI-compatible 视觉 API；Python 客户端/代理也可配置 `/responses` + `input_image`
## 1. 定位并备份现有配置
@@ -58,6 +58,8 @@ VISION_API_KEY=...
VISION_BASE_URL=...
VISION_MODEL=...
LANG=zh # 可选：视觉模型输出语言（zh/en），不填保持默认中文
+# VISION_API_PROTOCOL=chat_completions # 可选：Python 客户端/代理可改用 responses 协议
+# VISION_REASONING_EFFORT=medium # 可选：responses 协议下的推理强度
# VISION_USER_AGENT=custom-vision-client/1.0 # 可选：覆盖默认的浏览器兼容 User-Agent
```
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ea8348b..0c63ee2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,11 @@ All notable user-facing changes to agent-vision-toolkit are documented in this f
## [Unreleased]
+### Added
+
+- Let the shared Python vision client call either Chat Completions or Responses APIs, including optional reasoning effort and explicit `store: false` data handling.
+- Rewrite OpenAI Chat Completions `image_url` blocks through the existing vision-description pipeline with a host-neutral channel note.
+
### Fixed
- Send a browser-compatible, configurable User-Agent from the shared Python vision client so Cloudflare-backed OpenAI-compatible endpoints do not reject the default `Python-urllib` signature.
diff --git a/README.md b/README.md
index 1d1d8c0..dd1b1a5 100644
--- a/README.md
+++ b/README.md
@@ -151,7 +151,7 @@ VISION_BASE_URL=https://openrouter.ai/api/v1
VISION_MODEL=google/gemini-3.6-flash
```
-Any OpenAI-compatible endpoint that supports `/chat/completions` with `image_url` works (e.g. Aliyun DashScope: `https://dashscope.aliyuncs.com/compatible-mode/v1` + `qwen-vl-max-latest`). Add `LANG=en` for English descriptions (default is Chinese).
+Any OpenAI-compatible endpoint that supports `/chat/completions` with `image_url` works (e.g. Aliyun DashScope: `https://dashscope.aliyuncs.com/compatible-mode/v1` + `qwen-vl-max-latest`). The Python client/proxy can also use `/responses` with `input_image` by setting `VISION_API_PROTOCOL=responses`. Add `LANG=en` for English descriptions (default is Chinese).
**2. Put the CLIs on your PATH:**
@@ -322,7 +322,7 @@ Codex -> 127.0.0.1:19100 -> your existing text-only upstream
Environment variables
-The toolkit and proxy use only these environment variables; just three are required:
+The standalone CLIs and Python proxy use these environment variables; just three are required. The native Pi and OpenCode extensions use their own settings and currently call `/chat/completions` only.
| Variable | Required | Description |
|---|---:|---|
@@ -330,6 +330,8 @@ The toolkit and proxy use only these environment variables; just three are requi
| `VISION_BASE_URL` | Yes | OpenAI-compatible API base URL |
| `VISION_MODEL` | Yes | Multimodal model name |
| `LANG` | No | Vision model output language: `zh` (Chinese) or `en` (English); default `zh` |
+| `VISION_API_PROTOCOL` | No | Python client/proxy protocol: `chat_completions` (default) or `responses` |
+| `VISION_REASONING_EFFORT` | No | Optional provider-supported reasoning effort for the Python client/proxy when using `responses` |
| `VISION_USER_AGENT` | No | Outbound User-Agent for the Python client/proxy; defaults to a browser-compatible value and can be overridden for provider requirements |
@@ -349,7 +351,7 @@ The route whose connection (TCP/TLS handshake) succeeds is kept in memory and re
## Prerequisites
- A coding agent already working with a model, including a text-only model such as DeepSeek V4
-- An OpenAI-compatible vision API that supports `/chat/completions` and `image_url`
+- An OpenAI-compatible vision API that supports `/chat/completions` and `image_url`; the Python client/proxy can also use `/responses` with `input_image` via `VISION_API_PROTOCOL=responses`
- No other configuration is required
## FAQ
diff --git a/README_CN.md b/README_CN.md
index 971b072..9d87415 100644
--- a/README_CN.md
+++ b/README_CN.md
@@ -151,7 +151,7 @@ VISION_BASE_URL=https://openrouter.ai/api/v1
VISION_MODEL=google/gemini-3.6-flash
```
-任何支持 `/chat/completions` 与 `image_url` 的 OpenAI-compatible 端点都可以（如阿里云百炼：`https://dashscope.aliyuncs.com/compatible-mode/v1` + `qwen-vl-max-latest`）。需要英文描述时加 `LANG=en`（默认中文）。
+任何支持 `/chat/completions` 与 `image_url` 的 OpenAI-compatible 端点都可以（如阿里云百炼：`https://dashscope.aliyuncs.com/compatible-mode/v1` + `qwen-vl-max-latest`）。Python 客户端/代理也可设置 `VISION_API_PROTOCOL=responses` 使用 `/responses` + `input_image`。需要英文描述时加 `LANG=en`（默认中文）。
**2. 把 CLI 放进 PATH：**
@@ -319,7 +319,7 @@ Codex -> 127.0.0.1:19100 -> 用户原有的纯文本模型上游
环境变量
-工具箱与代理都只需要这些环境变量，必填的只有3个：
+独立 CLI 和 Python 代理使用这些环境变量，必填的只有 3 个。Pi 与 OpenCode 原生扩展使用各自设置，目前仅调用 `/chat/completions`。
| 变量 | 必需 | 说明 |
|---|---:|---|
@@ -327,6 +327,8 @@ Codex -> 127.0.0.1:19100 -> 用户原有的纯文本模型上游
| `VISION_BASE_URL` | 是 | OpenAI-compatible API 地址 |
| `VISION_MODEL` | 是 | 多模态模型名 |
| `LANG` | 否 | 视觉模型输出语言：`zh`=中文，`en`=English（默认 `zh`） |
+| `VISION_API_PROTOCOL` | 否 | Python 客户端/代理的视觉 API 协议：`chat_completions`（默认）或 `responses` |
+| `VISION_REASONING_EFFORT` | 否 | Python 客户端/代理使用 `responses` 时可选的服务商支持推理强度 |
| `VISION_USER_AGENT` | 否 | Python 客户端/代理的出站 User-Agent；默认使用浏览器兼容值，也可按服务商要求覆盖 |
@@ -346,7 +348,7 @@ Codex -> 127.0.0.1:19100 -> 用户原有的纯文本模型上游
## 前置条件
- 已接入(纯文本)模型（如 DeepSeek V4）并可正常使用的 coding agent
-- 一个支持 `/chat/completions` 与 `image_url` 的 OpenAI-compatible 视觉 API
+- 一个支持 `/chat/completions` 与 `image_url` 的 OpenAI-compatible 视觉 API；Python 客户端/代理也可通过 `VISION_API_PROTOCOL=responses` 使用 `/responses` + `input_image`
- 没有其他需要的配置
## 常见问题
diff --git a/skills/vision-tools/SKILL.md b/skills/vision-tools/SKILL.md
index a45cf3d..8f42552 100644
--- a/skills/vision-tools/SKILL.md
+++ b/skills/vision-tools/SKILL.md
@@ -15,7 +15,8 @@ description: >-
Five local CLIs that give a text-only agent eyes. They read one shared
vision config (`VISION_API_KEY` / `VISION_BASE_URL` / `VISION_MODEL` /
-`LANG`) — no extra credentials.
+`LANG`), plus the optional Python-client settings `VISION_API_PROTOCOL`,
+`VISION_REASONING_EFFORT`, and `VISION_USER_AGENT` — no extra credentials.
Pick the tool by the question you are answering:
diff --git a/tests/test_image_rewrite_shapes.py b/tests/test_image_rewrite_shapes.py
index 38ee2ab..88ce4fb 100644
--- a/tests/test_image_rewrite_shapes.py
+++ b/tests/test_image_rewrite_shapes.py
@@ -60,6 +60,25 @@ def test_shapes():
print("PASS: shapes A (content) and B (output) rewritten; text-only untouched")
+def test_openai_chat_images_are_rewritten():
+ mod = _load_proxy()
+ body = {"model": "deepseek-v4-flash", "messages": [{
+ "role": "user",
+ "content": [
+ {"type": "text", "text": "请读出图片里的标题"},
+ {"type": "image_url", "image_url": {"url": "data:image/png;base64,AAAA"}},
+ ],
+ }]}
+ assert asyncio.run(mod._rewrite_image_inputs(body))
+ content = body["messages"][0]["content"]
+ texts = [block["text"] for block in content if block.get("type") == "text"]
+ note = next(text for text in texts if text.startswith("[vision proxy]"))
+ assert "view_image" not in note, note
+ assert "submit the image again" in note, note
+ assert any("TEST-DESC" in text for text in texts), content
+ print("PASS: OpenAI Chat Completions image_url rewritten")
+
+
def test_parallel():
mod = _load_proxy()
real_desc = mod._image_desc_from_url
@@ -150,6 +169,7 @@ def test_channel_note_lands_once_on_the_first_image():
if __name__ == "__main__":
test_shapes()
+ test_openai_chat_images_are_rewritten()
test_parallel()
test_failure_is_not_forwarded()
test_failure_reason_is_included()
diff --git a/tests/test_vision_client.py b/tests/test_vision_client.py
index d76ff65..658de7b 100644
--- a/tests/test_vision_client.py
+++ b/tests/test_vision_client.py
@@ -22,10 +22,12 @@ class Handler(BaseHTTPRequestHandler):
calls = 0
last_body = b""
last_headers = {}
+ last_path = ""
def do_POST(self):
Handler.calls += 1
Handler.last_headers = dict(self.headers)
+ Handler.last_path = self.path
length = int(self.headers.get("Content-Length", 0))
Handler.last_body = self.rfile.read(length)
status = Handler.statuses.pop(0)
@@ -67,8 +69,12 @@ def main():
environment = dict(os.environ, VISION_API_KEY="test-key",
VISION_BASE_URL=f"http://127.0.0.1:{server.server_port}/v1",
VISION_MODEL="fixture-model")
+ environment.pop("VISION_API_PROTOCOL", None)
+ environment.pop("VISION_REASONING_EFFORT", None)
environment.pop("VISION_USER_AGENT", None)
saved = dict(os.environ)
+ os.environ.pop("VISION_API_PROTOCOL", None)
+ os.environ.pop("VISION_REASONING_EFFORT", None)
os.environ.pop("VISION_USER_AGENT", None)
os.environ.update(environment)
try:
@@ -134,7 +140,8 @@ def fail_with_secret(*_args, **_kwargs):
vision_client.describe_image("data:image/png;base64,AAAA")
finally:
os.environ.pop("LANG", None)
- text = json.loads(Handler.last_body)["messages"][0]["content"][0]["text"]
+ parts = json.loads(Handler.last_body)["messages"][0]["content"]
+ text = next(part["text"] for part in parts if part.get("type") == "text")
assert text.startswith("Please respond in English.")
assert Handler.calls == 1
@@ -144,14 +151,16 @@ def fail_with_secret(*_args, **_kwargs):
vision_client.describe_image("data:image/png;base64,AAAA")
finally:
os.environ.pop("LANG", None)
- text = json.loads(Handler.last_body)["messages"][0]["content"][0]["text"]
+ parts = json.loads(Handler.last_body)["messages"][0]["content"]
+ text = next(part["text"] for part in parts if part.get("type") == "text")
assert text.startswith("请使用简体中文回答。")
assert Handler.calls == 1
Handler.calls, Handler.statuses, Handler.bodies = 0, [200], []
os.environ.pop("LANG", None)
vision_client.describe_image("data:image/png;base64,AAAA")
- text = json.loads(Handler.last_body)["messages"][0]["content"][0]["text"]
+ parts = json.loads(Handler.last_body)["messages"][0]["content"]
+ text = next(part["text"] for part in parts if part.get("type") == "text")
assert "Please respond in English." not in text
assert "请使用简体中文回答。" not in text
assert Handler.calls == 1
@@ -159,10 +168,53 @@ def fail_with_secret(*_args, **_kwargs):
Handler.calls, Handler.statuses, Handler.bodies = 0, [200], []
vision_client.describe_image(["data:image/png;base64,AAAA", "data:image/png;base64,BBBB"])
content = json.loads(Handler.last_body)["messages"][0]["content"]
+ assert content[0].get("type") == "image_url", \
+ "vision payloads must put image parts before text for OpenCode Go MiMo compatibility"
+ assert Handler.last_headers.get("User-Agent") == vision_client.DEFAULT_USER_AGENT
image_parts = [part for part in content if part.get("type") == "image_url"]
assert len(image_parts) == 2, "a list of URLs must become one request with all images"
assert Handler.calls == 1
+ Handler.calls, Handler.statuses, Handler.bodies = 0, [200], [json.dumps({
+ "object": "response",
+ "output": [{
+ "type": "message",
+ "content": [{"type": "output_text", "text": "responses fixture answer"}],
+ }],
+ }).encode()]
+ os.environ["VISION_API_PROTOCOL"] = "responses"
+ os.environ["VISION_REASONING_EFFORT"] = "medium"
+ try:
+ assert vision_client.describe_image(
+ ["data:image/png;base64,AAAA", "data:image/png;base64,BBBB"],
+ prompt="read both images",
+ max_tokens=123,
+ ) == "responses fixture answer"
+ finally:
+ os.environ.pop("VISION_API_PROTOCOL", None)
+ os.environ.pop("VISION_REASONING_EFFORT", None)
+ assert Handler.last_path == "/v1/responses"
+ payload = json.loads(Handler.last_body)
+ content = payload["input"][0]["content"]
+ assert [part["type"] for part in content] == ["input_image", "input_image", "input_text"]
+ assert payload["store"] is False
+ assert payload["max_output_tokens"] == 123
+ assert payload["reasoning"] == {"effort": "medium"}
+ assert Handler.calls == 1
+
+ Handler.calls, Handler.statuses, Handler.bodies = 0, [], []
+ os.environ["VISION_API_PROTOCOL"] = "unsupported"
+ try:
+ try:
+ vision_client.describe_image("data:image/png;base64,AAAA")
+ except vision_client.VisionError as exc:
+ assert "Unsupported VISION_API_PROTOCOL" in str(exc)
+ else:
+ raise AssertionError("an unsupported protocol must fail before making a request")
+ finally:
+ os.environ.pop("VISION_API_PROTOCOL", None)
+ assert Handler.calls == 0
+
Handler.calls, Handler.statuses, Handler.bodies = 0, [200], []
with tempfile.TemporaryDirectory() as raw:
image = Path(raw) / "fixture.png"
@@ -174,6 +226,7 @@ def fail_with_secret(*_args, **_kwargs):
"VISION_API_KEY=test-key\n"
f"VISION_BASE_URL=http://127.0.0.1:{server.server_port}/v1\n"
"VISION_MODEL=fixture-model\n"
+ "VISION_API_PROTOCOL=chat_completions\n"
)
isolated_env = dict(environment, HOME=raw)
glance = Path(__file__).resolve().parent.parent / "bin/glance"
diff --git a/vision_client.py b/vision_client.py
index 995cca3..3a829cf 100644
--- a/vision_client.py
+++ b/vision_client.py
@@ -98,6 +98,20 @@ def _message_text(message: object) -> str:
return ""
+def _responses_text(response: object) -> str:
+ if not isinstance(response, dict) or not isinstance(response.get("output"), list):
+ return ""
+ return "\n".join(
+ part["text"]
+ for item in response["output"]
+ if isinstance(item, dict) and item.get("type") == "message"
+ and isinstance(item.get("content"), list)
+ for part in item["content"]
+ if isinstance(part, dict) and part.get("type") == "output_text"
+ and isinstance(part.get("text"), str)
+ ).strip()
+
+
def _redact(text: str, *secrets: str) -> str:
for secret in secrets:
if secret:
@@ -123,16 +137,40 @@ def describe_image(image_url: str | list[str], prompt: str | None = None, max_to
instruction = LANG_INSTRUCTIONS.get(os.environ.get("LANG", "").strip().lower())
if instruction:
text = f"{instruction}\n\n{text}"
- payload = {
- "model": _required("VISION_MODEL"),
- "messages": [{"role": "user", "content": [{"type": "text", "text": text}] + [
+ model = _required("VISION_MODEL")
+ protocol = os.environ.get("VISION_API_PROTOCOL", "").strip().lower() or "chat_completions"
+ if protocol == "responses":
+ payload = {
+ "model": model,
+ "store": False,
+ "input": [{"role": "user", "content": [
+ {"type": "input_image", "image_url": url} for url in urls
+ ] + [{"type": "input_text", "text": text}]}],
+ }
+ if max_tokens is not None:
+ payload["max_output_tokens"] = max_tokens
+ reasoning_effort = os.environ.get("VISION_REASONING_EFFORT", "").strip()
+ if reasoning_effort:
+ payload["reasoning"] = {"effort": reasoning_effort}
+ endpoint = "/responses"
+ extract_text = _responses_text
+ elif protocol == "chat_completions":
+ payload = {
+ "model": model,
+ "messages": [{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": url}} for url in urls
- ]}],
- }
- if max_tokens is not None:
- payload["max_tokens"] = max_tokens
+ ] + [{"type": "text", "text": text}]}],
+ }
+ if max_tokens is not None:
+ payload["max_tokens"] = max_tokens
+ endpoint = "/chat/completions"
+ extract_text = lambda data: _message_text(data["choices"][0]["message"]["content"])
+ else:
+ raise VisionError(
+ "Unsupported VISION_API_PROTOCOL; use chat_completions or responses"
+ )
request = urllib.request.Request(
- base_url + "/chat/completions",
+ base_url + endpoint,
data=json.dumps(payload).encode(),
headers={
"Content-Type": "application/json",
@@ -147,7 +185,7 @@ def describe_image(image_url: str | list[str], prompt: str | None = None, max_to
with urllib.request.urlopen(request, timeout=timeout) as response:
data = json.load(response)
try:
- text = _message_text(data["choices"][0]["message"]["content"])
+ text = extract_text(data)
except (KeyError, IndexError, TypeError) as exc:
raise VisionError("Vision API returned an incompatible response structure") from exc
if not text:
diff --git a/vision_proxy.py b/vision_proxy.py
index dda2502..dcad25f 100755
--- a/vision_proxy.py
+++ b/vision_proxy.py
@@ -105,6 +105,14 @@ def __str__(self):
"is written to answer that."
)
+_OPENAI_CHAT_CHANNEL_NOTE = (
+ "[vision proxy] Images reach you as text here: a vision model reads each image "
+ "and writes a description — you never receive visual tokens. Each description "
+ "is written to answer the text sent with that image. Whenever a description "
+ "misses what you need, state the missing detail and submit the image again with "
+ "that specific question so the next description can focus on it."
+)
+
# Codex-injected user-role blocks that are never "the user's current request".
_INJECTED_PREFIXES = ("", "", "# AGENTS.md instructions")
@@ -326,6 +334,38 @@ def _collect_anthropic_jobs(parsed):
return jobs
+def _openai_image_url(block):
+ """Return an OpenAI Chat Completions image URL from a content block."""
+ value = block.get("image_url")
+ if isinstance(value, str):
+ return value
+ if isinstance(value, dict) and isinstance(value.get("url"), str):
+ return value["url"]
+ return None
+
+
+def _collect_openai_chat_jobs(parsed):
+ """OpenAI Chat Completions API: messages[].content[] image_url blocks."""
+ jobs = []
+ for message in parsed["messages"]:
+ if not isinstance(message, dict) or message.get("role") != "user":
+ continue
+ content = message.get("content")
+ if not isinstance(content, list):
+ continue
+ text = "\n".join(
+ block["text"] for block in content
+ if isinstance(block, dict) and block.get("type") == "text"
+ and isinstance(block.get("text"), str)
+ )
+ for index, block in enumerate(content):
+ if isinstance(block, dict) and block.get("type") == "image_url":
+ url = _openai_image_url(block)
+ if url:
+ jobs.append((content, index, url, _vision_prompt(text, "user")))
+ return jobs
+
+
def _detect_format(parsed):
if isinstance(parsed.get("input"), list):
return "responses"
@@ -342,6 +382,8 @@ def _detect_format(parsed):
if not isinstance(block, dict):
continue
kind = block.get("type")
+ if kind == "image_url":
+ return "openai_chat"
if kind == "image":
return "anthropic"
if kind == "tool_result":
@@ -355,6 +397,7 @@ def _detect_format(parsed):
_FORMATS = {
"responses": (_collect_responses_jobs, lambda text: {"type": "input_text", "text": text}, _CHANNEL_NOTE),
"anthropic": (_collect_anthropic_jobs, lambda text: {"type": "text", "text": text}, _ANTHROPIC_CHANNEL_NOTE),
+ "openai_chat": (_collect_openai_chat_jobs, lambda text: {"type": "text", "text": text}, _OPENAI_CHAT_CHANNEL_NOTE),
}