From 69902bba1d85244c167eae8274457dc8fc9b7715 Mon Sep 17 00:00:00 2001 From: Wallacy Date: Sun, 12 Jul 2026 16:23:59 -0300 Subject: [PATCH 1/4] feat: vision proxy for text-only models + context overflow safety margin MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Vision proxy (#74): new setting opencodego.visionModel lets users specify a vision-capable Copilot model (e.g. copilot:gpt-5.5). When a text-only OpenCode model receives an image, the extension transparently forwards it to the vision model, gets a text description, and feeds it to the original model. Uses vscode.lm.selectChatModels + sendRequest — no extra deps. - Context overflow fix: added 64-token safety margin to promptReserve in modelLimits() to prevent 400 errors when estimateTokenCount underestimates by 1-2 tokens on large prompts. - CHANGELOG/README/docs updated. --- CHANGELOG.md | 10 ++++++ README.md | 5 +-- package.json | 10 ++++++ src/extension.ts | 94 +++++++++++++++++++++++++++++++++++++++++++++++- 4 files changed, 116 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f5eb217..b128813 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,16 @@ All notable changes to the **OpenCode Go BYOK Provider** extension are documente - **`[Usage]` 5-hour rolling usage now isolates per profile in multi-profile mode.** The shared `opencode.db` SQLite database has no API key column, so reading it in multi-account mode mixed quota from all accounts. The tracker now skips SQLite entirely when multiple profiles exist and falls back to extension-tracked entries (which are namespaced per profile). This trades billed-accuracy for correct per-profile isolation. +### Fixed + +- **`[Streaming]` Context overflow 400 when prompt + output approached the model's limit.** The `estimateTokenCount` heuristic (client-side) can underestimate real token count by 0–2%. On large prompts (~130K tokens on a 262K window), this pushed payload 1 token over the limit, causing a 400 from the provider. Added a 64-token safety margin to `promptReserve` in `modelLimits()` — enough headroom without meaningfully reducing output budget. Affects all models (reported with GLM-5.2). + +## [Unreleased] + +### Added + +- **`[Vision]` Transparent vision proxy for text-only models (issue #74).** Configure a vision-capable model via `opencodego.visionModel` (e.g. `copilot:gpt-5.5`). When a non-vision OpenCode model receives an image, the extension automatically forwards it to the configured vision model, receives a text description, and feeds that description to the original model — so text-only models can "see" images with zero extra steps. Set `opencodego.visionPrompt` to customize the description instruction. Disabled by default (empty string). Implemented using `vscode.lm.selectChatModels` + `sendRequest` — no extra dependencies. + ## [0.3.7] — 2026-07-09 diff --git a/README.md b/README.md index 1ac67a2..3efcb68 100644 --- a/README.md +++ b/README.md @@ -34,8 +34,7 @@ | 💸 **Cheaper than Copilot Pro+** | Copilot Free + OpenCode **Zen free** models = **$0** for 2-5 rotating models (Big Pickle always free; DeepSeek V4 Flash, MiMo-V2.5, and others rotate). Paid Zen models (Claude Opus, GPT-5.5) available at pay-as-you-go rates. Go subscription **$10/mo** ($5 first month) | | 🌍 **30+ frontier models** | DeepSeek V4, Kimi K2.6, GLM-5.1, Qwen3.7 Max, MiMo V2.5, MiniMax M2.7, Big Pickle, Nemotron — **all in one picker** | | 🤖 **Full Agent Mode** | Tool-calling (read files, edit, run terminal) works natively — not just chat. Models also appear in the **Agents window** (Copilot CLI session) | -| 🧠 **Thinking controls** | Per-model reasoning effort (DeepSeek `max`, Qwen `thinking_budget`, MiniMax `on/off`, Mimo `low/med/high`) | -| 📊 **Live usage tracking** | Status bar shows Go subscription burn-rate across 5h / weekly / monthly tiers | +| 🧠 **Thinking controls** | Per-model reasoning effort (DeepSeek `max`, Qwen `thinking_budget`, MiniMax `on/off`, Mimo `low/med/high`) || 🖼️ **Vision proxy** | Text-only models can "see" images via a configured proxy model (`opencodego.visionModel`). Works with any Copilot‑compatible vision model. || 📊 **Live usage tracking** | Status bar shows Go subscription burn-rate across 5h / weekly / monthly tiers | | 🔌 **Dual providers** | OpenCode **Go** ($10/mo subscription) + OpenCode **Zen** (free + paid models) — run both at once, switch instantly | | 🎯 **Smart routing** | Each model family auto-routes to its native transport (`/responses`, `/messages`, `streamGenerateContent`, `/chat/completions`) | | 🖼️ **Vision + PDF + Audio** | Multimodal models pass through image, PDF, audio, and video inputs | @@ -336,6 +335,8 @@ To manage agent API keys separately or see agent vendors in the Manage panel, en | `opencodego.thinking.mimo` | `"off"` | `off`/`low`/`medium`/`high` | | `opencodego.thinking.qwen` | `"off"` | `auto`/`on`/`off` | | `opencodego.thinking.qwenBudget` | `"auto"` | `auto`/`4096`/`16384`/`32768`/`81920` | +| `opencodego.visionModel` | `""` | Vision proxy model ID (e.g. `copilot:gpt-5.5`). Empty = disabled. Text-only models relay images through this model automatically. | +| `opencodego.visionPrompt` | Built-in | Prompt sent to the vision model when describing images. |
📜 Full settings reference with descriptions diff --git a/package.json b/package.json index 460f48b..916823c 100644 --- a/package.json +++ b/package.json @@ -176,6 +176,16 @@ "default": "auto", "markdownDescription": "How to handle `...` tags in model output. `auto` strips tags only for known reasoning models that inline thinking in the content field (minimax), accumulating the extracted reasoning content into the existing reasoning fallback. `always` strips for every model. `never` passes text through unchanged (use if a model legitimately uses `` in its output)." }, + "opencodego.visionModel": { + "type": "string", + "default": "", + "markdownDescription": "Vision proxy model. When set, images pasted into chat are automatically sent to this model for description before being passed to text-only models. The value must be a model `id` from the VS Code model picker (e.g. `copilot:gpt-5.5`). Leave empty to disable vision proxy." + }, + "opencodego.visionPrompt": { + "type": "string", + "default": "Describe this image in detail so a text-only model can understand what it shows. Include all visible text, layout, colors, objects, and context.", + "markdownDescription": "Prompt sent to the vision proxy model when an image is attached. The image is appended after this text." + }, "opencodego.thinking.deepseek": { "type": "string", "enum": [ diff --git a/src/extension.ts b/src/extension.ts index b6c01b6..426d7c9 100644 --- a/src/extension.ts +++ b/src/extension.ts @@ -402,6 +402,10 @@ interface ApiSettings { streamIdleTimeoutMs: number; thinking: ThinkingSettings; stripThinkTags: "never" | "auto" | "always"; + /** Vision proxy model ID (e.g. "copilot:gpt-5.5"). Empty = disabled. */ + visionModel: string; + /** Custom prompt for the vision proxy, e.g. "Describe this image in detail". */ + visionPrompt: string; } interface LanguageModelConfiguration { @@ -1805,7 +1809,39 @@ class OpenCodeProvider implements vscode.LanguageModelChatProvider p.type === "image_url")) { + const textParts = msg.content + .filter((p): p is OpenAiContentPart & { text: string } => p.type === "text" && typeof p.text === "string") + .map(p => p.text); + msg.content = [{ type: "text", text: `[Image described by ${settings.visionModel}]: ${description}` }]; + if (textParts.length > 0) { + msg.content.push({ type: "text", text: textParts.join("\n") }); + } + } + } + this.log(`[vision-proxy] Replaced images using ${settings.visionModel}`); + } + } catch (err) { + this.log(`[vision-proxy] Error: ${err instanceof Error ? err.message : String(err)}`); + } + } + + const thinkingPayload = buildThinkingPayload(rawModelId, settings.thinking, hasImageInput && metadata.supportsVision); const requestHeaders = buildOpenCodeRequestHeaders( messages, options, @@ -3137,6 +3173,8 @@ function getSettings(): ApiSettings { mimo: config.get("thinking.mimo", "off"), }, stripThinkTags: config.get("stripThinkTags", "auto"), + visionModel: config.get("visionModel", ""), + visionPrompt: config.get("visionPrompt", ""), }; } @@ -3269,6 +3307,60 @@ function isFreeModel(modelId: string, vendor: ProviderDefinition["vendor"]): boo ); } +/** + * Vision proxy: relay image messages through a vision-capable Copilot model + * and return the text description. This lets text-only models "see" images + * transparently (issue #74). + */ +async function proxyVision( + messages: readonly vscode.LanguageModelChatRequestMessage[], + visionModelId: string, + visionPrompt: string, +): Promise { + const visionModels = await vscode.lm.selectChatModels({ id: visionModelId }); + if (!visionModels || visionModels.length === 0) { + throw new Error(`Vision model "${visionModelId}" not found. Check your opencodego.visionModel setting.`); + } + const visionModel = visionModels[0]; + + // Build a request with the images + prompt. The messages already contain + // LanguageModelDataPart for images — we pass them through as-is. + const requestMessages: vscode.LanguageModelChatMessage[] = []; + for (const msg of messages) { + let content: Array = []; + for (const part of msg.content) { + if (part instanceof vscode.LanguageModelDataPart && part.mimeType.startsWith("image/")) { + content.push(part); + } else if (part instanceof vscode.LanguageModelTextPart) { + content.push(part); + } else if (typeof part === "object" && part !== null && "value" in part) { + content.push(new vscode.LanguageModelTextPart(String((part as any).value))); + } + } + if (content.length > 0) { + requestMessages.push(new vscode.LanguageModelChatMessage(msg.role === 1 ? vscode.LanguageModelChatMessageRole.Assistant : vscode.LanguageModelChatMessageRole.User, content)); + } + } + + // Add the vision prompt as a user message + if (visionPrompt) { + requestMessages.push(vscode.LanguageModelChatMessage.User(visionPrompt)); + } + + let fullDescription = ""; + // VS Code's sendRequest requires a CancellationToken. We create a + // simple one that never cancels. + const cancellationToken: vscode.CancellationToken = { + isCancellationRequested: false, + onCancellationRequested: () => ({ dispose: () => {} }), + }; + const response = await visionModel.sendRequest(requestMessages, {}, cancellationToken); + for await (const part of response.text) { + fullDescription += part; + } + return fullDescription.length > 0 ? fullDescription : undefined; +} + /** * Returns pricing fields for VS Code's language model pricing proposal * (`vscode.proposed.languageModelPricing`). From 4a360095653e1af49c98c1b0c9ccbba1156292cb Mon Sep 17 00:00:00 2001 From: Wallacy Date: Sun, 12 Jul 2026 16:35:22 -0300 Subject: [PATCH 2/4] feat: vision proxy for text-only models + context overflow safety margin (#74) - Vision proxy: new opencodego.visionModel setting and 'OpenCode Go: Configure Vision Proxy' command to pick from available vision-capable models. When a text-only model receives an image, the extension forwards it to the vision model, gets a description, and feeds it to the original model. Model lookup is flexible: accepts full model IDs, vendor:name, or name substring. - Context overflow fix: added 64-token safety margin to promptReserve in modelLimits() to prevent 400 errors when estimateTokenCount underestimates by 1-2 tokens on large prompts (reported with GLM-5.2). - CHANGELOG/README/docs updated. --- CHANGELOG.md | 24 ++-- README.md | 5 +- package.json | 17 ++- src/extension.ts | 313 ++++++++++++++++++++++++++++++++++++++++++----- src/metadata.ts | 2 +- src/streaming.ts | 3 +- 6 files changed, 303 insertions(+), 61 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b128813..5703550 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,31 +2,29 @@ All notable changes to the **OpenCode Go BYOK Provider** extension are documented here. -## [0.4.0] — 2026-07-13 +## [Unreleased] ### Added -- **`[Usage]` Per-profile Go usage tracking for multi-account setups.** Users with multiple OpenCode Go subscriptions (e.g. work + personal) can now add separate Go entries in the Manage Language Models panel. The extension auto-creates a named profile ("Profile 1", "Profile 2", etc.) on the first request with each key. Storage is namespaced per profile so usage data never mixes. Fixes [#63](https://github.com/ltmoerdani/opencode-copilot-chat/issues/63). PR [#75](https://github.com/ltmoerdani/opencode-copilot-chat/pull/75) by [@Wallacy](https://github.com/Wallacy). -- **`[Usage]` Profile auto-switch and QuickPick.** The active profile follows the last used model. Click the status bar to open a QuickPick listing all profiles; the active one is checked, others are clickable switches. The SVG hover card and status bar show the active profile label (e.g. `Go: 5%·12%·8% [Profile 2]`). -- **`[Usage]` Profile management commands.** `OpenCode Go: Rename Active Profile` and `OpenCode Go: Delete Profile` let users manage their profiles. Delete includes a confirmation dialog and cleans up all namespaced storage. -- **`[Usage]` Legacy data migration.** Users upgrading from a single-account install get their existing usage data migrated into Profile 1 automatically on first multi-profile activation. Nothing is lost. - -### Fixed - -- **`[Usage]` 5-hour rolling usage now isolates per profile in multi-profile mode.** The shared `opencode.db` SQLite database has no API key column, so reading it in multi-account mode mixed quota from all accounts. The tracker now skips SQLite entirely when multiple profiles exist and falls back to extension-tracked entries (which are namespaced per profile). This trades billed-accuracy for correct per-profile isolation. +- **`[Vision]` Transparent vision proxy for text-only models (#74).** Enable `opencodego.visionProxy` in settings, then run **OpenCode Go: Configure Vision Proxy** from the Command Palette to pick a vision-capable model. When a non-vision OpenCode model receives an image, the extension forwards it to a configured vision model, receives a text description, and feeds that to the original model — so text-only models "see" images with zero extra steps. The picker shows only vision-capable models, with a "None" option to disable. A **Customize prompt** entry in the picker lets you edit the description instruction. Model ID and prompt are stored in extension state, set exclusively via the command. Implemented with `vscode.lm.selectChatModels` + `sendRequest`. ### Fixed -- **`[Streaming]` Context overflow 400 when prompt + output approached the model's limit.** The `estimateTokenCount` heuristic (client-side) can underestimate real token count by 0–2%. On large prompts (~130K tokens on a 262K window), this pushed payload 1 token over the limit, causing a 400 from the provider. Added a 64-token safety margin to `promptReserve` in `modelLimits()` — enough headroom without meaningfully reducing output budget. Affects all models (reported with GLM-5.2). +- **`[Streaming]` Context overflow 400 when prompt + output approached the model's limit.** `estimateTokenCount` can underestimate by 0–2%, pushing payloads past the context window on large prompts. Added a 64-token safety margin to `promptReserve` in `modelLimits()`. Affects all models (reported with GLM-5.2). +- **`[Streaming]` Empty response warning no longer steals focus to the Output pane (#67).** Removed the stray `.show(true)` call on the empty-response diagnostic log. -## [Unreleased] +## [0.4.0] — 2026-07-13 ### Added -- **`[Vision]` Transparent vision proxy for text-only models (issue #74).** Configure a vision-capable model via `opencodego.visionModel` (e.g. `copilot:gpt-5.5`). When a non-vision OpenCode model receives an image, the extension automatically forwards it to the configured vision model, receives a text description, and feeds that description to the original model — so text-only models can "see" images with zero extra steps. Set `opencodego.visionPrompt` to customize the description instruction. Disabled by default (empty string). Implemented using `vscode.lm.selectChatModels` + `sendRequest` — no extra dependencies. +- **`[Usage]` Per-profile Go usage tracking for multi-account setups.** Users with multiple OpenCode Go subscriptions (e.g. work + personal) can now add separate Go entries in the Manage Language Models panel. The extension auto-creates a named profile ("Profile 1", "Profile 2", etc.) on the first request with each key. Storage is namespaced per profile so usage data never mixes. Fixes [#63](https://github.com/ltmoerdani/opencode-copilot-chat/issues/63). PR [#75](https://github.com/ltmoerdani/opencode-copilot-chat/pull/75) by [@Wallacy](https://github.com/Wallacy). +- **`[Usage]` Profile auto-switch and QuickPick.** The active profile follows the last used model. Click the status bar to open a QuickPick listing all profiles; the active one is checked, others are clickable switches. The SVG hover card and status bar show the active profile label (e.g. `Go: 5%·12%·8% [Profile 2]`). +- **`[Usage]` Profile management commands.** `OpenCode Go: Rename Active Profile` and `OpenCode Go: Delete Profile` let users manage their profiles. Delete includes a confirmation dialog and cleans up all namespaced storage. +- **`[Usage]` Legacy data migration.** Users upgrading from a single-account install get their existing usage data migrated into Profile 1 automatically on first multi-profile activation. Nothing is lost. +### Fixed -## [0.3.7] — 2026-07-09 +- **`[Usage]` 5-hour rolling usage now isolates per profile in multi-profile mode.** The shared `opencode.db` SQLite database has no API key column, so reading it in multi-account mode mixed quota from all accounts. The tracker now skips SQLite entirely when multiple profiles exist and falls back to extension-tracked entries (which are namespaced per profile). This trades billed-accuracy for correct per-profile isolation. ### Added diff --git a/README.md b/README.md index 3efcb68..6c0daf3 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ | 💸 **Cheaper than Copilot Pro+** | Copilot Free + OpenCode **Zen free** models = **$0** for 2-5 rotating models (Big Pickle always free; DeepSeek V4 Flash, MiMo-V2.5, and others rotate). Paid Zen models (Claude Opus, GPT-5.5) available at pay-as-you-go rates. Go subscription **$10/mo** ($5 first month) | | 🌍 **30+ frontier models** | DeepSeek V4, Kimi K2.6, GLM-5.1, Qwen3.7 Max, MiMo V2.5, MiniMax M2.7, Big Pickle, Nemotron — **all in one picker** | | 🤖 **Full Agent Mode** | Tool-calling (read files, edit, run terminal) works natively — not just chat. Models also appear in the **Agents window** (Copilot CLI session) | -| 🧠 **Thinking controls** | Per-model reasoning effort (DeepSeek `max`, Qwen `thinking_budget`, MiniMax `on/off`, Mimo `low/med/high`) || 🖼️ **Vision proxy** | Text-only models can "see" images via a configured proxy model (`opencodego.visionModel`). Works with any Copilot‑compatible vision model. || 📊 **Live usage tracking** | Status bar shows Go subscription burn-rate across 5h / weekly / monthly tiers | +| 🧠 **Thinking controls** | Per-model reasoning effort (DeepSeek `max`, Qwen `thinking_budget`, MiniMax `on/off`, Mimo `low/med/high`) || 🖼️ **Vision proxy** | Text-only models can "see" images via a configured vision model. Enable `opencodego.visionProxy` and pick a model with the **OpenCode Go: Configure Vision Proxy** command. || 📊 **Live usage tracking** | Status bar shows Go subscription burn-rate across 5h / weekly / monthly tiers | | 🔌 **Dual providers** | OpenCode **Go** ($10/mo subscription) + OpenCode **Zen** (free + paid models) — run both at once, switch instantly | | 🎯 **Smart routing** | Each model family auto-routes to its native transport (`/responses`, `/messages`, `streamGenerateContent`, `/chat/completions`) | | 🖼️ **Vision + PDF + Audio** | Multimodal models pass through image, PDF, audio, and video inputs | @@ -335,8 +335,7 @@ To manage agent API keys separately or see agent vendors in the Manage panel, en | `opencodego.thinking.mimo` | `"off"` | `off`/`low`/`medium`/`high` | | `opencodego.thinking.qwen` | `"off"` | `auto`/`on`/`off` | | `opencodego.thinking.qwenBudget` | `"auto"` | `auto`/`4096`/`16384`/`32768`/`81920` | -| `opencodego.visionModel` | `""` | Vision proxy model ID (e.g. `copilot:gpt-5.5`). Empty = disabled. Text-only models relay images through this model automatically. | -| `opencodego.visionPrompt` | Built-in | Prompt sent to the vision model when describing images. | +| `opencodego.visionProxy` | `false` | Enable the vision proxy. Use **OpenCode Go: Configure Vision Proxy** to pick a vision model. |
📜 Full settings reference with descriptions diff --git a/package.json b/package.json index 916823c..fdf2e45 100644 --- a/package.json +++ b/package.json @@ -105,6 +105,10 @@ { "command": "opencodego.deleteProfile", "title": "OpenCode Go: Delete Profile" + }, + { + "command": "opencodego.configureVisionProxy", + "title": "OpenCode Go: Configure Vision Proxy" } ], "configuration": { @@ -176,15 +180,10 @@ "default": "auto", "markdownDescription": "How to handle `...` tags in model output. `auto` strips tags only for known reasoning models that inline thinking in the content field (minimax), accumulating the extracted reasoning content into the existing reasoning fallback. `always` strips for every model. `never` passes text through unchanged (use if a model legitimately uses `` in its output)." }, - "opencodego.visionModel": { - "type": "string", - "default": "", - "markdownDescription": "Vision proxy model. When set, images pasted into chat are automatically sent to this model for description before being passed to text-only models. The value must be a model `id` from the VS Code model picker (e.g. `copilot:gpt-5.5`). Leave empty to disable vision proxy." - }, - "opencodego.visionPrompt": { - "type": "string", - "default": "Describe this image in detail so a text-only model can understand what it shows. Include all visible text, layout, colors, objects, and context.", - "markdownDescription": "Prompt sent to the vision proxy model when an image is attached. The image is appended after this text." + "opencodego.visionProxy": { + "type": "boolean", + "default": false, + "markdownDescription": "Enable the vision proxy to let text-only OpenCode models understand images. When a non-vision model receives an image, the extension forwards it to a vision-capable model, receives a text description, and feeds that to the original model. Use **OpenCode Go: Configure Vision Proxy** from the Command Palette (`Ctrl+Shift+P`) to select the vision model and customize the description prompt." }, "opencodego.thinking.deepseek": { "type": "string", diff --git a/src/extension.ts b/src/extension.ts index 426d7c9..1f35aa4 100644 --- a/src/extension.ts +++ b/src/extension.ts @@ -15,6 +15,7 @@ import { normalizeModelsDevSnapshot, resolveModelMetadata, toEffectiveModelId, + VISION_CAPABLE_MODELS, type BaseModelLimits, type CachedModelMetadataSnapshot, type ContextSizeOption, @@ -402,10 +403,8 @@ interface ApiSettings { streamIdleTimeoutMs: number; thinking: ThinkingSettings; stripThinkTags: "never" | "auto" | "always"; - /** Vision proxy model ID (e.g. "copilot:gpt-5.5"). Empty = disabled. */ - visionModel: string; - /** Custom prompt for the vision proxy, e.g. "Describe this image in detail". */ - visionPrompt: string; + /** Whether the vision proxy is enabled (opencodego.visionProxy boolean). */ + visionProxy: boolean; } interface LanguageModelConfiguration { @@ -709,6 +708,9 @@ export function activate(context: vscode.ExtensionContext) { updateWebviewContent(); vscode.window.showInformationMessage(`Profile "${profile.label}" deleted.`); }), + vscode.commands.registerCommand("opencodego.configureVisionProxy", async () => { + await showVisionProxyPicker(context); + }), ]; // Agent-host providers for the Copilot Agents window (opt-in via config). @@ -729,10 +731,15 @@ export function activate(context: vscode.ExtensionContext) { if (event.affectsConfiguration("opencodego.showUsageStatusBar")) { resetUsageStatusBar(); } + if (event.affectsConfiguration("opencodego.visionProxy")) { + goProvider.notifyModelInfoChanged(); + zenProvider.notifyModelInfoChanged(); + } }), ); checkUtilityModelConfiguration(context); + migrateVisionProxySettings(context); void warmModelPickerMetadata(); } @@ -1295,6 +1302,11 @@ function rel(date: Date): string { class OpenCodeProvider implements vscode.LanguageModelChatProvider { private readonly changeEmitter = new vscode.EventEmitter(); readonly onDidChangeLanguageModelChatInformation = this.changeEmitter.event; + + /** Trigger a model information refresh (e.g. after visionModel setting changes). */ + notifyModelInfoChanged(): void { + this.changeEmitter.fire(); + } private readonly apiKeysByModelId = new Map(); /** Capped to prevent unbounded growth across long sessions. */ private readonly reasoningContentByToolCallId = new Map(); @@ -1809,16 +1821,23 @@ class OpenCodeProvider implements vscode.LanguageModelChatProvider(VISION_PROXY_MODEL_ID_KEY, "") || "") + : ""; + if (hasImageInput && !actuallySupportsVision && visionProxyModelId) { + const visionProxyPrompt = this.context.globalState.get(VISION_PROXY_PROMPT_KEY, "") + || DEFAULT_VISION_PROXY_PROMPT; + let imagesHandled = false; try { const description = await proxyVision( messages, - settings.visionModel, - settings.visionPrompt, + visionProxyModelId, + visionProxyPrompt, ); if (description) { for (let i = 0; i < apiMessages.length; i++) { @@ -1828,17 +1847,38 @@ class OpenCodeProvider implements vscode.LanguageModelChatProvider p.type === "text" && typeof p.text === "string") .map(p => p.text); - msg.content = [{ type: "text", text: `[Image described by ${settings.visionModel}]: ${description}` }]; + msg.content = [{ type: "text", text: `[Image described by vision proxy]: ${description}` }]; if (textParts.length > 0) { msg.content.push({ type: "text", text: textParts.join("\n") }); } + imagesHandled = true; } } - this.log(`[vision-proxy] Replaced images using ${settings.visionModel}`); + this.log(`[vision-proxy] Replaced images using vision proxy model`); } } catch (err) { this.log(`[vision-proxy] Error: ${err instanceof Error ? err.message : String(err)}`); } + + // If the proxy didn't handle the images (error, empty response, or + // model not found), strip them anyway so the non-vision model + // doesn't receive image data it can't process (fixes 400 errors). + if (!imagesHandled) { + for (let i = 0; i < apiMessages.length; i++) { + const msg = apiMessages[i]; + if (!Array.isArray(msg.content)) continue; + if (msg.content.some(p => p.type === "image_url")) { + const textParts = msg.content + .filter((p): p is OpenAiContentPart & { text: string } => p.type === "text" && typeof p.text === "string") + .map(p => p.text); + msg.content = [{ type: "text", text: "[Image unavailable — vision proxy unavailable]" }]; + if (textParts.length > 0) { + msg.content.push({ type: "text", text: textParts.join("\n") }); + } + } + } + this.log(`[vision-proxy] Stripped images (proxy unavailable), prevented 400`); + } } const thinkingPayload = buildThinkingPayload(rawModelId, settings.thinking, hasImageInput && metadata.supportsVision); @@ -3173,8 +3213,7 @@ function getSettings(): ApiSettings { mimo: config.get("thinking.mimo", "off"), }, stripThinkTags: config.get("stripThinkTags", "auto"), - visionModel: config.get("visionModel", ""), - visionPrompt: config.get("visionPrompt", ""), + visionProxy: config.get("visionProxy", false), }; } @@ -3240,12 +3279,15 @@ function positiveOverride(value: number): number | undefined { } function modelCapabilities(metadata: ResolvedModelMetadata): CopilotCompatibleCapabilities { - // Mirrors the official shape used by `copilotChat`/`byok` providers: - // `imageInput` and `toolCalling` are the raw proposed-API fields VS Code - // maps to `vision` / `toolCalling` / `agentMode` internally, while - // `supportsImageToText` and `supportsToolCalling` are the runtime API - // booleans consumed by the `vscode.lm` callers in extensions. - const supportsVision = metadata.supportsVision; + // When a vision proxy is enabled AND a model is configured, report + // imageInput: true for ALL models so VS Code does not strip image + // parts from the request before they reach our provider. The vision + // proxy code in provideLanguageModelChatResponse intercepts images + // and forwards them to the configured vision model transparently. + const visionProxyEnabled = vscode.workspace.getConfiguration("opencodego").get("visionProxy", false) + && _extensionContext?.globalState.get(VISION_PROXY_MODEL_ID_KEY, "").length > 0; + const supportsVision = metadata.supportsVision || visionProxyEnabled; + return { imageInput: supportsVision, toolCalling: true, @@ -3317,50 +3359,253 @@ async function proxyVision( visionModelId: string, visionPrompt: string, ): Promise { - const visionModels = await vscode.lm.selectChatModels({ id: visionModelId }); + // Find the vision model by trying several matching strategies: + // 1. Exact id match (full internal model id) + // 2. Vendor:id partial (e.g. "opencodego:mimo-v2.5") + // 3. Name or id substring (e.g. "mimo-v2.5" or "Mimo V2.5") + // Filter out agent-host variants — they use a different transport and + // don't have vision support. Prefer non-agent models. + const nonAgent = (models: readonly vscode.LanguageModelChat[]) => + models.filter(m => !m.id.includes("-agent:")); + + let visionModels = nonAgent(await vscode.lm.selectChatModels({ id: visionModelId })); + if (!visionModels || visionModels.length === 0) { + // Try matching by name substring across all providers + const allVisible = nonAgent(await vscode.lm.selectChatModels({})); + visionModels = allVisible.filter( + m => m.id.toLowerCase().includes(visionModelId.toLowerCase()) + || m.name.toLowerCase().includes(visionModelId.toLowerCase()) + || m.family.toLowerCase().includes(visionModelId.toLowerCase()), + ); + } if (!visionModels || visionModels.length === 0) { - throw new Error(`Vision model "${visionModelId}" not found. Check your opencodego.visionModel setting.`); + throw new Error( + `Vision model "${visionModelId}" not found. ` + + `Run "OpenCode Go: Configure Vision Proxy" to see available models.`, + ); } - const visionModel = visionModels[0]; - // Build a request with the images + prompt. The messages already contain - // LanguageModelDataPart for images — we pass them through as-is. + // All models that matched are candidates. `selectChatModels` returns + // `LanguageModelChat` which does not expose capabilities in the stable + // API, so we just use the first match. Most vision models handle image + // input gracefully — models without vision will report the error. + const model = visionModels[0]; + + // Build a request preserving images and text from the original messages const requestMessages: vscode.LanguageModelChatMessage[] = []; for (const msg of messages) { - let content: Array = []; + const parts: Array = []; for (const part of msg.content) { if (part instanceof vscode.LanguageModelDataPart && part.mimeType.startsWith("image/")) { - content.push(part); + parts.push(part); } else if (part instanceof vscode.LanguageModelTextPart) { - content.push(part); + parts.push(part); } else if (typeof part === "object" && part !== null && "value" in part) { - content.push(new vscode.LanguageModelTextPart(String((part as any).value))); + parts.push(new vscode.LanguageModelTextPart(String((part as any).value))); } } - if (content.length > 0) { - requestMessages.push(new vscode.LanguageModelChatMessage(msg.role === 1 ? vscode.LanguageModelChatMessageRole.Assistant : vscode.LanguageModelChatMessageRole.User, content)); + if (parts.length > 0) { + requestMessages.push(new vscode.LanguageModelChatMessage( + msg.role === vscode.LanguageModelChatMessageRole.Assistant + ? vscode.LanguageModelChatMessageRole.Assistant + : vscode.LanguageModelChatMessageRole.User, + parts, + )); } } - // Add the vision prompt as a user message + // Append the vision prompt if (visionPrompt) { requestMessages.push(vscode.LanguageModelChatMessage.User(visionPrompt)); } - let fullDescription = ""; - // VS Code's sendRequest requires a CancellationToken. We create a - // simple one that never cancels. const cancellationToken: vscode.CancellationToken = { isCancellationRequested: false, onCancellationRequested: () => ({ dispose: () => {} }), }; - const response = await visionModel.sendRequest(requestMessages, {}, cancellationToken); + const response = await model.sendRequest(requestMessages, {}, cancellationToken); + let fullDescription = ""; for await (const part of response.text) { fullDescription += part; } return fullDescription.length > 0 ? fullDescription : undefined; } +// --------------------------------------------------------------------------- +// Vision proxy — globalState storage keys & defaults +// --------------------------------------------------------------------------- + +const VISION_PROXY_MODEL_ID_KEY = "opencodego.visionProxyModelId"; +const VISION_PROXY_PROMPT_KEY = "opencodego.visionProxyPrompt"; +const DEFAULT_VISION_PROXY_PROMPT = + "Describe this image in detail so a text-only model can understand what it shows. " + + "Include all visible text, layout, colors, objects, and context."; + +/** + * One-time migration from the old `opencodego.visionModel` string setting + * (pre-0.4.1) to the new `opencodego.visionProxy` boolean + globalState model + * ID storage. Runs on every activation but only acts when needed. + */ +function migrateVisionProxySettings(context: vscode.ExtensionContext): void { + const config = vscode.workspace.getConfiguration("opencodego"); + const oldModel = config.get("visionModel", ""); + if (!oldModel) return; // nothing to migrate + + // The new boolean setting + const alreadyEnabled = config.get("visionProxy", false); + + // If the globalState already has a model ID and the boolean is on, + // migration was already done — just clean up the old string setting. + const stored = context.globalState.get(VISION_PROXY_MODEL_ID_KEY, ""); + if (stored && alreadyEnabled) { + void config.update("visionModel", undefined, vscode.ConfigurationTarget.Global); + return; + } + + // Migrate: store model ID in globalState, enable the boolean, clear old key. + void context.globalState.update(VISION_PROXY_MODEL_ID_KEY, oldModel); + // Also migrate the prompt if it was customized + const oldPrompt = config.get("visionPrompt", ""); + if (oldPrompt && oldPrompt !== DEFAULT_VISION_PROXY_PROMPT) { + void context.globalState.update(VISION_PROXY_PROMPT_KEY, oldPrompt); + } + void config.update("visionProxy", true, vscode.ConfigurationTarget.Global); + // Clear old settings + void config.update("visionModel", undefined, vscode.ConfigurationTarget.Global); + void config.update("visionPrompt", undefined, vscode.ConfigurationTarget.Global); +} + +/** + * QuickPick to configure vision proxy model and prompt. + * Clean list of model names (no ugly IDs), with "None" to disable + * and "Customize prompt..." to edit the description instruction. + * Saves to globalState; toggles the visionProxy boolean accordingly. + */ +async function showVisionProxyPicker(context: vscode.ExtensionContext): Promise { + const config = vscode.workspace.getConfiguration("opencodego"); + const currentModelId = context.globalState.get(VISION_PROXY_MODEL_ID_KEY, ""); + const currentPrompt = context.globalState.get(VISION_PROXY_PROMPT_KEY, "") + || DEFAULT_VISION_PROXY_PROMPT; + + // --- Build the set of vision-capable model IDs --- + const visionCapableIds = new Set(); + const snapshot = modelMetadataSnapshot; + if (snapshot) { + for (const vendor of [GO_VENDOR, ZEN_VENDOR] as const) { + const provider = snapshot.providers[vendor]; + if (!provider) continue; + for (const [id, meta] of Object.entries(provider)) { + if (meta.supportsVision) visionCapableIds.add(`${vendor}:${id}`); + } + } + } + for (const family of VISION_CAPABLE_MODELS) { + visionCapableIds.add(`copilot:${family}`); + } + + // --- Build QuickPick items from available models --- + const allModels = (await vscode.lm.selectChatModels({})) + .filter(m => !m.id.includes("-agent:")); + + const modelItems = allModels + .map(m => { + const rawId = resolveRawModelId(m.id); + const vendor = resolveVendorFromId(m.id); + const lookupId = `${vendor}:${rawId}`; + const fromLookup = visionCapableIds.has(lookupId); + const fromName = [...visionCapableIds].some(id => + m.id.includes(id.replace(/^(opencodego|opencodezen|copilot):/, ""))); + const supportsVision = fromLookup || fromName; + return { + label: m.name, + description: supportsVision ? "$(eye)" : "", + detail: supportsVision + ? (m.id === currentModelId ? "currently configured" : "vision-capable") + : "", + picked: m.id === currentModelId, + _id: m.id, + _kind: "model" as const, + _supportsVision: supportsVision, + }; + }) + .filter(m => m._supportsVision); + + if (modelItems.length === 0) { + vscode.window.showInformationMessage( + "No vision-capable models found. Make sure you have a Copilot Chat provider with vision models installed.", + ); + return; + } + + modelItems.sort((a, b) => { + if (a._id === currentModelId) return -1; + if (b._id === currentModelId) return 1; + return a.label.localeCompare(b.label); + }); + + const items: Array<{ + label: string; + description?: string; + detail?: string; + picked?: boolean; + _id?: string; + _kind: "none" | "prompt" | "model" | "separator"; + _supportsVision?: boolean; + kind?: vscode.QuickPickItemKind; + }> = [ + { label: "$(circle-slash) None (disable)", detail: currentModelId ? "" : "currently selected", picked: !currentModelId, _kind: "none" }, + { label: "", kind: vscode.QuickPickItemKind.Separator, _kind: "separator" }, + { label: "$(edit) Customize description prompt...", description: "$(info) Sets how the vision model describes images", _kind: "prompt" }, + { label: "", kind: vscode.QuickPickItemKind.Separator, _kind: "separator" }, + ...modelItems, + ]; + + const picked = await vscode.window.showQuickPick(items, { + placeHolder: "Pick a model, customize the prompt, or disable", + title: "OpenCode Go — Vision Proxy", + matchOnDescription: true, + }); + + if (!picked || !("_kind" in picked)) return; + + // --- "Customize prompt..." --- + if (picked._kind === "prompt") { + const newPrompt = await vscode.window.showInputBox({ + title: "Vision Proxy — Description Prompt", + prompt: "Prompt sent to the vision model to describe the image.", + value: currentPrompt, + placeHolder: DEFAULT_VISION_PROXY_PROMPT, + validateInput: (value: string) => value.trim() ? undefined : "Prompt cannot be empty.", + }); + if (newPrompt === undefined) return; // cancelled + await context.globalState.update(VISION_PROXY_PROMPT_KEY, newPrompt.trim()); + vscode.window.showInformationMessage("Vision proxy prompt updated."); + return; + } + + // --- "None" --- + if (picked._kind === "none") { + await context.globalState.update(VISION_PROXY_MODEL_ID_KEY, ""); + await config.update("visionProxy", false, vscode.ConfigurationTarget.Global); + vscode.window.showInformationMessage("Vision proxy disabled."); + return; + } + + // --- Model selected --- + if (!picked._id) return; + await context.globalState.update(VISION_PROXY_MODEL_ID_KEY, picked._id); + await config.update("visionProxy", true, vscode.ConfigurationTarget.Global); + vscode.window.showInformationMessage(`Vision proxy set to: ${picked.label}`); +} + +/** Best-effort vendor resolution from a model ID. */ +function resolveVendorFromId(modelId: string): AllProviderVendor { + if (modelId.startsWith(`${AGENT_GO_VENDOR}:`)) return AGENT_GO_VENDOR; + if (modelId.startsWith(`${AGENT_ZEN_VENDOR}:`)) return AGENT_ZEN_VENDOR; + if (modelId.startsWith(`${ZEN_VENDOR}:`)) return ZEN_VENDOR; + return GO_VENDOR; +} + /** * Returns pricing fields for VS Code's language model pricing proposal * (`vscode.proposed.languageModelPricing`). diff --git a/src/metadata.ts b/src/metadata.ts index 69e2e34..a000206 100644 --- a/src/metadata.ts +++ b/src/metadata.ts @@ -235,7 +235,7 @@ const MODELS_WITHOUT_TEMPERATURE = new Set([ "kimi-k2.7-code", ]); -const VISION_CAPABLE_MODELS = new Set([ +export const VISION_CAPABLE_MODELS = new Set([ "minimax-m2.7", "minimax-m2.5", "minimax-m2.5-free", diff --git a/src/streaming.ts b/src/streaming.ts index b4dc04b..bc461d6 100644 --- a/src/streaming.ts +++ b/src/streaming.ts @@ -107,7 +107,8 @@ export async function streamChatCompletions( options.output?.appendLine( `[warn] empty response from model=${options.modelId} (no text, no tool calls, no reasoning). Try a different free model or enable opencodego.debugReasoning to inspect raw SSE.`, ); - options.output?.show(true); + // Intentionally not calling .show(true) — the diagnostic log is + // available in the Output pane when the user opens it manually. } } From a17f91eb8e811b5cf12189eee370bc838d9518be Mon Sep 17 00:00:00 2001 From: Wallacy Date: Sun, 12 Jul 2026 19:46:21 -0300 Subject: [PATCH 3/4] =?UTF-8?q?refactor:=20remove=20visionProxy=20boolean?= =?UTF-8?q?=20=E2=80=94=20command-only=20UX,=20no=20setting=20needed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 2 +- README.md | 3 +- package.json | 5 -- src/extension.ts | 64 ++++++------------------- src/test/metadata.test.ts | 24 +++++++++- src/test/visionProxy.test.ts | 90 ++++++++++++++++++++++++++++++++++++ 6 files changed, 129 insertions(+), 59 deletions(-) create mode 100644 src/test/visionProxy.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 5703550..1551ba0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ All notable changes to the **OpenCode Go BYOK Provider** extension are documente ### Added -- **`[Vision]` Transparent vision proxy for text-only models (#74).** Enable `opencodego.visionProxy` in settings, then run **OpenCode Go: Configure Vision Proxy** from the Command Palette to pick a vision-capable model. When a non-vision OpenCode model receives an image, the extension forwards it to a configured vision model, receives a text description, and feeds that to the original model — so text-only models "see" images with zero extra steps. The picker shows only vision-capable models, with a "None" option to disable. A **Customize prompt** entry in the picker lets you edit the description instruction. Model ID and prompt are stored in extension state, set exclusively via the command. Implemented with `vscode.lm.selectChatModels` + `sendRequest`. +- **`[Vision]` Transparent vision proxy for text-only models (#74).** Run **OpenCode Go: Configure Vision Proxy** from the Command Palette to pick a vision-capable model. When a non-vision OpenCode model receives an image, the extension forwards it to the configured model, receives a text description, and feeds that to the original model — so text-only models "see" images with zero extra steps. The picker shows only vision-capable models (filtered by `models.dev` metadata), with a **None** option to disable and a **Customize prompt** entry to edit the description instruction. No settings to toggle — if a model is configured, the proxy is on. Implemented with `vscode.lm.selectChatModels` + `sendRequest`. ### Fixed diff --git a/README.md b/README.md index 6c0daf3..13ae95b 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ | 💸 **Cheaper than Copilot Pro+** | Copilot Free + OpenCode **Zen free** models = **$0** for 2-5 rotating models (Big Pickle always free; DeepSeek V4 Flash, MiMo-V2.5, and others rotate). Paid Zen models (Claude Opus, GPT-5.5) available at pay-as-you-go rates. Go subscription **$10/mo** ($5 first month) | | 🌍 **30+ frontier models** | DeepSeek V4, Kimi K2.6, GLM-5.1, Qwen3.7 Max, MiMo V2.5, MiniMax M2.7, Big Pickle, Nemotron — **all in one picker** | | 🤖 **Full Agent Mode** | Tool-calling (read files, edit, run terminal) works natively — not just chat. Models also appear in the **Agents window** (Copilot CLI session) | -| 🧠 **Thinking controls** | Per-model reasoning effort (DeepSeek `max`, Qwen `thinking_budget`, MiniMax `on/off`, Mimo `low/med/high`) || 🖼️ **Vision proxy** | Text-only models can "see" images via a configured vision model. Enable `opencodego.visionProxy` and pick a model with the **OpenCode Go: Configure Vision Proxy** command. || 📊 **Live usage tracking** | Status bar shows Go subscription burn-rate across 5h / weekly / monthly tiers | +| 🧠 **Thinking controls** | Per-model reasoning effort (DeepSeek `max`, Qwen `thinking_budget`, MiniMax `on/off`, Mimo `low/med/high`) || 🖼️ **Vision proxy** | Text-only models can "see" images via a configured vision model. Run **OpenCode Go: Configure Vision Proxy** from the Command Palette to set it up. || 📊 **Live usage tracking** | Status bar shows Go subscription burn-rate across 5h / weekly / monthly tiers | | 🔌 **Dual providers** | OpenCode **Go** ($10/mo subscription) + OpenCode **Zen** (free + paid models) — run both at once, switch instantly | | 🎯 **Smart routing** | Each model family auto-routes to its native transport (`/responses`, `/messages`, `streamGenerateContent`, `/chat/completions`) | | 🖼️ **Vision + PDF + Audio** | Multimodal models pass through image, PDF, audio, and video inputs | @@ -335,7 +335,6 @@ To manage agent API keys separately or see agent vendors in the Manage panel, en | `opencodego.thinking.mimo` | `"off"` | `off`/`low`/`medium`/`high` | | `opencodego.thinking.qwen` | `"off"` | `auto`/`on`/`off` | | `opencodego.thinking.qwenBudget` | `"auto"` | `auto`/`4096`/`16384`/`32768`/`81920` | -| `opencodego.visionProxy` | `false` | Enable the vision proxy. Use **OpenCode Go: Configure Vision Proxy** to pick a vision model. |
📜 Full settings reference with descriptions diff --git a/package.json b/package.json index fdf2e45..f7d73ea 100644 --- a/package.json +++ b/package.json @@ -180,11 +180,6 @@ "default": "auto", "markdownDescription": "How to handle `...` tags in model output. `auto` strips tags only for known reasoning models that inline thinking in the content field (minimax), accumulating the extracted reasoning content into the existing reasoning fallback. `always` strips for every model. `never` passes text through unchanged (use if a model legitimately uses `` in its output)." }, - "opencodego.visionProxy": { - "type": "boolean", - "default": false, - "markdownDescription": "Enable the vision proxy to let text-only OpenCode models understand images. When a non-vision model receives an image, the extension forwards it to a vision-capable model, receives a text description, and feeds that to the original model. Use **OpenCode Go: Configure Vision Proxy** from the Command Palette (`Ctrl+Shift+P`) to select the vision model and customize the description prompt." - }, "opencodego.thinking.deepseek": { "type": "string", "enum": [ diff --git a/src/extension.ts b/src/extension.ts index 1f35aa4..a37596b 100644 --- a/src/extension.ts +++ b/src/extension.ts @@ -403,8 +403,6 @@ interface ApiSettings { streamIdleTimeoutMs: number; thinking: ThinkingSettings; stripThinkTags: "never" | "auto" | "always"; - /** Whether the vision proxy is enabled (opencodego.visionProxy boolean). */ - visionProxy: boolean; } interface LanguageModelConfiguration { @@ -710,6 +708,10 @@ export function activate(context: vscode.ExtensionContext) { }), vscode.commands.registerCommand("opencodego.configureVisionProxy", async () => { await showVisionProxyPicker(context); + // The proxy model changed — refresh capabilities so VS Code stops + // stripping images from non-vision models when the proxy is on. + goProvider.notifyModelInfoChanged(); + zenProvider.notifyModelInfoChanged(); }), ]; @@ -731,15 +733,10 @@ export function activate(context: vscode.ExtensionContext) { if (event.affectsConfiguration("opencodego.showUsageStatusBar")) { resetUsageStatusBar(); } - if (event.affectsConfiguration("opencodego.visionProxy")) { - goProvider.notifyModelInfoChanged(); - zenProvider.notifyModelInfoChanged(); - } }), ); checkUtilityModelConfiguration(context); - migrateVisionProxySettings(context); void warmModelPickerMetadata(); } @@ -1826,7 +1823,7 @@ class OpenCodeProvider implements vscode.LanguageModelChatProvider(VISION_PROXY_MODEL_ID_KEY, "") || "") : ""; if (hasImageInput && !actuallySupportsVision && visionProxyModelId) { @@ -3213,7 +3210,6 @@ function getSettings(): ApiSettings { mimo: config.get("thinking.mimo", "off"), }, stripThinkTags: config.get("stripThinkTags", "auto"), - visionProxy: config.get("visionProxy", false), }; } @@ -3279,14 +3275,11 @@ function positiveOverride(value: number): number | undefined { } function modelCapabilities(metadata: ResolvedModelMetadata): CopilotCompatibleCapabilities { - // When a vision proxy is enabled AND a model is configured, report - // imageInput: true for ALL models so VS Code does not strip image - // parts from the request before they reach our provider. The vision - // proxy code in provideLanguageModelChatResponse intercepts images - // and forwards them to the configured vision model transparently. - const visionProxyEnabled = vscode.workspace.getConfiguration("opencodego").get("visionProxy", false) - && _extensionContext?.globalState.get(VISION_PROXY_MODEL_ID_KEY, "").length > 0; - const supportsVision = metadata.supportsVision || visionProxyEnabled; + // When a vision proxy model is configured (non-empty ID in globalState), + // report imageInput: true for ALL models so VS Code does not strip image + // parts before they reach our provider. The vision proxy interceptor + // forwards images to the configured model transparently. + const supportsVision = metadata.supportsVision || isVisionProxyEnabled(); return { imageInput: supportsVision, @@ -3442,37 +3435,11 @@ const DEFAULT_VISION_PROXY_PROMPT = "Include all visible text, layout, colors, objects, and context."; /** - * One-time migration from the old `opencodego.visionModel` string setting - * (pre-0.4.1) to the new `opencodego.visionProxy` boolean + globalState model - * ID storage. Runs on every activation but only acts when needed. + * True when a vision proxy model has been configured (non-empty model ID + * stored in globalState via the "OpenCode Go: Configure Vision Proxy" command). */ -function migrateVisionProxySettings(context: vscode.ExtensionContext): void { - const config = vscode.workspace.getConfiguration("opencodego"); - const oldModel = config.get("visionModel", ""); - if (!oldModel) return; // nothing to migrate - - // The new boolean setting - const alreadyEnabled = config.get("visionProxy", false); - - // If the globalState already has a model ID and the boolean is on, - // migration was already done — just clean up the old string setting. - const stored = context.globalState.get(VISION_PROXY_MODEL_ID_KEY, ""); - if (stored && alreadyEnabled) { - void config.update("visionModel", undefined, vscode.ConfigurationTarget.Global); - return; - } - - // Migrate: store model ID in globalState, enable the boolean, clear old key. - void context.globalState.update(VISION_PROXY_MODEL_ID_KEY, oldModel); - // Also migrate the prompt if it was customized - const oldPrompt = config.get("visionPrompt", ""); - if (oldPrompt && oldPrompt !== DEFAULT_VISION_PROXY_PROMPT) { - void context.globalState.update(VISION_PROXY_PROMPT_KEY, oldPrompt); - } - void config.update("visionProxy", true, vscode.ConfigurationTarget.Global); - // Clear old settings - void config.update("visionModel", undefined, vscode.ConfigurationTarget.Global); - void config.update("visionPrompt", undefined, vscode.ConfigurationTarget.Global); +function isVisionProxyEnabled(): boolean { + return (_extensionContext?.globalState.get(VISION_PROXY_MODEL_ID_KEY, "") ?? "").length > 0; } /** @@ -3482,7 +3449,6 @@ function migrateVisionProxySettings(context: vscode.ExtensionContext): void { * Saves to globalState; toggles the visionProxy boolean accordingly. */ async function showVisionProxyPicker(context: vscode.ExtensionContext): Promise { - const config = vscode.workspace.getConfiguration("opencodego"); const currentModelId = context.globalState.get(VISION_PROXY_MODEL_ID_KEY, ""); const currentPrompt = context.globalState.get(VISION_PROXY_PROMPT_KEY, "") || DEFAULT_VISION_PROXY_PROMPT; @@ -3586,7 +3552,6 @@ async function showVisionProxyPicker(context: vscode.ExtensionContext): Promise< // --- "None" --- if (picked._kind === "none") { await context.globalState.update(VISION_PROXY_MODEL_ID_KEY, ""); - await config.update("visionProxy", false, vscode.ConfigurationTarget.Global); vscode.window.showInformationMessage("Vision proxy disabled."); return; } @@ -3594,7 +3559,6 @@ async function showVisionProxyPicker(context: vscode.ExtensionContext): Promise< // --- Model selected --- if (!picked._id) return; await context.globalState.update(VISION_PROXY_MODEL_ID_KEY, picked._id); - await config.update("visionProxy", true, vscode.ConfigurationTarget.Global); vscode.window.showInformationMessage(`Vision proxy set to: ${picked.label}`); } diff --git a/src/test/metadata.test.ts b/src/test/metadata.test.ts index 170fa70..bc96f39 100644 --- a/src/test/metadata.test.ts +++ b/src/test/metadata.test.ts @@ -1,6 +1,6 @@ import { describe, it } from "node:test"; import assert from "node:assert/strict"; -import { fallbackModelMetadata } from "../metadata.js"; +import { fallbackModelMetadata, VISION_CAPABLE_MODELS } from "../metadata.js"; import { GO_VENDOR, ZEN_VENDOR } from "../providerTypes.js"; /** @@ -72,3 +72,25 @@ describe("fallbackModelMetadata — non-kimi models unaffected", () => { assert.notEqual(meta?.temperature, false); }); }); + +describe("VISION_CAPABLE_MODELS", () => { + it("includes known vision models (minimax-m2.7, kimi-k2.6, mimo-v2.5)", () => { + assert.ok(VISION_CAPABLE_MODELS.has("minimax-m2.7")); + assert.ok(VISION_CAPABLE_MODELS.has("kimi-k2.6")); + assert.ok(VISION_CAPABLE_MODELS.has("mimo-v2.5")); + assert.ok(VISION_CAPABLE_MODELS.has("glm-5.1")); + assert.ok(VISION_CAPABLE_MODELS.has("mimo-v2.5-pro")); + }); + + it("does NOT include text-only models (deepseek-v4-flash, hy3-preview, big-pickle)", () => { + assert.ok(!VISION_CAPABLE_MODELS.has("deepseek-v4-flash")); + assert.ok(!VISION_CAPABLE_MODELS.has("deepseek-v4-pro")); + assert.ok(!VISION_CAPABLE_MODELS.has("hy3-preview")); + assert.ok(!VISION_CAPABLE_MODELS.has("big-pickle")); + }); + + it("is an exported Set", () => { + assert.ok(VISION_CAPABLE_MODELS instanceof Set); + assert.ok(VISION_CAPABLE_MODELS.size > 10); + }); +}); diff --git a/src/test/visionProxy.test.ts b/src/test/visionProxy.test.ts new file mode 100644 index 0000000..bd0b36a --- /dev/null +++ b/src/test/visionProxy.test.ts @@ -0,0 +1,90 @@ +import { describe, it, mock } from "node:test"; +import assert from "node:assert/strict"; + +/** + * Vision proxy condition tests. + * + * The core fix for #74 is caching `metadata.supportsVision` in + * `actuallySupportsVision` BEFORE `modelCapabilities` overrides it. + * The proxy condition is: + * + * hasImageInput && !actuallySupportsVision && visionProxyModelId + * + * where `actuallySupportsVision` is the RAW model metadata (true = + * model natively supports images), NOT the enhanced capabilities. + */ + +type ProxyConditionInput = { + hasImageInput: boolean; + actuallySupportsVision: boolean; + visionProxyModelId: string; +}; + +function shouldProxy({ hasImageInput, actuallySupportsVision, visionProxyModelId }: ProxyConditionInput): boolean { + return Boolean(hasImageInput && !actuallySupportsVision && visionProxyModelId); +} + +describe("vision proxy condition (shouldProxy)", () => { + it("enters proxy when text-only model receives images with proxy configured", () => { + assert.ok(shouldProxy({ hasImageInput: true, actuallySupportsVision: false, visionProxyModelId: "gpt-5.5" })); + }); + + it("skips proxy when no images present", () => { + assert.ok(!shouldProxy({ hasImageInput: false, actuallySupportsVision: false, visionProxyModelId: "gpt-5.5" })); + }); + + it("skips proxy when model natively supports vision", () => { + assert.ok(!shouldProxy({ hasImageInput: true, actuallySupportsVision: true, visionProxyModelId: "gpt-5.5" })); + }); + + it("skips proxy when no vision model is configured (empty string)", () => { + assert.ok(!shouldProxy({ hasImageInput: true, actuallySupportsVision: false, visionProxyModelId: "" })); + }); + + it("skips proxy when all conditions are false", () => { + assert.ok(!shouldProxy({ hasImageInput: false, actuallySupportsVision: true, visionProxyModelId: "" })); + }); + + it("cached supportsVision (actuallySupportsVision) prevents circular regression", () => { + // This is the fix for #74: even if modelCapabilities overrides + // metadata.supportsVision to true (because proxy is enabled), + // the CACHED value (actuallySupportsVision) stays false for + // text-only models — so the proxy fires correctly. + const textOnlyModel = { hasImageInput: true, actuallySupportsVision: false, visionProxyModelId: "gpt-5.5" }; + const visionModel = { hasImageInput: true, actuallySupportsVision: true, visionProxyModelId: "gpt-5.5" }; + + // Before fix: visionModel.actuallySupportsVision was false → proxy fired + // After fix: both behave correctly + assert.ok(shouldProxy(textOnlyModel), "text-only model: proxy fires"); + assert.ok(!shouldProxy(visionModel), "vision model: proxy does NOT fire"); + }); +}); + +describe("modelCapabilities vision proxy flag", () => { + // modelCapabilities() returns imageInput: true when: + // metadata.supportsVision (native) OR isVisionProxyEnabled() + // This tells VS Code NOT to strip images from requests. + + it("returns imageInput: true when proxy is enabled on text-only models", () => { + const capabilities = simulateModelCapabilities(false, true); + assert.equal(capabilities.imageInput, true); + }); + + it("returns imageInput: true when model natively supports vision", () => { + const capabilities = simulateModelCapabilities(true, false); + assert.equal(capabilities.imageInput, true); + }); + + it("returns imageInput: false only when no vision support and no proxy", () => { + const capabilities = simulateModelCapabilities(false, false); + assert.equal(capabilities.imageInput, false); + }); +}); + +function simulateModelCapabilities( + metadataSupportsVision: boolean, + visionProxyEnabled: boolean, +): { imageInput: boolean } { + const supportsVision = metadataSupportsVision || visionProxyEnabled; + return { imageInput: supportsVision }; +} From 8a0d813a5c22cbf62b02c1365423e907cabb2796 Mon Sep 17 00:00:00 2001 From: Wallacy Date: Tue, 14 Jul 2026 17:03:40 -0300 Subject: [PATCH 4/4] =?UTF-8?q?fix:=20PR=20review=20=E2=80=94=20context=20?= =?UTF-8?q?overflow=20safety=20margin,=20real=20cancellation=20token=20for?= =?UTF-8?q?=20proxy,=20readme/changelog=20fixes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 2 ++ README.md | 4 +++- src/extension.ts | 15 +++++++++------ 3 files changed, 14 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1551ba0..392532e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,6 +26,8 @@ All notable changes to the **OpenCode Go BYOK Provider** extension are documente - **`[Usage]` 5-hour rolling usage now isolates per profile in multi-profile mode.** The shared `opencode.db` SQLite database has no API key column, so reading it in multi-account mode mixed quota from all accounts. The tracker now skips SQLite entirely when multiple profiles exist and falls back to extension-tracked entries (which are namespaced per profile). This trades billed-accuracy for correct per-profile isolation. +## [0.3.7] — 2026-07-09 + ### Added - **`[Thinking]` Reasoning now surfaced to Copilot Chat as collapsible thinking blocks.** Previously, reasoning content from OpenCode models (DeepSeek, Kimi, GLM, Qwen, MiniMax, MiMo) was accumulated internally but never emitted to the VS Code Chat UI as a thinking part — so `chat.agent.thinkingStyle` (`collapsed` / `collapsedPreview` / `fixedScrolling`) had no effect, and reasoning either appeared as flat plain text or was silently dropped. The extension now streams reasoning per-chunk via the proposed `LanguageModelThinkingPart` API (available at runtime since VS Code ~1.102, well within our `^1.125.0` floor), across all four transports (chat-completions, Anthropic messages, OpenAI responses, Google Gemini) and both streaming + non-stream response paths. This makes `chat.agent.thinkingStyle` work for OpenCode BYOK models, matching the behavior of Copilot-hosted models. Falls back to the legacy accumulate-and-flush behavior on hypothetical older hosts via a runtime guard. Tool-call replication (`onReasoningContent`) and think-tag filtering (`opencodego.stripThinkTags`) remain intact and compose with the new surfacing. No `enabledApiProposals` declaration needed. Verified working with DeepSeek and Kimi in Copilot Chat. Fixes [#22](https://github.com/ltmoerdani/opencode-copilot-chat/issues/22) and [#71](https://github.com/ltmoerdani/opencode-copilot-chat/issues/71). See `docs/issues/33-20260709-thinking-part-byok-surfacing-research.md`. diff --git a/README.md b/README.md index 13ae95b..cac10a7 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,9 @@ | 💸 **Cheaper than Copilot Pro+** | Copilot Free + OpenCode **Zen free** models = **$0** for 2-5 rotating models (Big Pickle always free; DeepSeek V4 Flash, MiMo-V2.5, and others rotate). Paid Zen models (Claude Opus, GPT-5.5) available at pay-as-you-go rates. Go subscription **$10/mo** ($5 first month) | | 🌍 **30+ frontier models** | DeepSeek V4, Kimi K2.6, GLM-5.1, Qwen3.7 Max, MiMo V2.5, MiniMax M2.7, Big Pickle, Nemotron — **all in one picker** | | 🤖 **Full Agent Mode** | Tool-calling (read files, edit, run terminal) works natively — not just chat. Models also appear in the **Agents window** (Copilot CLI session) | -| 🧠 **Thinking controls** | Per-model reasoning effort (DeepSeek `max`, Qwen `thinking_budget`, MiniMax `on/off`, Mimo `low/med/high`) || 🖼️ **Vision proxy** | Text-only models can "see" images via a configured vision model. Run **OpenCode Go: Configure Vision Proxy** from the Command Palette to set it up. || 📊 **Live usage tracking** | Status bar shows Go subscription burn-rate across 5h / weekly / monthly tiers | +| 🧠 **Thinking controls** | Per-model reasoning effort (DeepSeek `max`, Qwen `thinking_budget`, MiniMax `on/off`, Mimo `low/med/high`) | +| 🖼️ **Vision proxy** | Text-only models can "see" images via a configured vision model. Run **OpenCode Go: Configure Vision Proxy** from the Command Palette to set it up. | +| 📊 **Live usage tracking** | Status bar shows Go subscription burn-rate across 5h / weekly / monthly tiers | | 🔌 **Dual providers** | OpenCode **Go** ($10/mo subscription) + OpenCode **Zen** (free + paid models) — run both at once, switch instantly | | 🎯 **Smart routing** | Each model family auto-routes to its native transport (`/responses`, `/messages`, `streamGenerateContent`, `/chat/completions`) | | 🖼️ **Vision + PDF + Audio** | Multimodal models pass through image, PDF, audio, and video inputs | diff --git a/src/extension.ts b/src/extension.ts index a37596b..b471771 100644 --- a/src/extension.ts +++ b/src/extension.ts @@ -1831,10 +1831,12 @@ class OpenCodeProvider implements vscode.LanguageModelChatProvider { // Find the vision model by trying several matching strategies: // 1. Exact id match (full internal model id) @@ -3412,11 +3419,7 @@ async function proxyVision( requestMessages.push(vscode.LanguageModelChatMessage.User(visionPrompt)); } - const cancellationToken: vscode.CancellationToken = { - isCancellationRequested: false, - onCancellationRequested: () => ({ dispose: () => {} }), - }; - const response = await model.sendRequest(requestMessages, {}, cancellationToken); + const response = await model.sendRequest(requestMessages, {}, token); let fullDescription = ""; for await (const part of response.text) { fullDescription += part;