👁️ Vision MCP Server

Give your AI agent eyes. An MCP server providing multimodal vision capabilities through Gemini-native and OpenAI-compatible APIs, with automatic provider fallback.

让你的 AI 代理拥有视觉能力。 通过任何 OpenAI 兼容的视觉模型，提供图像分析、OCR 文字识别、图像对比和视频分析。

Features · Quick Start · Tools · Models · 中文说明

✨ Features

Tool	Description
🔍 `analyze_image`	Analyze images; `overview`/`focus`/`extract` modes with a persistent analysis workspace
🔎 `inspect_image`	Follow-up questions on an existing workspace by `analysis_id` (region-aware)
🧹 `release_analysis`	Release a workspace: delete cached image and history
📝 `ocr_image`	Extract text from images (plain text / Markdown / JSON)
🔀 `compare_images`	Compare 2–4 images side by side
🎬 `analyze_video`	Analyze video content (requires video-capable model)

Plus:

🗂️ Stateful analysis workspace — analysis_id is returned and accepted explicitly (MCP 2026-07-28 style); no hidden MCP session is used
✂️ Region crop & upscale — normalized coordinates → crop → Lanczos upscale → data URI; never AI super-resolution (protects OCR of figures/code)
🌐 Multi-provider — Gemini native and OpenAI-compatible APIs
🔁 Quota-aware pools — Rotates Gemini models and advances only on HTTP 429
🎚️ Quality tiers — Keep scarce premium quota for explicit quality: "high" calls
📁 Local files & URLs — Auto-converts local files to base64; remote images cached to temp files with TTL
⚙️ Configurable — Environment variables, config files, or both

🚀 Quick Start

1. Install

git clone https://github.com/xiaobright/vision-mcp-pool.git
cd vision-mcp-pool
npm install && npm run build

2. Configure

Copy .env.example to .env, then configure the default Gemini profile (base URL + key are shared by all model pools):

VISION_BASE_URL=https://generativelanguage.googleapis.com
VISION_API_KEY=your-api-key
VISION_AUTH_MODE=x-goog-api-key
VISION_LITE_MODELS=gemini-3.1-flash-lite,gemini-3.5-flash-lite
VISION_FLASH_MODELS=gemini-3-flash-preview,gemini-3.5-flash,gemini-3.6-flash
VISION_GEMMA_MODELS=gemma-4-26b-a4b-it,gemma-4-31b-it

📄 Or use config.json

{
  "baseUrl": "http://your-server:port/v1/chat/completions",
  "model": "Qwen3-VL-32B",
  "apiKey": "your-api-key",
  "maxTokens": 4096,
  "temperature": 0.7
}

3. Run

npm start

The server communicates over stdio, designed to be launched by an MCP client such as Claude Code.

🔌 Claude Code Integration

Add to your ~/.mcp.json:

{
  "mcpServers": {
    "vision": {
      "command": "node",
      "args": ["/path/to/vision-mcp-pool/dist/index.js"],
      "env": {
        "VISION_BASE_URL": "http://your-server:port/v1/chat/completions",
        "VISION_MODEL": "Qwen3-VL-32B",
        "VISION_API_KEY": "your-api-key"
      }
    }
  }
}

Replace /path/to/vision-mcp-pool with the actual install path.

⚙️ Provider Routing

Profiles are selected in this order:

Request	Order
`quality: "auto"`	Lite pool → Flash pool → Gemma 4 pool → SiliconFlow
`quality: "fast"`	Lite pool → SiliconFlow
`quality: "high"`	Flash pool → Gemma 4 pool → Lite pool → SiliconFlow

Calls round-robin within each pool. Only HTTP 429 advances to the next model or tier. Authentication failures, malformed requests, timeouts, network failures, and server errors stop immediately so configuration and infrastructure problems remain visible.

Gemini endpoints may be either a host root (the server appends /v1beta/models/{model}:generateContent), a complete :generateContent URL, or a URL containing a {model} placeholder. Supported authentication modes are x-goog-api-key, bearer, query, and none.

See .env.example for the default profile (lite/flash/gemma model pools sharing one base URL and key) and the OpenAI-compatible emergency fallback. Model IDs must exactly match the names accepted by the selected API or proxy.

⚙️ Configuration Reference

Configuration priority: environment variables > config file (config.json / vision-mcp.config.json) > defaults

In environment-variable mode the lite/flash/gemma pools share VISION_BASE_URL / VISION_API_KEY / VISION_AUTH_MODE; the emergency fallback has its own endpoint and key. (VISION_MODEL / VISION_API_FORMAT are not read in env mode - model IDs come from the pools, and the format is fixed: Gemini for the three pools, OpenAI for emergency. Use config.json profiles for per-profile format/model.)

Variable	Default	Description
`VISION_BASE_URL`	(required)	Gemini API host root; shared by lite/flash/gemma pools
`VISION_API_KEY`	(empty)	API key shared by lite/flash/gemma pools
`VISION_AUTH_MODE`	`x-goog-api-key`	Auth mode: `x-goog-api-key` / `bearer` / `query` / `none`
`VISION_LITE_MODELS`	`gemini-3.1-flash-lite,gemini-3.5-flash-lite`	Comma-separated lite-tier model IDs
`VISION_FLASH_MODELS`	`gemini-3-flash-preview,gemini-3.5-flash,gemini-3.6-flash`	Comma-separated flash-tier model IDs
`VISION_GEMMA_MODELS`	`gemma-4-26b-a4b-it,gemma-4-31b-it`	Comma-separated gemma-tier model IDs
`VISION_EMERGENCY_BASE_URL`	-	OpenAI-compatible emergency fallback endpoint
`VISION_EMERGENCY_MODEL`	`Qwen/Qwen3.5-4B`	Emergency fallback model
`VISION_EMERGENCY_API_KEY`	(empty)	Emergency fallback API key
`VISION_MAX_TOKENS`	`4096`	Max response tokens
`VISION_TEMPERATURE`	`0.3`	Sampling temperature
`VISION_TIMEOUT_MS`	`60000`	Per-request timeout in milliseconds
`VISION_ALLOWED_DIRS`	(all paths)	Comma-separated allowlist of directories for local image paths
`VISION_ANALYSIS_TTL_MINUTES`	`30`	Analysis workspace lifetime in minutes
`VISION_ANALYSIS_MAX_SESSIONS`	`100`	Max concurrent analysis workspaces

🛠️ Tools Reference

`analyze_image`

Analyze an image with a vision language model. Without analysis_id it creates a new analysis workspace; with analysis_id it continues the existing one. For verbatim small text, amounts, code, or tables, use mode=extract - or, for strict OCR / complex layouts, call the dedicated PaddleOCR MCP instead.

Parameter	Type	Required	Default	Description
`image`	string	✅	—	Local file path or URL (ignored when `analysis_id` is given)
`prompt`	string		overview default	Analysis prompt; a structural prompt is used for `overview` when omitted
`mode`	`"overview"` \| `"focus"` \| `"extract"`		`overview` (new) / `focus` (continue)	See workflow below
`quality`	`"auto"` \| `"fast"` \| `"high"`		`"auto"`	Provider quality tier
`region`	`{x,y,width,height}`		—	Normalized region (0–1) for `focus`/`extract`
`analysis_id`	string		—	Continue an existing workspace
`detail`	`"low"` \| `"high"` \| `"auto"`		`"auto"`	Detail level

Result (both structuredContent and a JSON TextContent):

{
  "analysis_id": "img_a83f9c2b4d5e6f708192",
  "answer": "这是一个构建监控页面",
  "summary": "页面由顶部导航、左侧菜单和中央内容区组成",
  "regions": [{ "id": "header", "label": "顶部导航", "box": { "x": 0, "y": 0, "width": 1, "height": 0.14 } }],
  "evidence": [],
  "uncertainties": ["右下角日志文字过小"],
  "suggested_questions": ["读取右下角完整错误日志"],
  "model": "gemini-3.1-flash-lite"
}

`inspect_image`

Follow-up on an existing workspace. Region targeting supports both region_id (from the overview result) and an explicit normalized region.

Parameter	Type	Required	Default	Description
`analysis_id`	string	✅	—	Workspace id from `analyze_image`
`question`	string	✅	—	Follow-up question
`region_id`	string		—	Region id from the overview result, e.g. `"main_content"`
`region`	`{x,y,width,height}`		—	Normalized region (0–1)
`mode`	`"focus"` \| `"extract"`		`"focus"`	`extract` = read small text / tables / code verbatim
`quality`	`"auto"` \| `"fast"` \| `"high"`		`"auto"`	Provider quality tier

`release_analysis`

Parameter	Type	Required	Description
`analysis_id`	string	✅	Workspace to release (deletes cached image and history)

`ocr_image`

Extract text from an image using OCR.

Parameter	Type	Required	Default	Description
`image`	string	✅	—	Local file path or URL
`languages`	string		`""`	Language hint, e.g. `"zh,en"`
`format`	`"plain"` \| `"markdown"` \| `"json"`		`"plain"`	Output format

For strict OCR of complex layouts (dense tables, formulas, multi-page PDFs), prefer the dedicated PaddleOCR MCP; ocr_image here is a lightweight VLM-based extraction.

`compare_images`

Compare 2–4 images and describe differences/similarities.

Parameter	Type	Required	Default	Description
`images`	string[]	✅	—	2–4 image sources
`prompt`	string		`"Compare these images..."`	Comparison prompt

`analyze_video`

Analyze video content. Requires a model with video support (e.g., Qwen3-VL).

Parameter	Type	Required	Default	Description
`video`	string	✅	—	Local file path or URL
`prompt`	string		`"Describe what happens in this video."`	Analysis prompt

🗂️ Image Analysis Workspace

The analysis workflow is stateless from the transport's perspective: no hidden MCP session is used. analyze_image returns a random, high-entropy analysis_id (img_…); later inspect_image calls pass it back explicitly. The vision API stays stateless — every request rebuilds its context (current image or crop + overview summary + relevant region + last 2–3 turns + current question).

首次看到图片        → analyze_image(mode=overview)   → 整体结构 + regions
选中某个区域        → inspect_image(region_id|region) → 局部追问
小字/金额/代码/表格  → inspect_image(mode=extract)     → 逐字读取

Region crops: normalized coordinates → pixel coordinates → sharp crop → Lanczos upscale → capped data URI. Out-of-bounds regions are clamped; crop failures fall back to the full image. AI super-resolution is intentionally not used.
Context window: history is truncated (last 2–3 turns) and the full base64 is never stored per turn — only the source path (remote images are cached to a temp dir with TTL).
Lifecycle: sessions expire after VISION_ANALYSIS_TTL_MINUTES (default 30), capped at VISION_ANALYSIS_MAX_SESSIONS (default 100), swept automatically, and temp files are removed when the process exits or the workspace is released.
Security: local paths can be restricted with VISION_ALLOWED_DIRS; remote fetches are http(s)-only with a 20 MB cap; API keys never appear in tool results.
Cross-model continuity: the workspace is local to the server, so switching Gemini Lite → Flash → Gemma → SiliconFlow on 429 keeps working with the same analysis_id.

🤖 Supported Models

Model	Provider	Image	Video	Notes
Qwen3-VL	Self-hosted / API	✅	✅	Recommended. Full multimodal support
GPT-4o	OpenAI	✅	❌	Strong image analysis
LLaVA	Self-hosted	✅	❌	Open-source alternative
InternVL	Self-hosted	✅	⚠️	Strong multilingual OCR

Any model served via vLLM, Ollama, LMDeploy, or other OpenAI-compatible servers should work.

Supported formats: JPEG, PNG, GIF, WebP, BMP, SVG | MP4, AVI, MOV, MKV, WebM

📁 Project Structure

vision-mcp-pool/
├── src/
│   ├── index.ts              # MCP server entry point
│   ├── config.ts             # Configuration loader
│   ├── types.ts              # TypeScript type definitions
│   ├── analysis/
│   │   ├── analysis-types.ts # AnalysisSession / Region / OverviewResult types
│   │   ├── analysis-store.ts # In-memory workspace store (TTL, eviction, temp files)
│   │   ├── image-transform.ts# sharp crop / upscale / data URI, source resolution
│   │   └── prompts.ts        # Default overview prompt & context reconstruction
│   ├── tools/
│   │   ├── analyze-image.ts  # overview / focus / extract + workspace create/reuse
│   │   ├── inspect-image.ts  # region-aware follow-up
│   │   ├── release-analysis.ts
│   │   ├── analysis-common.ts# shared schemas, result builder, region helpers
│   │   ├── ocr-image.ts
│   │   ├── compare-images.ts
│   │   └── analyze-video.ts
│   └── utils/
│       ├── api-client.ts     # Provider pools: only HTTP 429 advances tiers
│       ├── file-handler.ts   # Local file → base64
│       └── json.ts           # Extract JSON objects from model output
├── test/                     # Unit + hermetic MCP tests (npm test)
├── scripts/                  # Manual real-API tests (not in npm test)
├── package.json
├── tsconfig.json
├── .env.example
└── LICENSE

📄 License

MIT

🇨🇳 中文说明

功能

analyze_image — 使用视觉语言模型分析图像；支持 overview（整体结构+区域）、focus（局部）、extract（逐字读取）三种模式，并维护带 analysis_id 的分析工作区
inspect_image — 基于 analysis_id 追问图片细节，支持 region_id 或归一化 region 定位，mode=extract 用于小字/金额/代码/表格
release_analysis — 释放工作区，删除缓存的图片与分析历史
ocr_image — OCR 文字识别，支持纯文本、Markdown、JSON 输出
compare_images — 对比 2–4 张图像，识别差异和相似之处
analyze_video — 分析视频内容（需要 Qwen3-VL 等支持视频的模型）

典型流程：analyze_image(mode=overview) 先拿到整体结构与区域 → 用返回的 analysis_id 调 inspect_image 追问指定区域 → 需要精确文字时用 mode=extract。该流程不依赖隐式 MCP session，analysis_id 由工具显式返回与接收，429 降级换模型后仍可继续。

与 PaddleOCR MCP 的分工

本服务擅长图像理解（看图说话、UI/截图分析、区域追问、图像对比、视频分析），其中的 ocr_image / mode=extract 适合一般文字提取。精确 OCR 与复杂版面还原（密集表格、公式、多页 PDF、扫描件）请使用独立的 PaddleOCR MCP（PP-StructureV3），两者互补：本服务"看懂图"，PaddleOCR"读准字"。

快速开始

git clone https://github.com/xiaobright/vision-mcp-pool.git
cd vision-mcp-pool
npm install && npm run build

配置 .env：

VISION_BASE_URL=http://your-server:port/v1/chat/completions
VISION_MODEL=Qwen3-VL-32B
VISION_API_KEY=your-api-key

在 Claude Code 的 ~/.mcp.json 中添加：

{
  "mcpServers": {
    "vision": {
      "command": "node",
      "args": ["/path/to/vision-mcp-pool/dist/index.js"],
      "env": {
        "VISION_BASE_URL": "http://your-server:port/v1/chat/completions",
        "VISION_MODEL": "Qwen3-VL-32B",
        "VISION_API_KEY": "your-api-key"
      }
    }
  }
}

将 /path/to/vision-mcp-pool 替换为实际安装路径。

🌐 国内访问 Google Gemini API

Gemini API 在国内无法直连。推荐通过 Vercel + 自有域名部署反向代理（Serverless Function），在 vercel.json 中指定美国节点实现国内直连：

// vercel.json
{ "regions": ["iad1"] }

// api/[[...path]].ts  (Vercel Serverless Function)
export default async function (req: Request) {
  const url = new URL(req.url);
  const target = `https://generativelanguage.googleapis.com${url.pathname}${url.search}`;
  const headers = new Headers(req.headers);
  return fetch(target, { method: req.method, headers, body: req.body });
}

相比 Cloudflare Worker（anycast 边缘网络，无法指定特定国家节点），Vercel 可固定部署区域，代理更稳定可控。将部署后的域名填入 VISION_BASE_URL 即可。

Acknowledgments

Forked from Loveacup/vision-mcp-server. 主要增强：

Gemini 原生 API 支持（非仅 OpenAI 兼容格式）
多账号模型号池机制，充分利用各模型免费额度（lite/flash/gemma/emergency 四级降级）
自动降级与重试（5xx/429 自动切换 profile，JSON 解析容错）
PaddleOCR 分工文档（精确 OCR 场景配合 paddleocr-api）

Name		Name	Last commit message	Last commit date
Latest commit History 12 Commits
opencode/skills/vision-ocr		opencode/skills/vision-ocr
scripts		scripts
src		src
test		test
.env.example		.env.example
.gitattributes		.gitattributes
.gitignore		.gitignore
LICENSE		LICENSE
METHODOLOGY.md		METHODOLOGY.md
README.md		README.md
claude-code-config.example.json		claude-code-config.example.json
config.example.json		config.example.json
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json

Navigation Menu

Repository files navigation

👁️ Vision MCP Server

✨ Features

🚀 Quick Start

1. Install

2. Configure

3. Run

🔌 Claude Code Integration

⚙️ Provider Routing

⚙️ Configuration Reference

🛠️ Tools Reference

`analyze_image`

`inspect_image`

`release_analysis`

`ocr_image`

`compare_images`

`analyze_video`

🗂️ Image Analysis Workspace

🤖 Supported Models

📁 Project Structure

📄 License

🇨🇳 中文说明

功能

与 PaddleOCR MCP 的分工

快速开始

🌐 国内访问 Google Gemini API

Acknowledgments

About

Releases

Packages

Contributors

Languages

Folders and files

Latest commit

History

Repository files navigation

👁️ Vision MCP Server

✨ Features

🚀 Quick Start

1. Install

2. Configure

3. Run

🔌 Claude Code Integration

⚙️ Provider Routing

⚙️ Configuration Reference

🛠️ Tools Reference

analyze_image

inspect_image

release_analysis

ocr_image

compare_images

analyze_video

🗂️ Image Analysis Workspace

🤖 Supported Models

📁 Project Structure

📄 License

🇨🇳 中文说明

功能

与 PaddleOCR MCP 的分工

快速开始

🌐 国内访问 Google Gemini API

Acknowledgments

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`analyze_image`

`inspect_image`

`release_analysis`

`ocr_image`

`compare_images`

`analyze_video`