JAiRouter

AI 模型服务统一网关

一行代码接入 Ollama、vLLM、GPUStack 等所有模型服务

中文 • 文档 • English • 讨论

---

Get started

docker run -d --name jairouter -p 8080:8080 sodlinken/jairouter:latest

Access Web Console

Open http://localhost:8080/admin in your browser.

Default Login Credentials:

• Username: admin
• Password: ChangeMeOnFirstStartup123456 (development default)

💡 Security Note: For production, set a strong password via environment variable:

docker run -d --name jairouter -p 8080:8080 \
  -e INITIAL_ADMIN_PASSWORD="YourSecurePassword!" \
  sodlinken/jairouter:latest

That's it. No configuration needed.

---

What is JAiRouter?

JAiRouter is a unified gateway for AI model services that provides:

• OpenAI-compatible API — Use any OpenAI SDK, zero code changes
• Smart Load Balancing — Automatic request distribution with health checks
• Rate Limiting & Circuit Breaking — Protect your services from overload
• Web Console — Visual management for services, instances, and monitoring

┌─────────────────────────────────────────────────────────────────┐
│                         Your Application                         │
│                    (OpenAI SDK / LangChain / HTTP)              │
└─────────────────────────────────┬───────────────────────────────┘
                                  │
                                  ▼ OpenAI API
┌─────────────────────────────────────────────────────────────────┐
│                         JAiRouter                                │
│  ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐   │
│  │ Router  │ │Balance  │ │Limit    │ │Circuit  │ │  Auth   │   │
│  └─────────┘ └─────────┘ └─────────┘ └─────────┘ └─────────┘   │
└─────────────────────────────────┬───────────────────────────────┘
                                  │
       ┌──────────────┬───────────┼───────────┬──────────────┐
       ▼              ▼           ▼           ▼              ▼
   ┌───────┐     ┌───────┐   ┌───────┐   ┌───────┐     ┌───────┐
   │Ollama │     │ vLLM  │   │GPUStack│  │Xinference│  │OpenAI │
   └───────┘     └───────┘   └───────┘   └───────┘     └───────┘

---

Why JAiRouter?

Without JAiRouter	With JAiRouter
Configure each model endpoint separately	Single unified API endpoint
Manual failover when services go down	Automatic circuit breaker
Implement auth for each service	JWT + API Key built-in
Scattered logs and metrics	Centralized dashboard
Restart service to update config	Web console, hot reload

---

Features

Feature	Description
🔌 OpenAI Compatible	All /v1/* endpoints work with OpenAI SDKs
⚖️ Load Balancing	Round-robin, weighted, least-connections, IP-hash, consistent-hash
🛡️ Rate Limiting	Token bucket, leaky bucket, sliding window algorithms
🔥 Circuit Breaker	Auto failover with configurable thresholds
🔐 Authentication	JWT + API Key dual authentication
📊 Observability	Prometheus metrics, OpenTelemetry tracing
💾 Persistence	Redis / File storage for cluster deployment
🎛️ Web Console	Dashboard, service management, version control

---

Supported Backends

Backend	Chat	Embedding	Rerank	TTS	STT	Image
Ollama	✅	✅	-	-	-	-
vLLM	✅	✅	-	-	-	-
GPUStack	✅	✅	✅	✅	✅	✅
Xinference	✅	✅	✅	✅	✅	✅
LocalAI	✅	✅	-	✅	✅	✅
OpenAI	✅	✅	-	✅	✅	✅

---

Quick Start

1. Start JAiRouter

# Using Docker
docker run -d --name jairouter -p 8080:8080 sodlinken/jairouter:latest

# Using Docker Compose
curl -O https://raw.githubusercontent.com/Lincoln-cn/JAiRouter/master/docker-compose.yml
docker compose up -d

2. Access Web Console

Open http://localhost:8080 in your browser.

Default credentials:

• Username: admin
• Password: Check logs with docker logs jairouter | grep password

3. Configure Your First Service

Via Web Console or API:

# Add an Ollama instance
curl -X POST http://localhost:8080/api/config/instance/add/chat \
  -H "Content-Type: application/json" \
  -d '{
    "name": "llama3.2",
    "baseUrl": "http://localhost:11434",
    "path": "/v1/chat/completions",
    "weight": 1
  }'

4. Make API Calls

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed"  # JAiRouter handles auth
)

response = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

---

Documentation

Resource	Link
📖 Full Documentation	https://jairouter.com
📘 API Reference	http://localhost:8080/swagger-ui
🚀 Deployment Guide	https://jairouter.com/en/deployment/
🔧 Configuration	https://jairouter.com/en/configuration/
📊 Monitoring	https://jairouter.com/en/monitoring/

---

Benchmarks

Performance comparison with direct Ollama access:

Scenario	Direct Ollama	Via JAiRouter	Overhead
Single request	1.2s	1.21s	<1%
100 concurrent	45s	48s	~6%
With rate limiting	N/A	Configurable	-
With circuit breaker	N/A	Auto failover	-

Benchmarks run on: Ubuntu 22.04, 16 cores, 32GB RAM, Ollama 0.1.27

---

Comparison

Feature	JAiRouter	Nginx	One-API	LangChain
OpenAI Compatible	✅	❌	✅	✅
Load Balancing	✅	✅	✅	❌
Rate Limiting	✅	✅	✅	❌
Circuit Breaker	✅	❌	❌	❌
Web Console	✅	❌	✅	❌
Config Hot Reload	✅	❌	✅	❌
Version Control	✅	❌	❌	❌
Distributed Tracing	✅	❌	❌	✅

---

Roadmap

• Core gateway functionality
• Multiple backend adapters
• Load balancing & rate limiting
• Circuit breaker
• Web management console
• JWT + API Key authentication
• Distributed tracing
• Configuration version control
• Plugin system for custom adapters
• GraphQL API support

LTS Release: v2.6.11 is the final Long-Term Support version, maintained until 2028-05.

---

Contributing

We welcome contributions! Please see our Contributing Guide.

# Clone and build
git clone https://github.com/Lincoln-cn/JAiRouter.git
cd JAiRouter
mvn clean package -DskipTests

# Run locally
java -jar target/modelrouter.jar

---

Support

• Documentation: https://jairouter.com
• Issues: GitHub Issues
• Discussions: GitHub Discussions

---

License

JAiRouter is released under the Apache 2.0 License.

---

Star ⭐ this repo if you find it useful!

Made with ❤️ by the JAiRouter Team