Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@
[![GitHub Pages](https://img.shields.io/badge/docs-GitHub%20Pages-blue)](https://mixaill76.github.io/auto_ai_router/)
[![license](https://img.shields.io/github/license/MiXaiLL76/auto_ai_router.svg)](https://github.com/MiXaiLL76/auto_ai_router/blob/main/LICENSE)

High-performance proxy router for LLM APIs with automatic load balancing, rate limiting, and fail2ban protection. Routes requests to OpenAI, Vertex AI, Gemini AI Studio, Anthropic, Comet API, and other Auto AI Router instances.
High-performance proxy router for LLM and image APIs with automatic load balancing, rate limiting, and fail2ban protection. Routes requests to OpenAI, Vertex AI, Gemini AI Studio, Anthropic, Comet API, Sosana.art images, and other Auto AI Router instances.

## Key Features

- **Multi-provider support** — OpenAI, Vertex AI, Gemini, Anthropic, Comet API, Proxy chains
- **Multi-provider support** — OpenAI, Vertex AI, Gemini, Anthropic, Comet API, Sosana.art images, Proxy chains
- **Round-robin load balancing** — across multiple credentials per model
- **Rate limiting** — per-credential and per-model RPM/TPM controls
- **Fail2ban** — automatic provider banning on repeated errors
Expand Down
17 changes: 17 additions & 0 deletions config.yaml.example
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,15 @@ credentials:
rpm: 60
tpm: -1

# Sosana.art image generation API (OpenAI Images-compatible through the router)
# Async image tasks can run longer than short chat requests; keep request_timeout/write_timeout >= 2m.
- name: "sosana_images"
type: "sosana"
api_key: "os.environ/SOSANA_API_KEY"
base_url: "https://sosana.art"
rpm: 60
tpm: -1

# CheapGPT / AIProductiv using the Anthropic-compatible Messages API
- name: "cheapgpt_anthropic"
type: "anthropic"
Expand Down Expand Up @@ -113,6 +122,14 @@ models:
rpm: 100
tpm: 50000

# Sosana.art image model exposed through /v1/images/generations and /v1/images/edits.
# image_size selects banana-2-1k/2k/4k-compliant dynamically; default is 1K.
- name: "google/gemini-3.1-flash-image-preview"
model: "banana-2-{image_size}-compliant"
credential: sosana_images
rpm: 60
tpm: -1

# Comet API Claude aliases for public model names.
- name: "anthropic/claude-haiku-4.5"
model: "claude-haiku-4-5-20251001"
Expand Down
22 changes: 11 additions & 11 deletions docs/getting-started/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,17 +172,17 @@ Each credential defines a connection to an LLM provider. See [Providers](../prov

Common fields for all credentials:

| Field | Type | Description |
| ------------------ | ------ | ------------------------------------------------------------------------------------------- |
| `name` | string | Unique credential identifier |
| `type` | string | Provider type: `openai`, `anthropic`, `cometapi`, `vertex-ai`, `gemini`, `bedrock`, `proxy` |
| `rpm` | int | Requests per minute limit (-1 = unlimited) |
| `tpm` | int | Tokens per minute limit (-1 = unlimited) |
| `is_fallback` | bool | Use as fallback when primary credentials are exhausted |
| `reasoning_only` | bool | Route only requests that explicitly enable reasoning/thinking |
| `scopes` | list | Optional client scopes allowed to use and see this credential |
| `denied_scopes` | list | Optional client scopes that must not use or see this credential |
| `forbidden_scopes` | list | Alias for `denied_scopes` |
| Field | Type | Description |
| ------------------ | ------ | ---------------------------------------------------------------------------------------------------------------------- |
| `name` | string | Unique credential identifier |
| `type` | string | Provider type: `openai`, `anthropic`, `cometapi`, `sosana`, `vertex-ai`, `gemini`, `bedrock`, `proxy`, `air`, `proman` |
| `rpm` | int | Requests per minute limit (-1 = unlimited) |
| `tpm` | int | Tokens per minute limit (-1 = unlimited) |
| `is_fallback` | bool | Use as fallback when primary credentials are exhausted |
| `reasoning_only` | bool | Route only requests that explicitly enable reasoning/thinking |
| `scopes` | list | Optional client scopes allowed to use and see this credential |
| `denied_scopes` | list | Optional client scopes that must not use or see this credential |
| `forbidden_scopes` | list | Alias for `denied_scopes` |

### Scoped credential visibility

Expand Down
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ graph LR

## Features

- **Multi-provider routing** — OpenAI, Vertex AI, Gemini AI Studio, Anthropic, Comet API
- **Multi-provider routing** — OpenAI, Vertex AI, Gemini AI Studio, Anthropic, Comet API, Sosana.art images
- **AIR chains** — forward to other Auto AI Router instances as fallback
- **Round-robin balancing** — distribute load across multiple credentials
- **Two-level rate limiting** — per-credential RPM/TPM + per-model limits
Expand Down
1 change: 1 addition & 0 deletions docs/providers/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ Auto AI Router supports multiple LLM providers. Each provider type has its own a
| [OpenAI](openai.md) | `openai` | `api_key`, `base_url` | API Key |
| [Anthropic](anthropic.md) | `anthropic` | `api_key`, `base_url` | API Key |
| [Comet API](cometapi.md) | `cometapi` | `api_key`, `base_url` | API Key |
| [Sosana.art](sosana.md) | `sosana` | `api_key`, `base_url` | API Key via Bearer |
| [ProMan](proman.md) | `proman` | `api_key`, `base_url` | API Key |
| [AWS Bedrock](bedrock.md) | `bedrock` | `api_key`, `base_url` | Bearer Token |
| [Vertex AI](vertex.md) | `vertex-ai` | `project_id`, `location`, `credentials_file` or `credentials_json` | OAuth2 / Service Account |
Expand Down
96 changes: 96 additions & 0 deletions docs/providers/sosana.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
# Sosana.art

Sosana.art is supported as an image-only provider for the OpenAI-compatible
Images API. The router accepts `/v1/images/generations` and `/v1/images/edits`,
submits a Sosana Banana async task, polls it, and returns an OpenAI Images
response with `data[].b64_json`.

Chat Completions, Responses API, Embeddings, video, and slides are not routed to
Sosana in this integration.

## Configuration

```yaml
credentials:
- name: "sosana_images"
type: "sosana"
api_key: "os.environ/SOSANA_API_KEY"
base_url: "https://sosana.art"
rpm: 60
tpm: -1

models:
- name: "google/gemini-3.1-flash-image-preview"
model: "banana-2-{image_size}-compliant"
credential: sosana_images
rpm: 60
tpm: -1
```

The credential value is configured as `api_key`. The router sends it to Sosana
as `Authorization: Bearer <api_key>`, matching Sosana's API contract.

The dynamic model template maps `image_size` to Sosana's concrete image models:
`banana-2-1k-compliant`, `banana-2-2k-compliant`, and
`banana-2-4k-compliant`. If `image_size` is omitted, the router uses `1K`.
This integration maps Sosana only for `google/gemini-3.1-flash-image-preview`;
other image families should be served by their native providers or fallback
proxies.

Sosana Banana tasks are asynchronous and can take longer than short chat
completion requests. For production Sosana credentials, set the router
`request_timeout` and HTTP `write_timeout` to at least `2m`.

## Behavior

- `n` must be `1`.
- Requests selected to a Sosana credential are skipped when they require
controls that Sosana does not support. The router then tries another primary
credential for the same model and then the configured fallback proxy cascade.
If no compatible provider is available, the router returns a local 400.
- Default response format and `response_format: "b64_json"` return
`data[].b64_json`.
- `response_format: "url"` is not routed to Sosana because URL responses require
VSELLM-owned rehosting before they can hide Sosana storage. Another provider
may handle it through normal fallback routing.
- `image_size` may be omitted or set to `1K`, `2K`, or `4K`; `0.5K` is not
routed to Sosana. Pixel `size` values are accepted only when they match the
documented Gemini `image_size` + `aspect_ratio` table for `1K`, `2K`, or
`4K`.
- `/v1/images/edits` accepts PNG input images only, up to 14 files, and sends
them as `data:image/png;base64,...` values in Sosana `image_urls`.
- Mask images are not supported.
- Output is PNG. `output_format` may be omitted or set to `png`; other formats
are not routed to Sosana.
- The router sends `prompt_optimization: false` so Sosana returns `MODERATED`
instead of rewriting moderated prompts into safe alternatives.

On completion, Sosana returns a public object URL in `result_file_url`. The
router downloads that object only from allowed Sosana/CDN hosts, does not follow
redirects, does not forward the Sosana `Authorization` header, keeps the download
bounded to 32 MiB, verifies the body is PNG, and base64-encodes it into the
OpenAI-compatible JSON response. The upstream object URL is not returned to
clients.

## Billing

Sosana vendor prices are not used at request time and are not returned to
clients. Successful image requests log `ImageCount=1`; spend is calculated from
the internal price registry or LiteLLM model table using `output_cost_per_image`.
When `image_size` selects a concrete tier, the spend lookup uses the concrete
model first, for example `banana-2-2k-compliant`, and then falls back to the
public model name if no concrete price is configured.

## Error Masking

Sosana upstream HTTP errors and terminal task errors are masked before they are
returned to clients. The router preserves the appropriate HTTP status but
replaces provider details with neutral OpenAI-compatible error bodies.

For operator debugging, structured logs may include a truncated textual upstream
error body with `response_body_masked=true`. Raw image bytes and full result
URLs are not logged.

If Sosana is hidden behind another proxy credential, the upstream router should
propagate the credential marker used by this router so proxy-chain errors can be
masked as Sosana errors too.
8 changes: 5 additions & 3 deletions internal/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ const (
ProviderTypeGemini ProviderType = "gemini"
ProviderTypeAnthropic ProviderType = "anthropic"
ProviderTypeCometAPI ProviderType = "cometapi"
ProviderTypeSosana ProviderType = "sosana"
ProviderTypeProMan ProviderType = "proman"
ProviderTypeBedrock ProviderType = "bedrock"
ProviderTypeProxy ProviderType = "proxy"
Expand All @@ -42,7 +43,7 @@ func (p ProviderType) LogValue() slog.Value {
// IsValid checks if the provider type is valid
func (p ProviderType) IsValid() bool {
switch p {
case ProviderTypeOpenAI, ProviderTypeVertexAI, ProviderTypeGemini, ProviderTypeAnthropic, ProviderTypeCometAPI, ProviderTypeProMan, ProviderTypeBedrock, ProviderTypeProxy, ProviderTypeAIR:
case ProviderTypeOpenAI, ProviderTypeVertexAI, ProviderTypeGemini, ProviderTypeAnthropic, ProviderTypeCometAPI, ProviderTypeSosana, ProviderTypeProMan, ProviderTypeBedrock, ProviderTypeProxy, ProviderTypeAIR:
return true
}
return false
Expand All @@ -59,6 +60,8 @@ func normalizeProviderType(raw string) ProviderType {
switch strings.ToLower(strings.TrimSpace(raw)) {
case "comet-api", "comet_api":
return ProviderTypeCometAPI
case "sosana-art", "sosana_art":
return ProviderTypeSosana
case "aar", "auto-ai-router", "auto_ai_router":
return ProviderTypeAIR
case "pro-man", "pro_man":
Expand Down Expand Up @@ -791,7 +794,6 @@ func (c *CredentialConfig) UnmarshalYAML(value *yaml.Node) error {
if c.IsFallback, err = parseField(temp.IsFallback, false, strconv.ParseBool, "is_fallback for credential '"+c.Name+"'"); err != nil {
return err
}

// Copy models decoded via YAML anchors / inline definitions
c.Models = temp.Models

Expand Down Expand Up @@ -1646,7 +1648,7 @@ func (c *Config) Validate() error {

// Validate provider type
if !cred.Type.IsValid() {
return fmt.Errorf("credential %s: invalid type: %s (must be 'openai', 'vertex-ai', 'gemini', 'anthropic', 'cometapi', 'proman', 'bedrock', 'proxy', or 'air')", cred.Name, cred.Type)
return fmt.Errorf("credential %s: invalid type: %s (must be 'openai', 'vertex-ai', 'gemini', 'anthropic', 'cometapi', 'sosana', 'proman', 'bedrock', 'proxy', or 'air')", cred.Name, cred.Type)
}
if cred.AuthType != "" && cred.AuthType != "bearer" && cred.AuthType != "x-api-key" {
return fmt.Errorf("credential %s: invalid auth_type: %s (must be 'bearer' or 'x-api-key')", cred.Name, cred.AuthType)
Expand Down
102 changes: 102 additions & 0 deletions internal/config/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -830,6 +830,7 @@ func TestProviderType_IsValid(t *testing.T) {
{"openai", ProviderTypeOpenAI, true},
{"vertex-ai", ProviderTypeVertexAI, true},
{"cometapi", ProviderTypeCometAPI, true},
{"sosana", ProviderTypeSosana, true},
{"air", ProviderTypeAIR, true},
{"proman", ProviderTypeProMan, true},
{"invalid", ProviderType("azure"), false},
Expand Down Expand Up @@ -865,6 +866,33 @@ rpm: 60
assert.Equal(t, ProviderTypeCometAPI, cred.Type)
}

func TestCredentialConfig_NormalizeSosanaProviderType(t *testing.T) {
tests := []struct {
name string
raw string
}{
{"canonical", "sosana"},
{"dash alias", "sosana-art"},
{"underscore alias", "sosana_art"},
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var cred CredentialConfig
err := yaml.Unmarshal([]byte(`
name: sosana
type: `+tt.raw+`
api_key: key
base_url: https://sosana.art
rpm: 60
`), &cred)

require.NoError(t, err)
assert.Equal(t, ProviderTypeSosana, cred.Type)
})
}
}

func TestCredentialConfig_NormalizeAIRProviderTypeAliases(t *testing.T) {
tests := []struct {
name string
Expand Down Expand Up @@ -893,6 +921,44 @@ rpm: 60
}
}

func TestConfig_Validate_SosanaRequiresAPIKeyAndBaseURL(t *testing.T) {
tests := []struct {
name string
apiKey string
baseURL string
wantErr string
}{
{name: "valid", apiKey: "key", baseURL: "https://sosana.art"},
{name: "missing api key", baseURL: "https://sosana.art", wantErr: "api_key is required"},
{name: "missing base url", apiKey: "key", wantErr: "base_url is required"},
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cfg := &Config{
Server: ServerConfig{
Port: 8080,
MaxBodySizeMB: 10,
MasterKey: "test-key",
RequestTimeout: 30 * time.Second,
},
Credentials: []CredentialConfig{
{Name: "sosana", Type: ProviderTypeSosana, APIKey: tt.apiKey, BaseURL: tt.baseURL, RPM: 10},
},
Fail2Ban: Fail2BanConfig{MaxAttempts: 3},
}

err := cfg.Validate()
if tt.wantErr == "" {
assert.NoError(t, err)
} else {
require.Error(t, err)
assert.Contains(t, err.Error(), tt.wantErr)
}
})
}
}

func TestCredentialConfig_NormalizeProManProviderType(t *testing.T) {
var cred CredentialConfig
err := yaml.Unmarshal([]byte(`
Expand Down Expand Up @@ -1340,6 +1406,42 @@ monitoring:
assert.Equal(t, 2, cfg.Server.MaxProviderRetries, "Default MaxProviderRetries should be 2")
}

func TestLoad_ServerTimeoutDefaults(t *testing.T) {
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.yaml")

configContent := `
server:
port: 8080
max_body_size_mb: 10
master_key: "sk-test"

fail2ban:
max_attempts: 3
ban_duration: permanent
error_codes: [401]

credentials:
- name: "test"
type: "openai"
api_key: "sk-test"
base_url: "https://api.openai.com"
rpm: 10

monitoring:
prometheus_enabled: false
`
err := os.WriteFile(configPath, []byte(configContent), 0644)
require.NoError(t, err)

cfg, err := Load(configPath)
require.NoError(t, err)
assert.Equal(t, 60*time.Second, cfg.Server.RequestTimeout)
assert.Equal(t, 60*time.Second, cfg.Server.ReadTimeout)
assert.Equal(t, 60*time.Second, cfg.Server.WriteTimeout)
assert.Equal(t, 2*time.Minute, cfg.Server.IdleTimeout)
}

func TestLoad_MaxProviderRetries_Custom(t *testing.T) {
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.yaml")
Expand Down
Loading