Skip to content

feat: add LiteLLM as custom model provider#168

Open
RheagalFire wants to merge 1 commit into
deta:mainfrom
RheagalFire:feat/add-litellm-provider
Open

feat: add LiteLLM as custom model provider#168
RheagalFire wants to merge 1 commit into
deta:mainfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire
Copy link
Copy Markdown

@RheagalFire RheagalFire commented May 25, 2026

Summary

  • Adds LiteLLM as a custom model provider, enabling users to route AI requests through 100+ LLM providers (OpenAI, Anthropic, Azure, Bedrock, Vertex AI, Groq, Ollama, etc.) via a single LiteLLM proxy.
  • Follows the existing custom model pattern (same structure as Ollama, OpenRouter, HuggingFace).

Motivation

LiteLLM acts as a unified AI gateway. Instead of configuring individual provider API keys, users can point Surf at a single LiteLLM proxy that handles provider routing, load balancing, and cost tracking across 100+ providers. Useful for users who need providers not yet directly supported (e.g., AWS Bedrock, Vertex AI, Mistral, Cohere), or teams running a centralized LLM proxy.

Changes

  • packages/types/src/ai.types.ts - added LiteLLM = 'LiteLLM' to CUSTOM_MODELS enum, added LiteLLM entry to CUSTOM_MODEL_DEFINITIONS with default proxy URL, model docs link, and API key docs link
  • packages/icons/src/lib/Icons/LiteLLM.svelte - new icon component (bullet train, matching LiteLLM's brand)
  • packages/icons/src/lib/main.ts - registered LiteLLM icon in the icon map and exports

Tests

Live E2E - basic completion via LiteLLM proxy routing to Claude Sonnet 4.6:

=== Test 1: Basic completion ===
Status: 200
Model: claude-sonnet-4-6
Content: OK
Finish: stop

Live E2E - streaming via LiteLLM proxy:

=== Test 2: Streaming ===
Streamed: Hello!

SSE chunks parsed correctly through the same /v1/chat/completions endpoint Surf uses for Ollama and OpenRouter custom models.

Error handling - invalid model:

=== Test 3: Invalid model ===
Status: 200
Error handled: YES

All 3 tests use raw fetch("http://localhost:4000/v1/chat/completions") matching the exact call path Surf uses for custom models via OPEN_AI_PATH_SUFFIX.

Risk / Compatibility

  • Additive only. No existing models or providers modified.
  • No new dependencies - LiteLLM proxy is OpenAI-compatible, uses the same request/response format as Ollama and OpenRouter.
  • 3 files changed, 18 lines added.

Example usage

1. Start a LiteLLM proxy:

pip install litellm
litellm --model anthropic/claude-3-5-sonnet --port 4000

2. In Surf Settings > AI > Custom Model, select "LiteLLM"

3. Enter model name using LiteLLM's provider/model format:

openai/gpt-4o-mini           # OpenAI via LiteLLM
anthropic/claude-3-5-sonnet  # Anthropic via LiteLLM
azure/gpt-4                  # Azure via LiteLLM
bedrock/anthropic.claude-3   # AWS Bedrock via LiteLLM
vertex_ai/gemini-pro         # Vertex AI via LiteLLM
groq/llama-3.3-70b           # Groq via LiteLLM

4. Edit the proxy URL

https://my-proxy.example.com/v1/chat/completions   # remote proxy

5. Enter your LiteLLM proxy API key (if required)

Signed-off-by: RheagalFire <arishalam121@gmail.com>
@RheagalFire RheagalFire force-pushed the feat/add-litellm-provider branch from ebbeb75 to fa511f6 Compare May 25, 2026 22:01
@RheagalFire
Copy link
Copy Markdown
Author

cc @aavshr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant