llmconduit

LLM API gateway for local and OpenAI-compatible chat-completions backends.

It accepts OpenAI Responses, OpenAI Chat Completions, and Anthropic Messages requests, normalizes them, and forwards them to an upstream /v1/chat/completions server. It can also run server-side tools such as Brave Search.

Build

cargo build --release

Configure

./target/release/llmconduit configure

The default config path is:

~/.config/llmconduit/config.yaml

Minimal config:

bind_addr: "127.0.0.1:4000"
upstream_base_url: "http://127.0.0.1:8000/v1"
upstream_model: "Qwen3.5"

Multi-upstream model routing:

upstreams:
  - name: "local"
    upstream_base_url: "http://127.0.0.1:8000/v1"
  - name: "openrouter"
    upstream_base_url: "https://openrouter.ai/api/v1"
    upstream_api_key: "..."

When upstreams is configured, llmconduit exposes the ordered union of the primary upstream model catalogs. If a request omits model, passes a blank model, or requests a model that is not currently available, llmconduit uses the first model from the first upstream with a catalog entry. Requested model names are normalized against the catalogs, so aliases such as different case or punctuation route to the exact model id exposed by the backend. If multiple upstreams expose the same model id, the first upstream wins.

Optional nested fallback providers:

upstreams:
  - name: "local"
    upstream_base_url: "http://127.0.0.1:8000/v1"
    fallback_upstreams:
      - name: "backup"
        upstream_base_url: "https://openrouter.ai/api/v1"
        upstream_api_key: "..."
        upstream_model: "openai/gpt-4.1-mini"
        exposed_model: "GPT-4.1-mini"
        upstream_chat_kwargs:
          provider:
            order:
              - z-ai
            allow_fallbacks: true

If a selected upstream fails before producing the first chat chunk, only that upstream's nested fallback_upstreams are tried. llmconduit does not treat the next model-routing upstream as a failure fallback. Fallback models are not shown in /v1/models unless exposed_model is set. A fallback upstream_model is optional; when set, fallback requests use that model, otherwise they keep the routed primary model id. exposed_model advertises a fallback model under a client-facing alias and routes requests for that alias to the declaring fallback provider. Fallback upstream_chat_kwargs are merged only when that fallback is selected, with per-model kwargs and explicit request values taking precedence.

The legacy top-level upstream_* and fallback_upstreams settings still work when upstreams is not configured.

Global and per-model request defaults:

system_prompt_prefix: |
  Shared instructions prepended to every request.

upstream_chat_kwargs:
  stream_reasoning: true

model_profile_templates:
  thinking:
    separate_reasoning: true
    chat_template_kwargs:
      enable_thinking: true

model_profiles:
  GLM-5.1:
    extends:
      - thinking
    chat_template_kwargs:
      clear_thinking: false

  Kimi-K2.6:
    extends:
      - thinking
    system_prompt_prefix: |
      Extra Kimi-specific instructions.
    chat_template_kwargs:
      preserve_thinking: true

system_prompt_prefix is prepended to all Responses, Chat Completions, and Anthropic Messages requests. A profile-specific prefix is appended after the global prefix. upstream_chat_kwargs merge in this order: top-level defaults, matched model profile templates, matched model profile, then explicit request values. In model profiles and templates, extra profile-level keys are shorthand for upstream chat kwargs; the explicit upstream_chat_kwargs wrapper still works and overrides the shorthand when both set the same key. When a profile extends multiple templates, the extends list is applied in declaration order: later entries override earlier ones, and the profile's own fields override all templates.

Reserved * profile

A profile keyed * is a pure fallback for per-model settings. When a request names a model that no specific model_profiles entry matches, the * profile stands in as that model's profile: its upstream_chat_kwargs and system_prompt_prefix apply. When a specific profile DOES match, the * profile is not consulted at all - an explicit match never inherits unset fields from *. The * profile can itself extend templates, so extending a shared template is the way to give * and explicit profiles common defaults. Use model_profile_templates (extends) to share fields between explicit profiles, not *.

Per-model profile matching precedence, highest to lowest:

1. The request model - matched by name (case-insensitive) against model_profiles.
2. The resolved/upstream model (after upstream_model rewriting) - matched by name.
3. The reserved * profile - used only when neither 1 nor 2 matches.

Top-level config is the base below all profiles: upstream_chat_kwargs is the deep-merge base, and system_prompt_prefix is always prepended. Client request values still override profile settings, as described above.

model_profiles:
  # Fallback for any model without an explicit profile.
  "*":
    upstream_chat_kwargs:
      chat_template_kwargs:
        enable_thinking: true

  GLM-5.2:
    upstream_chat_kwargs:
      chat_template_kwargs:
        enable_thinking: false

With this config, a request for GLM-5.2 uses only the GLM-5.2 profile (enable_thinking: false); the * profile contributes nothing. A request for any other model (e.g. Qwen-3) falls back to * (enable_thinking: true).

Model capabilities

A profile's capabilities block overrides the Anthropic model capabilities advertised on /v1/models for Anthropic clients.

model_profiles:
  "*":
    capabilities:
      thinking:
        types: [adaptive, enabled]
      effort:
        levels: [max, xhigh, high, medium, low, minimal, none]
      image_input: false

• supported is the only knob and defaults to true. The simple caps (batch, citations, code_execution, image_input, pdf_input, structured_outputs) accept a bare bool as shorthand for {supported: <bool>}.
• thinking.types, effort.levels, and context_management.features list the advertised sub-entries; each inherits the cap's supported flag.
• Unknown cap keys, effort levels, thinking types, and context-management features are rejected at load.
• A configured cap replaces the base (upstream-supplied, else the default capabilities) for that cap key, wholesale; unconfigured caps keep the base. A matched profile without a capabilities block gets no fill-in from the * profile. Caps resolve per upstream id: an id-keyed profile, else the first alias whose upstream_model targets the id, else the reserved * profile.

Optional Brave Search:

brave_api_key: "..."

Run

./target/release/llmconduit start

Useful flags:

./target/release/llmconduit start --raw
./target/release/llmconduit start --with-debug-ui

The gateway listens on http://127.0.0.1:4000 by default.

Codex

[model_providers.llmconduit]
name = "llmconduit"
base_url = "http://127.0.0.1:4000/v1"
wire_api = "responses"
requires_openai_auth = false

[profiles.llmconduit]
model_provider = "llmconduit"
model = "Qwen3.5"

codex -p llmconduit "what files are in this directory?"

Docker

docker build -t llmconduit .
docker run --rm -p 4000:4000 \
  --add-host=host.docker.internal:host-gateway \
  -e LLMCONDUIT_UPSTREAM_BASE_URL=http://host.docker.internal:8000/v1 \
  llmconduit

Endpoints

Endpoint	Description
POST /v1/responses	OpenAI Responses API
POST /v1/chat/completions	OpenAI Chat Completions API
POST /v1/messages	Anthropic Messages API
GET /v1/models	Proxied model list
GET /healthz	Health check
GET /debug	Debug UI when started with --with-debug-ui

Environment

Common overrides:

LLMCONDUIT_BIND_ADDR
LLMCONDUIT_UPSTREAM_BASE_URL
LLMCONDUIT_UPSTREAM_API_KEY
LLMCONDUIT_UPSTREAM_MODEL
LLMCONDUIT_SYSTEM_PROMPT_PREFIX
LLMCONDUIT_UPSTREAM_CHAT_KWARGS_JSON
LLMCONDUIT_UPSTREAM_FAILURE_COOLDOWN_SECS
LLMCONDUIT_BRAVE_MAX_RESULTS
LLMCONDUIT_REQUEST_TIMEOUT_SECS
LLMCONDUIT_CONNECT_TIMEOUT_SECS
LLMCONDUIT_MAX_WEB_SEARCH_ROUNDS
LLMCONDUIT_MAX_REPLAY_ENTRIES
LLMCONDUIT_FLATTEN_CONTENT
BRAVE_SEARCH_API_KEY
OPENAI_API_KEY

OPENAI_API_KEY is used as a fallback upstream API key.

Request Logs

Set this in config to write upstream chat requests as JSONL:

upstream_request_log_path: "/tmp/llmconduit-upstream.jsonl"

Then inspect prefix stability:

llmconduit analyze-log

Test

cargo test

License

MIT