llama-dash

llama-dash turns a self-hosted local inference box into an observable, policy-controlled AI gateway: one UI for model state, request history, API keys, routing rules, proxy metrics, and client setup. The implemented inference backend is currently llama-swap over llama.cpp.

It is the single public entrypoint for OpenAI-compatible and Anthropic-compatible clients. llama-dash owns proxy policy, logging, auth, routing, and backend normalization, your selected inference backend owns local model processes and inference when traffic is routed to local models.

OpenAI SDK / Claude Code / Continue / Open WebUI
                    │
                    ▼
              llama-dash :3000
      dashboard · auth · logs · routing · metrics
             │                     │
             ▼                     ▼
      llama-swap :8080         direct /v1 upstreams
  llama.cpp models · peers      OpenAI · Anthropic

✨ What it does

• Watch the box — live request, token, model, upstream, GPU, and update status in one dashboard.
• Manage models — load/unload models, inspect per-model stats and capability metadata, view residency history, and edit llama-swap config with validation.
• Proxy clients — expose one OpenAI/Anthropic-compatible /v1/* endpoint for local models, peers, direct upstreams, Claude Code, Continue, Open WebUI, and more.
• Track requests — searchable request history with filters, histograms, detail views, attribution metadata, token counts, and cost estimates.
• Control access — dashboard login, hashed API keys, per-key RPM/TPM limits, model allow-lists, MCP relay allow-lists, and per-key usage breakdowns.
• Enforce policy — routing rules for model rewrites, rejects, passthrough auth, direct HTTPS upstreams, encrypted credentials, system prompts, and global request size limits.
• Test models — playgrounds for chat, image, speech, and transcription, including article-to-speech extraction.
• Export ops data — raw log streams, retention controls, request auditing, and low-cardinality Prometheus metrics at /metrics.

Dashboard Live traffic, tokens, model residency, upstream and GPU health	Playground Chat against local endpoints with request/response inspection	Request detail Routing, attribution, latency, tokens, and payload metadata	Logs Raw llama-swap, proxy, and upstream streams in one viewer
Model detail Load history, stats, recent requests, and config context	Speech playground Read any article and audio testing	Policies Aliases, routing rules, passthrough auth, and request limits	Requests Searchable history with filters, sorting, and histogram

⚡ Quick start (Docker Compose)

Choose the compose file that matches your GPU vendor. Both setups use ./config/config.yaml for llama-swap config, ./models/ for model files, and expose llama-dash on http://localhost:3000.

First create your env and config files, then set the required secrets:

cp .env.example .env       # then set BETTER_AUTH_SECRET and CREDENTIAL_ENCRYPTION_KEY
cp config/config.example.yaml config/config.yaml   # edit models

The compose files load .env via env_file and won't start without it. Generate a session secret with openssl rand -base64 33. See the Environment table for all values.

AMD / ROCm

docker compose -f docker-compose.amd.yaml up -d

docker-compose.amd.yaml runs ghcr.io/mostlygeek/llama-swap:rocm, passes through /dev/kfd and /dev/dri, and also mounts /dev/dri into llama-dash so AMD GPU stats work in the dashboard. The config directory is mounted into both services so llama-dash can atomically save config.yaml and llama-swap can reload it through -watch-config.

NVIDIA / CUDA

docker compose -f docker-compose.nvidia.yaml up -d

docker-compose.nvidia.yaml runs ghcr.io/mostlygeek/llama-swap:cuda and requests gpus: all for the llama-swap service. This requires the NVIDIA Container Toolkit on the host. The config directory is mounted into both services so llama-dash can atomically save config.yaml and llama-swap can reload it through -watch-config.

🏗️ Manual setup

Requirements

• Node 24+
• pnpm
• A reachable llama-swap instance

Install

cp .env.example .env   # edit INFERENCE_BASE_URL to point at your instance
pnpm install
pnpm db:migrate        # creates data/dash.db
pnpm dev               # http://localhost:5173

🏔️ Environment

Copy .env.example to .env and fill in the values.

See the environment variables reference for the complete list of variables, defaults, and deployment notes.

✴️ Claude Code / Anthropic passthrough

Route any Anthropic SDK, including Claude Code, through llama-dash for logging, filtering, per-request inspection, and Anthropic subscription passthrough.

See the Claude Code & Anthropic passthrough guide for client setup, routing rules, subscription OAuth, and provider-key flows. For remote MCP servers, see the MCP relays guide.

🤖 Acknowledgements

This project was developed with significant assistance from LLMs. Architecture decisions, implementation, and documentation were all shaped through human-AI collaboration.

📝 License

MIT