Your private LLM inference hub — local hardware, rented GPUs, or managed APIs. One TUI. One proxy. Zero vendor lock-in.
The main menu — launch endpoints, browse models, run diagnostics, all from your terminal.
You've got a local GPU, a Vast.ai account, maybe a Together AI key — but juggling three different CLIs, SSH tunnels, and config files is a pain. LocalRouter unifies all of that behind a single interactive TUI and a transparent local proxy (localhost:8888). Your clients never know which backend is active.
pip install localrouter # or: git clone + pip install -e .
localrouter # launch the TUI
That's it. No YAML configs to write, no Docker required for local mode, no separate tunnel scripts to remember.
| Feature | What it does | |
|---|---|---|
| 🖥️ | Local inference | Run GGUF models on your own hardware — Vulkan, ROCm, CUDA, or CPU via llama.cpp |
| ☁️ | Vast.ai rental | One-click GPU rental: pick a tier (4090 → 8×H100) → recipe → geo → launch |
| ⚡ | vLLM serving | Tensor-parallel serving for 1T+ MoE models across multi-GPU clusters |
| 🤝 | Together AI | Connect 229+ managed models, hot-swap providers mid-session |
| 🔀 | Unified proxy | localhost:8888 routes to whichever provider is active — swap backends, not code |
| 📊 | Usage tracking | Per-provider cost logging (JSONL), session totals, rate-limit monitoring |
| 🔍 | Batch compare | Send the same prompt to multiple providers, see results & metrics side-by-side |
| 🧙 | Launch wizard | Guided flow: GPU tier → model recipe → geo preference → offer → spin up |
| ✏️ | Recipe editor | TUI-based editor for recipes, GPU tiers, and docker images — no manual TOML editing |
| 🔧 | Deep diagnostics | SSH probes, download speed checks, stall detection & automatic recovery |
# From PyPI (when published) or from source:
git clone https://github.com/buckster123/LocalRouter.git
cd LocalRouter
pip install -e .
# Now you have the `localrouter` command globally🖥️ Local mode — no API keys, no cloud, just your GPU
Point recipes at your GGUF models and go:
# Prerequisites: compiled llama.cpp + GGUF models in ~/models/
localrouter # TUI → Local → Launch → pick recipeThe TUI auto-discovers:
llama-serverbinaries fromPATHor~/llama.cpp/build*/bin/- GGUF files in
~/models/(configurable) - Available backends (Vulkan, ROCm/HIP, CUDA, CPU)
☁️ Vast.ai GGUF mode — rent a GPU, get an endpoint
pip install vastai # one-time
vastai set api-key <your-key> # from console.vast.ai
localrouter # TUI → Launch → Vast GGUFThe wizard walks you through: GPU tier → model → geo → offer → launch. Use Watch to follow boot progress, Tunnel to forward locally.
⚡ vLLM mode — tensor-parallel for massive MoE models
For models too large for llama.cpp — like DeepSeek V4 Pro (1.6T params) across 4-8 GPUs:
localrouter # TUI → Launch → pick a vLLM recipevLLM recipes auto-configure tensor parallelism, FlashInfer attention, FP8 KV cache, and chunked prefill. The launch script detects GPU count and sets everything up.
Pre-configured for DeepSeek V4 Pro and Flash on H100, H200, B200, and A100 clusters.
🤝 Together AI — managed inference, 229+ models
mkdir -p ~/.vastai-gguf
cat > ~/.vastai-gguf/config.toml << 'EOF'
[provider.together]
api_key = "sk-xxxxx"
base_url = "https://api.together.ai/v1"
EOFThe TUI picks it up automatically — browse models, pin your choice, hot-swap.
Client (curl / Python / Hermes / …)
│
▼
localhost:8888 ← always the same URL
│
┌────┴─────┐
│ Proxy │ ← routes to active provider
└────┬─────┘
│
┌─────┼──────┬──────────┐
▼ ▼ ▼ ▼
Local Vast vLLM Together
(llama) (GGUF)(tensor) (managed)
OpenAI-compatible API: /v1/chat/completions, /v1/completions, health check at /health.
Switch providers from the TUI — clients don't change a thing.
Everything is driven by recipes.toml — 70 pre-configured recipes across 19 GPU tiers and 4 providers. Edit recipes from the TUI (Editor menu) or directly in TOML:
# Local GPU
[[recipes]]
name = "local-qwen35-9b"
provider = "local"
label = "Qwen3.5-9B Q4_K_M (local Vulkan)"
model_path = "~/models/Qwen3.5-9B-Q4_K_M.gguf"
port = 8100
ctx = 32768
backend = "vulkan"
# Rented GPU via Vast.ai (llama.cpp)
[[recipes]]
name = "qwen36-35b-h100"
label = "Qwen3.6-35B-A3B Q8_0 128K ctx"
gpu = "h100-sxm"
model_repo = "unsloth/Qwen3.6-35B-A3B-GGUF"
model_quant = "Q8_0"
ctx = 131072
# vLLM tensor-parallel (multi-GPU clusters)
[[recipes]]
name = "dsv4-pro-5xh200"
provider = "vllm"
label = "DSv4-Pro 1.6T FP4+FP8 384K ctx (5×H200)"
gpu = "h200-sxm-5x"
model_id = "deepseek-ai/DeepSeek-V4-Pro"
ctx = 393216
image_type = "vllm"
kv_cache_dtype = "fp8"
reasoning_parser = "deepseek_r1"
# Managed API
[[recipes]]
name = "together-qwen3-32b"
provider = "together"
label = "Qwen3-32B (managed)"
model_id = "Qwen/Qwen3-32B"From consumer cards to datacenter clusters:
| Tier | VRAM | Use case |
|---|---|---|
| RTX 4090 | 24 GB | Budget GGUF (Qwen 9-27B) |
| RTX 5090 | 32 GB | Sweet spot for 27-35B GGUF |
| RTX PRO 6000 | 96 GB | Large models, huge context |
| H100 SXM | 80 GB | Datacenter single-GPU |
| 2×–4× H100 | 160–320 GB | DSv4-Flash GGUF |
| 2×–5× H200 | 282–705 GB | DSv4-Flash/Pro via vLLM |
| 4×–8× B200 | 768+ GB | DSv4-Pro at full quality |
| Menu item | What it does |
|---|---|
| Launch | Guided wizard: Local / Vast GGUF / vLLM / Together → spin up |
| Local | Manage local llama.cpp endpoints (launch / status / logs / stop) |
| Providers | Configure API keys and base URLs |
| Together | Browse Together AI models, pin a choice |
| Batch | Compare multiple providers side-by-side |
| Watch | Live boot watcher — polls status + logs until healthy |
| Diagnose | Usage stats, rate limits, SSH probes, stall detection |
| Instances | List active Vast instances, reattach |
| HF Browse | Browse HuggingFace model files, pin a quant |
| Editor | Recipes, GPU tiers, docker images — full CRUD with validation |
| Tunnel | SSH tunnel: up / status / down / logs |
| Smoke | Provider-aware smoke tests (health, completion, tools, throughput) |
| Proxy | Start/stop the local proxy on localhost:8888 |
| Destroy | Tear down current Vast instance |
Running with an H100 SXM on Vast.ai — tunnel up, endpoint healthy, 3 slots.
localrouter/
├── menus/
│ ├── main.py # Main menu loop
│ ├── local_menus.py # Local llama.cpp management
│ ├── vast_menus.py # Vast.ai + vLLM launch wizard
│ ├── provider_menus.py # Provider config & switching
│ ├── editor_menus.py # Recipe/tier/image editor TUI
│ └── tool_menus.py # Tunnel, proxy, diagnostics
├── config.py # Settings & recipe loading
├── recipe_editor.py # TOML read/write, recipe CRUD, validation
├── providers.py # Provider abstraction layer
├── proxy.py # Transparent proxy server
├── local_endpoint.py # Local llama-server lifecycle
├── vast_ops.py # Vast.ai API operations
├── hf_browser.py # HuggingFace model browser
├── cost.py # Cost tracking & rate limits
└── helpers.py # Shared utilities
~5,000 lines of Python. 18 modules. pip install -e . and you're done.
Every completion is logged to ~/.vastai-gguf/usage.log (JSONL):
{"ts":"2026-05-02T20:15:32","provider":"vast_gguf","model":"Qwen3.6-35B-A3B-Q8_0",
"prompt_tokens":42,"completion_tokens":128,"cost":0.0012}Local inference logs at $0. View stats in the Diagnose screen or programmatically:
python3 -c "from usage_tracker import format_summary; print(format_summary(24))"- Vast.ai:
launch.sh/launch_vllm.shbind to127.0.0.1:8000inside the container. Access only via SSH tunnel (tools/vast_tunnel.sh) or the local proxy. - Local: Binds to
127.0.0.1by default — never exposed to the network. - Config: API keys stored in
~/.vastai-gguf/config.toml(not in the repo).
| Dependency | Required for |
|---|---|
| Python 3.10+ | Everything |
questionary, rich, tomli_w |
TUI + recipe editor (auto-installed) |
vastai CLI |
Vast.ai mode only |
| llama.cpp (compiled) | Local mode only |
| Together AI API key | Together mode only |
aiohttp |
Proxy server (pip install localrouter[proxy]) |
MIT — do what you want with it.