End-to-end documentation to set up your own local & fully private LLM server on Debian. Equipped with chat, web search, RAG, model management, MCP servers, image generation, and TTS.

The operations layer for your local LLM stack

A robust, production-ready Python toolkit to automate the synchronization between a directory of .gguf model files and a llama-swap config.yaml

LLM routing proxy for coding harnesses. Auto-routes to cloud or local inference via Bonsai LLM classification. Fallback, prompt rewriting, MCP code review, Signal/Discord Remote Communication

Add image-generation models to Open WebUI as selectable models, routed through llama-swap (ComfyUI and OpenAI-style backends).

GGUF Model Manager for llama-swap — Browse HuggingFace, download, manage models in one place.

Auto-configure opencode to use a local llama-swap instance with model and context detection

Config-driven local LLM toolkit for llama.cpp and llama-swap, with a FastAPI Web UI, eval/benchmark helpers, and deployment packaging.

Desktop control center for llama.cpp and llama-swap. Configure GGUF models, manage runtimes, and monitor local inference without editing YAML.

Run GGUF language models locally with llama.cpp, llama-swap, Vulkan GPU acceleration, automatic model discovery, systemd integration, and an OpenAI-compatible API.

Custom Llama Swap Container Image

llama-swap config for MTP speculative decoding on AMD RX 7900 XTX (ROCm). Qwen3.6-35B-A3B-MTP and Gemma 4 26B-A4B-MTP with VRAM-tuned context sizing.

Real-time, zero-dependency dashboard for a llama-swap server — live model, token throughput, context-window usage, GPU/system stats, and streaming logs in a single page.

Launch and optimize llama.cpp servers automatically across Linux, macOS, and Windows using hardware detection and configuration tuning.

Autonomous overnight LLM eval pipeline for local GGUF models — multi-turn agentic tasks, dimension-routed dual-judge scoring, SQLite-backed comparison reports. Built for llama.cpp + llama-swap on dual-GPU rigs.

One OpenAI endpoint for local LLM engines — with cross-engine memory management

One OpenAI-compatible endpoint in front of a heterogeneous local inference fleet. Routes each request to the node that already has the model warm in VRAM, with capability filtering, health failover, and per-node load caps. Built on LiteLLM and llama-swap.

One-click Windows setup for llama-swap: LLMs + GPU apps on one graphics card, with tray control and self-verifying updates

Start/stop your Llama Swap models with ulauncher

FLAI is a self-hosted, privacy-first AI platform. Local assistant for chat, voice, image/video gen, doc Q&A & camera analysis. Open source, GPU-optimized, multi-user with request queuing. Data never leaves your machine.