A local-first, multi-tenant customer-intake chatbot for small businesses — Telegram bot + web chat + admin dashboard, powered entirely by a local LLM (llama.cpp / Gemma 4 E2B). No cloud APIs, no data ever leaves your machine.
Built around a real small business (Mike's Plumbing Solutions) and hardened against prompt injection with a 3,940-test suite (1,830 good-path + 2,110 attacks). The production model passes 99.97%.
- Telegram bot + customer web chat — intake, emergency triage, multilingual replies, voice and photo understanding
- Admin dashboard — live conversation viewer (SSE push), agent takeover
- Multi-tenant — one install serves several businesses, each with its own system prompt, SQLite DB, and uploads
- Fully local — text + vision + audio inference in a llama.cpp container; runs on AMD, Nvidia, or CPU
- Attack-hardened — the prompt is validated against prompt injection, social engineering, encoding tricks, and vision attacks
main.py wires:
Store (SQLite) → LlamaModel (llama.cpp container) → Bot (Telegram) → Admin (HTTP/SSE) → WebChat (HTTP)| Component | File | Port | Role |
|---|---|---|---|
| Entry point | src/.../main.py |
— | Wires all components |
| Model | model.py |
53658 | llama.cpp server manager (text/vision/audio) |
| Store | store.py |
— | SQLite + live SSE event bus |
| Telegram bot | telegram_bot/ |
— | Long-poll bot with admin takeover |
| Admin | admin.py |
61301 | Dashboard, conversation viewer |
| Web chat | web_chat.py |
59628 | Customer-facing chat UI |
┌──────────┐ ┌─────────────┐ ┌───────────────────┐ ┌───────────────┐
│ Telegram │ ─ │ plumber-bot │ ─ │ llama.cpp (model) │ ─ │ Gemma 4 E2B │
│ / Web │ │ container │ │ container │ │ GGUF (local) │
└──────────┘ └─────────────┘ └───────────────────┘ └───────────────┘
│
SQLite + uploads (./data, bind-mounted)podman(ordocker) — runs the bot + model containers- A GGUF model (~3 GB; see below)
- RAM: ~6 GB free for CPU; a GPU (AMD/Nvidia) strongly recommended for usable speed
- For voice messages:
ffmpeg(already in the bot container)
./scripts/download-model.sh # Bartowski Q4_K_M (~3.3 GB) — easy default, 99.85% qualityThis is a pre-built, ungated GGUF. For the production model (QAT IQ3_XXS + speculative decoding, 99.97%), see Model options.
cp .env_sample .env
# edit .env: set TELEGRAM_BOT_TOKEN (from @BotFather) and the two secrets# AMD GPU (ROCm) — author's setup; builds a custom llama-rocm image first (see below)
./scripts/start-llama.sh --backend rocm --model gemma4-e2b-q4km
# Nvidia GPU (CUDA) — uses the official llama.cpp CUDA image
./scripts/start-llama.sh --backend cuda --model gemma4-e2b-q4km
# CPU only (including Docker on Mac ARM) — official llama.cpp CPU image (linux/arm64)
./scripts/start-llama.sh --backend cpu --model gemma4-e2b-q4kmMac Apple Silicon (M-series) — native Metal GPU. Docker Desktop on Mac can't pass the Metal GPU into Linux containers. For the best performance, run
llama-servernatively outside Docker (this gives you Metal GPU acceleration). Once it's running on port 53658, start the bot container with--backend cpu(the bot connects via localhost — the Metal-hosted model is indistinguishable from the Docker one):# 1. Install llama.cpp natively brew install llama.cpp # 2. Start the model server with Metal GPU llama-server \ -m models/gguf/gemma4-e2b-q4km/gemma-4-E2B-it-Q4_K_M.gguf \ --mmproj models/gguf/gemma4-e2b-q4km/mmproj-BF16.gguf \ -ngl 999 -c 8192 --host 127.0.0.1 --port 53658 --parallel 1 # 3. Start the bot container (CPU backend — connects to native model on :53658) ./scripts/start-llama.sh --backend cpu --model gemma4-e2b-q4km
--model is required — there is no silent default. It names a subdir under models/gguf/ (gemma4-e2b-q4km for the downloaded preset). Speculative decoding is automatic only when the domain-distilled draft model is present — that draft is a build artifact (research/07-distillation), not downloadable, so a fresh install runs plain Q4_K_M. To run the production stack (QAT IQ3_XXS + speculative decoding, 99.97%), build the model per research/12-qat-q3 + research/15-spec-decoding and pass --model iq3xxs-candidate.
The first run also builds the plumber-bot image. When up:
- Admin dashboard: http://127.0.0.1:61301 (password =
ADMIN_PASSWORD) - Web chat: http://127.0.0.1:59628
- Telegram: message your bot
Stop everything with ./scripts/stop.sh (add --all to also stop model containers).
AMD (ROCm) one-time build. The
rocmbackend uses a custom image tuned for gfx1151 (Strix Halo). Build it once:./llama-rocm/build.sh # needs llama.cpp source; see llama-rocm/DockerfileThe
cudaandcpubackends pull official images automatically — no build step.
CPU tuning. The defaults (
LLAMA_CTX=65536,LLAMA_CACHE_RAM=5120) assume ample memory. On a smaller CPU box, lower them, e.g.:LLAMA_CTX=8192 LLAMA_CACHE_RAM=1024 ./scripts/start-llama.sh --backend cpu --model gemma4-e2b-q4km
All configuration is via environment variables (load them in .env, or pass -e to the container). Key variables:
| Variable | Required | Default | Purpose |
|---|---|---|---|
TELEGRAM_BOT_TOKEN |
yes | — | Bot token from @BotFather |
ADMIN_PASSWORD |
yes | — | Admin dashboard login |
WEB_SESSION_SECRET |
yes | — | Web chat session signing (use a random string) |
TELEGRAM_ALLOWED_CHAT_IDS |
no | (all chats) | Restrict the bot to specific chat IDs (CSV) |
DEFAULT_TENANT |
no | mikes_plumbing |
Tenant used when a chat isn't explicitly routed |
CHAT_ROUTING |
no | — | chat_id:tenant_id,... routing |
BACKEND |
no | rocm |
rocm | cuda | cpu (also a --backend flag) |
LLAMA_CTX / LLAMA_CACHE_RAM |
no | 65536 / 5120 | llama.cpp context size + prompt-cache RAM (MB) |
See .env_sample for a copy-paste template.
# 1. Smoke test the running model stack (text + vision + audio) — needs a running model on 53658
python3 scripts/smoke_test.py --url http://127.0.0.1:53658
# 2. Unit tests (no model needed)
python3 -m pytest tests/ -v
# 3. Full 3,940-test hardened-prompt suite — needs a running model on 53658
cd bench
python3 run_tests.py --backend llama-gemma4-e2b-q4km # full 3940 (~25 min)
python3 run_tests.py --backend llama-gemma4-e2b-q4km --limit 50 # quick sample
python3 run_tests.py --backend llama-gemma4-e2b-q4km --category prompt_injectionThe bench suite's --backend is just a label — it always hits whatever model is serving on port 53658. Test cases live in bench/tests/ (good/ and attack/).
The smoke test references fixture files under
data/uploads/. On a fresh clone those won't exist; the checks that need them are skipped, and text/vision checks that don't need local files still run.
Add your own business (tenant). Copy a tenant config and edit it:
cp src/plumber_bot/shared/tenants/mikes_plumbing.yaml \
src/plumber_bot/shared/tenants/your_company.yamlEach tenant YAML defines tenant_id, company_name, contact details, greeting, intake fields, and the full system_prompt. Point the bot at it with DEFAULT_TENANT=your_company (or CHAT_ROUTING).
Edit the prompt. The system prompt lives in the tenant YAML and is seeded into data/system_prompt.txt on first run. Editing the YAML and bouncing the bot re-seeds it.
Served ports. Override with flags: --admin-port, --web-port, --model-port.
| Model | Size | Quality | Speed (author HW) | How to get it |
|---|---|---|---|---|
| Bartowski Q4_K_M (easy default) | 3.3 GB | 99.85% (3934/3940) | ~66 tok/s | ./scripts/download-model.sh |
| QAT IQ3_XXS + spec decoding (production) | 2.5 GB | 99.97% (3939/3940) | ~93 tok/s | build via research/12-qat-q3 + research/15-spec-decoding |
| QAT Q4*K_XL (Unsloth) *(rollback)_ | 2.5 GB | 99.82% | ~74 tok/s | research/01-unsloth-qat |
All target Google Gemma 4 E2B (multimodal: text + vision + audio). The production model also runs a 380 MB domain-distilled Qwen 0.5B as a speculative-decoding draft (1.43× faster, zero quality risk).
This repo doubles as a reproducible record of how the model was squeezed onto local hardware. Full notes in research/; condensed findings:
| Approach | Quality | Speed | Verdict |
|---|---|---|---|
| QAT IQ3_XXS + imatrix + spec decoding | 3939/3940 (99.97%) | 93 tok/s | ✅ Current prod |
| Spec decoding (domain-distilled 0.5B draft) | +1 test | 1.43× faster | ✅ Zero-risk speedup |
| Importance-aware quant (imatrix) | 3934/3940 | 67 tok/s | ✅ Best per-bit precision |
Concurrent batching (--parallel 16) |
— | ~2–3× aggregate | ✅ Under load |
| Prefix caching (system prompt) | — | 9× warm TTFT | ✅ Already optimal |
| Cascade router (distill fast-tier) | 3929/3940 | ~1.6× fast-path | |
| Pruning (all 4 methods) | — | 0% speedup | ❌ Memory-bound HW |
| LoRA / full fine-tune on QAT weights | 3.7% / 0% | — | ❌ Catastrophic forgetting |
Key takeaway: on memory-bound hardware, the wins come from importance-aware quantization, speculative decoding, concurrent batching, and prefix caching — not from pruning or fine-tuning. See research/README.md for the full study.
sme_bot/
├── src/plumber_bot/ # the bot (config, model, store, telegram, admin, web chat, tenants)
├── scripts/ # download-model.sh, start-llama.sh, stop.sh, smoke_test.py
├── llama-rocm/ # AMD ROCm llama.cpp image (build.sh + Dockerfile)
├── Dockerfile # bot container
├── bench/ # 3940-test hardened-prompt suite + runner
├── tests/ # unit tests
├── research/ # quantization/pruning/distillation/spec-decoding studies (reproducible)
├── docs/ # background reading + reference benchmarks
├── models/ # GGUF weights (gitignored — download separately)
└── data/ # per-tenant SQLite + uploads (gitignored — runtime state)MIT © Yang Ye.
The bundled model (Gemma 4 E2B) is subject to Google's Gemma terms — download and use of the weights is governed by those terms, not this license.