Production-ready realtime voice AI agent built on FastAPI, Anthropic Claude Opus 4.7, Twilio Media Streams, Deepgram Nova-3 STT (speech-to-text), and ElevenLabs TTS (text-to-speech). Handles inbound PSTN (public switched telephone network) calls and browser WebRTC (Web Real-Time Communication), runs an agentic tool-use loop with prompt caching and adaptive thinking, and persists every turn to PostgreSQL.
- PSTN voice in/out via Twilio Programmable Voice + Media Streams (μ-law 8kHz over WebSocket)
- Browser WebRTC endpoint for low-latency demos
- Streaming STT with Deepgram Nova-3 (multilingual: English + Thai)
- LLM (large language model) Claude Opus 4.7 with adaptive thinking, prompt caching on the system prompt and tool definitions, and a manual agentic tool-use loop tuned for sub-second voice latency
- Streaming TTS with ElevenLabs (eleven_turbo_v2_5)
- Real barge-in — assistant playback runs as a cancellable
asynciotask; a partial user transcript interrupts it mid-chunk - Tool use — appointment availability, booking, SMS (short message service) confirmation
- Persistence — call sessions, transcript turns, tool calls (PostgreSQL + SQLAlchemy 2.0 async + Alembic)
- Hardening — Twilio webhook HMAC (hash-based message authentication code) validation, per-process
CallGatecaps concurrent calls, fail-fast settings validation in production - Observability — structlog JSON logs, request-ID middleware with contextvars correlation across async boundaries, per-stage latency (STT endpointing, LLM TTFT (time-to-first-token), tool call, TTS TTFB (time-to-first-byte))
- Containerized — multi-stage Dockerfile (non-root, healthcheck), docker-compose for local dev (Postgres + Redis + one-shot Alembic migration)
- CI (continuous integration) — code quality (ruff format + lint + mypy
--strict), tests with coverage, docker build — GitHub Actions - Typed pipeline —
LLMClient/STTClient/TTSClientProtocols make every external service swappable for fakes in tests
PSTN ──▶ Twilio ──▶ /voice/incoming (TwiML)
│
▼
/voice/stream (WebSocket, μ-law 8kHz)
│
┌───────────────────┼────────────────────────┐
▼ ▼ ▼
Audio buf Deepgram WS ElevenLabs
(μ-law⇄PCM) (streaming STT) (streaming TTS)
│ ▲
▼ │
Orchestrator ──▶ Claude Opus 4.7
│ (adaptive thinking,
│ prompt caching,
│ tool use loop)
▼
PostgreSQL (sessions, turns, tool calls)
See docs/ARCHITECTURE.md for the deep dive, and docs/COST.md for monthly cost-of-ownership estimates at different call volumes.
- Python 3.12+
- Docker & docker compose
- A Twilio number with Programmable Voice
- Deepgram, ElevenLabs, Anthropic API keys
- A public HTTPS tunnel to your laptop for Twilio webhooks —
ngrok, Cloudflare Tunnel
(
cloudflared tunnel --url http://localhost:8000), or any equivalent.
Env selection happens one step before the app runs — the Dockerfile (or
the make env helper) copies the right per-env template to .env, and the
app always reads plain .env. Templates use ${VAR:-default} expansion so
real env vars (shell, CI/CD, Kubernetes secret, AWS Secrets Manager) always
win over the file default.
# layout:
# .env.development — safe defaults, committed (works out of the box)
# .env.staging — staging template, committed
# .env.production — production template, committed (no secrets!)
# .env — what the app reads (gitignored, generated by the step below)
Local (venv) dev:
make env # defaults to ENV=development
# or: make env ENV=stagingDocker / compose: the .env.${ENV} template is baked into the image
via a Dockerfile ARG. Default is development; switch with an env var:
docker compose up --build # development
ENV=production docker compose up --build # bakes .env.productionStaging / production secrets come from the real environment (shell,
CI/CD, Secrets Manager, Kubernetes) — the ${VAR:-default} shape means
you never commit real values:
export ANTHROPIC_API_KEY=sk-ant-...
export DEEPGRAM_API_KEY=...
export ELEVENLABS_API_KEY=...
export TWILIO_ACCOUNT_SID=...
export TWILIO_AUTH_TOKEN=...
export TWILIO_FROM_NUMBER=+1...
export PUBLIC_BASE_URL=https://your-tunnel-host
export DATABASE_URL=postgresql+asyncpg://...
export REDIS_URL=redis://...In ENV=production, startup aborts with an explicit list of missing
secrets if any are absent — no silent degradation on the first inbound call.
docker compose up --build
# Compose order: db → migrate (alembic upgrade head, exits 0) → app
# API: http://localhost:8000
# Health: http://localhost:8000/health
# Docs: http://localhost:8000/docsPoint your Twilio number's Voice → A Call Comes In webhook to (HTTP POST):
https://<your-tunnel-host>/voice/incoming
Twilio webhook signatures are HMAC-verified against PUBLIC_BASE_URL, so
that value must match the tunnel URL exactly (including https://).
Dial your Twilio number. The agent answers, listens, and books an appointment.
The same orchestrator backs wss://<host>/webrtc/signal for low-latency
browser demos (PCM16 16kHz mono, base64 frames). Useful for showing the
pipeline without owning a phone number.
make env # copy .env.development -> .env (or: make env ENV=staging)
make install # create .venv and install app + dev deps
docker compose up -d db redis
make migrate # alembic upgrade head
make run # uvicorn --reloadmake test # pytest
make lint # ruff format --check + ruff check
make typecheck # mypy --strict
make fmt # ruff format + ruff --fix.pre-commit-config.yaml wires the same gates (plus hadolint and gitleaks)
into a git hook — run pre-commit install once per checkout.
.
├── app/
│ ├── main.py # FastAPI app, lifespan, exception handlers, /health
│ ├── config.py # pydantic-settings (SecretStr, fail-fast validators)
│ ├── logging.py # structlog JSON logger
│ ├── middleware.py # request-ID + contextvars binding + access log
│ ├── security.py # Twilio webhook HMAC validation
│ ├── concurrency.py # CallGate: semaphore + nowait acquire
│ ├── routers/
│ │ ├── twilio.py # POST /voice/incoming + WS /voice/stream
│ │ ├── webrtc.py # WS /webrtc/signal
│ │ └── sessions.py # GET /sessions/{call_sid}
│ ├── pipeline/
│ │ ├── orchestrator.py # per-call coordinator + cancellable TTS task for barge-in
│ │ ├── stt_deepgram.py # Deepgram Nova-3 streaming WS client (STTClient)
│ │ ├── llm_claude.py # Anthropic SDK, manual tool loop, caching (LLMClient)
│ │ ├── tts_eleven.py # ElevenLabs streaming TTS client (TTSClient)
│ │ └── audio.py # μ-law ⇄ PCM16 conversion
│ ├── tools/
│ │ ├── registry.py # name → spec + handler map
│ │ ├── check_availability.py
│ │ ├── book_slot.py
│ │ └── send_confirmation.py # Twilio SMS (no-op without creds)
│ ├── persistence/
│ │ ├── db.py # async engine + session_scope()
│ │ ├── models.py # CallSession, TranscriptTurn, ToolCallRecord
│ │ └── repositories.py # SessionRepository façade
│ └── prompts/system.md # cached system prompt
├── migrations/ # Alembic env + versions/
│ └── versions/0001_initial_schema.py
├── tests/ # pytest (audio, tools, llm wiring, health, orchestrator)
├── docs/ARCHITECTURE.md # pipeline, latency budget, design rationale
├── .github/workflows/ci.yml # quality → test → docker build
├── .pre-commit-config.yaml # ruff, mypy, hadolint, gitleaks
├── Dockerfile # multi-stage, non-root, healthcheck
├── docker-compose.yml # db + redis + one-shot migrate + app
├── Makefile # install / fmt / lint / typecheck / test / run / migrate
├── alembic.ini
└── pyproject.toml # deps + ruff + pytest + mypy config
Opus 4.7 ships with adaptive thinking — the model decides when to reason vs. respond fast. For voice (where latency matters more than the last 5% of reasoning quality), this beats fixed budget_tokens (which is also no longer accepted on 4.7).
The system prompt and the tool JSON schemas don't change between turns. We mark them with cache_control: {"type": "ephemeral"} so every follow-up turn in the same call reads them at the cached rate (~10× cheaper, faster TTFT).
The SDK's tool_runner is great for batch agents, but for voice we need to:
- start streaming TTS as soon as the assistant emits text (before tool calls finish)
- log per-tool-call latency for observability
- short-circuit if the user starts speaking again (barge-in)
- Sub-300ms partial transcripts
- Multilingual (English + Thai in one stream)
- Confidence scores per word for adaptive backchannel timing
MIT — see LICENSE.