voice-agent

A local-first real-time voice agent: speech → text → tool-using LLM → speech, with a reproducible end-to-end latency benchmark. STT and TTS run locally on your Mac; the LLM is pluggable (OpenAI gpt-4o-mini by default).

The benchmark is self-contained: it synthesizes its own test utterances with macOS say, so the latency numbers reproduce on any Apple-Silicon Mac with no recorded audio. Every reported number below is a real measurement committed as a JSON receipt in results/latency.json.

Architecture

  ┌────────────┐    ┌──────────────────────────┐    ┌────────────┐
  │   audio    │    │      LLM agent           │    │   speech   │
  │  (wav/mic) │    │  OpenAI gpt-4o-mini       │    │  (aiff)    │
  └─────┬──────┘    │  + tool-calling          │    └─────▲──────┘
        │           └──────────┬───────────────┘          │
        │ STT                  │ tools                     │ TTS
        ▼                      ▼                           │
  ┌────────────┐   ┌─────────────────────────┐   ┌─────────┴──────┐
  │faster-whisper│ │ get_current_time         │   │  macOS `say`   │
  │  base.en int8│ │ convert_units            │   │  (local)       │
  │   (local)    │ │ lookup_shipment_eta      │   └────────────────┘
  └────────────┘   └─────────────────────────┘

  stt.py            tools.py / llm.py / agent.py            tts.py
  └──────────── timing.py wraps every stage (p50/p95) ──────────────┘

Each stage is injected via a small protocol (Transcriber, a reply() object, Synthesizer), so the orchestration, tool dispatch, and percentile math are all unit-tested with no network and no audio device.

Quickstart

# Apple-Silicon macOS, Python 3.11+
uv venv && . .venv/bin/activate
uv pip install -e ".[dev]"          # faster-whisper, openai, pydantic, pytest, ruff
export OPENAI_API_KEY=sk-...

# Process one audio file (faster-whisper downloads base.en ~150MB on first run):
voice-agent --audio sample.wav

Output:

You said: How many kilometers are in 5 miles?
Agent:    There are approximately 8.05 kilometers in 5 miles.
Audio:    /var/folders/.../voice_agent_reply.aiff
Timings:  stt=317ms llm=2846ms tts=1077ms total=4240ms

Mic mode (live)

voice-agent --mic --seconds 5

Live mic capture is implemented and documented but not unit-tested — there is no audio device in CI. It needs sounddevice + soundfile (pip install sounddevice soundfile) and reuses the exact same agent pipeline as file mode, which the benchmark exercises end-to-end.

Latency

Measured by scripts/bench.py (8 representative utterances, 3 of which trigger tools) on this machine. Full per-utterance receipts: results/latency.json / results/benchmark.md.

• Machine: Apple M4, macOS 26.2, Python 3.13
• STT: faster-whisper base.en (int8, CPU) · LLM: OpenAI gpt-4o-mini · TTS: macOS say

Stage	p50 (ms)	p95 (ms)
STT	318	337
LLM (+tools)	1868	2895
TTS	1064	1220
End-to-end	3176	4275

Honest notes

• The LLM stage dominates and is the only networked stage; it's also the noisiest (p95 ~2.9 s vs p50 ~1.9 s). STT and TTS are local and stable. Swapping in a streaming or local LLM is the obvious next latency win.
• These are batch, non-streaming turn timings (full transcript → full reply → full audio render), not first-token / first-audio latency. They reproduce by running make bench.
• Numbers are specific to an Apple M4. A different Mac, model size, or network will differ — rerun the benchmark to get your own receipts; nothing here is hard-coded.

Reproduce the benchmark

make bench     # runs scripts/bench.py, rewrites results/latency.json + benchmark.md
make demo      # renders a sample utterance and runs one turn
make test      # ruff + pytest (no network, no audio)

Tools

The agent can call three demo tools (src/voice_agent/tools.py): get_current_time, convert_units (length), and lookup_shipment_eta (canned offline dataset). The benchmark confirms each runs end-to-end — e.g. the shipment replies contain data that exists only in the local tool dataset, proving the model's tool call was dispatched and fed back.

License

MIT © 2026 Mark Teji