feat(voice): add duplex TTS benchmarking and validated metrics

.gitignore

@@ -274,6 +274,11 @@ traces/
 benchmark_output/
 benchmark_outputs/
 microbench_output*
+capacity_search_output/
+
+# local/runtime files
+.env
+configs/
 
 # uv lock file
-uv.lock
+uv.lock

README.md

@@ -10,7 +10,7 @@ microbenchmarks. One tool, any workload.
 
 **From isolated requests to complex agentic sessions, Veeksha captures the full complexity of modern LLM workloads.**
 
-👉 **[Why Veeksha?](https://project-vajra.github.io/veeksha/getting_started/why_veeksha.html)** — Learn what sets Veeksha apart  
+👉 **[Why Veeksha?](https://project-vajra.github.io/veeksha/getting_started/why_veeksha.html)** — Learn what sets Veeksha apart
 📚 **[Documentation](https://project-vajra.github.io/veeksha)** — Full guides and API reference
 
 ## Quick start

configs/stt_vajra.yaml

@@ -0,0 +1,58 @@
+seed: 42
+output_dir: "benchmark_output/stt_vajra"
+
+client:
+  type: stt
+  provider: vajra_openai_realtime
+  api_base: "http://localhost:8003"
+  # Drives Vajra through the OpenAI-compatible realtime transcription WebSocket.
+  model: mistralai/Voxtral-Mini-4B-Realtime-2602
+  sample_rate: 16000
+  ws_chunk_size: 2048
+  ws_realtime_pacing: true
+  request_timeout: 7200
+
+session_generator:
+  type: trace
+  trace_file: traces/asr/aa_public/manifest.jsonl
+  wrap_mode: true
+  flavor:
+    type: audio
+    audio_dir: ""
+
+traffic_scheduler:
+  type: concurrent
+  target_concurrent_sessions: 8
+  rampup_seconds: 0
+
+evaluators:
+  - type: performance
+    target_channels: ["audio"]
+    stream_metrics: true
+    stream_metrics_interval: 10.0
+    slos:
+      # Capacity gate: p99 first-partial latency climbs once the engine can't
+      # keep up at 1x pacing. (RTF is >=1 by construction when paced.)
+      - name: "P99 time-to-first-chunk under 500ms"
+        metric: ttfc
+        percentile: 0.99
+        value: 500
+        type: constant
+
+runtime:
+  max_sessions: -1
+  benchmark_timeout: 300
+  post_timeout_grace_seconds: 120
+  num_client_threads: 2
+  num_dispatcher_threads: 2
+  num_completion_threads: 2
+
+trace_recorder:
+  enabled: false
+
+wandb:
+  enabled: false
+  project: "stt-benchmark"
+  run_name: "stt_vajra_openai_realtime_voxtral"
+  tags: ["stt", "vajra", "voxtral", "openai_realtime"]
+  log_artifacts: false

configs/stt_vllm_realtime.yaml

@@ -0,0 +1,57 @@
+seed: 42
+output_dir: "benchmark_output/stt_vllm_realtime"
+
+client:
+  type: stt
+  provider: vllm_realtime
+  api_base: "http://localhost:8025"
+  model: mistralai/Voxtral-Mini-4B-Realtime-2602
+  sample_rate: 16000
+  ws_chunk_size: 2048
+  ws_realtime_pacing: true
+  request_timeout: 7200
+
+session_generator:
+  type: trace
+  trace_file: traces/asr/aa_public/manifest.jsonl
+  wrap_mode: true
+  flavor:
+    type: audio
+    audio_dir: ""
+
+traffic_scheduler:
+  type: concurrent
+  target_concurrent_sessions: 8
+  rampup_seconds: 0
+
+evaluators:
+  - type: performance
+    target_channels: ["audio"]
+    stream_metrics: true
+    stream_metrics_interval: 10.0
+    slos:
+      # Capacity gate: p99 first-partial latency climbs once the engine can't
+      # keep up at 1x pacing. (RTF is >=1 by construction when paced.)
+      - name: "P99 time-to-first-chunk under 500ms"
+        metric: ttfc
+        percentile: 0.99
+        value: 500
+        type: constant
+
+runtime:
+  max_sessions: -1
+  benchmark_timeout: 300
+  post_timeout_grace_seconds: 120
+  num_client_threads: 2
+  num_dispatcher_threads: 2
+  num_completion_threads: 2
+
+trace_recorder:
+  enabled: false
+
+wandb:
+  enabled: false
+  project: "stt-benchmark"
+  run_name: "stt_vllm_voxtral"
+  tags: ["stt", "vllm_realtime", "voxtral"]
+  log_artifacts: false

docs/ASR_BENCHMARKING.md

@@ -0,0 +1,215 @@
+# ASR Benchmarking
+
+## Trace Generation
+
+Generate the public ASR trace with:
+
+```bash
+.venv/bin/python scripts/prepare_audio_traces.py --clips-per-dataset 128
+```
+
+This writes `traces/asr/aa_public/manifest.jsonl` plus WAV files under
+`traces/asr/aa_public/audio/`.
+
+`--max-duration` controls final clip length and defaults to 30 seconds. For
+timestamped clips, longer source files are split on word boundaries. With
+`--without-word-timestamping`, longer untimestamped clips are skipped.
+
+```bash
+.venv/bin/python scripts/prepare_audio_traces.py \
+  --clips-per-dataset 128 \
+  --max-duration 30
+```
+
+Use `--target-duration` to build clips around a requested duration instead of
+splitting by `--max-duration`. Source clips are repeated before NeMo alignment
+until they cover the target duration, then the aligned result is truncated to
+the requested length. `--target-duration` and `--max-duration` are mutually
+exclusive.
+
+```bash
+.venv/bin/python scripts/prepare_audio_traces.py \
+  --clips-per-dataset 128 \
+  --target-duration 600
+```
+
+Word-level reference timings for the interactivity metric are generated by
+default with NeMo forced alignment in Docker:
+
+```bash
+.venv/bin/python scripts/prepare_audio_traces.py --clips-per-dataset 128
+```
+
+To skip word timestamping:
+
+```bash
+.venv/bin/python scripts/prepare_audio_traces.py \
+  --clips-per-dataset 128 \
+  --without-word-timestamping
+```
+
+AMI can be prepared through the same script. The script downloads and caches the
+AMI manual word annotations and Mix-Headset WAVs under `benchmark_output/`:
+
+```bash
+.venv/bin/python scripts/prepare_audio_traces.py \
+  --datasets ami_word_timed
+```
+
+This writes `traces/asr/ami_word_timed/manifest.jsonl` plus WAV files under
+`traces/asr/ami_word_timed/audio/`.
+
+The trace uses the public Artificial Analysis cleaned datasets, VoxPopuli and
+Earnings22, as a recognizable external point of reference. Earnings22 examples
+are kept as full, potentially long requests rather than being chunked. The trace
+does not attempt to reproduce the full Artificial Analysis benchmark exactly:
+the proprietary AA-AgentTalk dataset is not available, their custom normalizer
+is not open sourced, and Veeksha measures behavior across engines so any point
+of difference is fine as long as it's consistent for all engines.
+
+## Request Metrics
+
+Request-level metrics are written to `request_level_metrics.jsonl`.
+
+Common audio metrics:
+
+- `ttfc`: time from the first audio byte sent to the first transcript delta
+  whose own payload carries text after control-token cleaning (or to the final
+  transcript when no such delta arrives), in milliseconds. Empty
+  progress/keepalive deltas do not count.
+- `end_to_end_latency`: time from client request start to request completion, in
+  milliseconds.
+- `generated_audio_duration`: audio duration represented by the request, in
+  milliseconds. For STT this is the input clip duration.
+- `rtf`: real-time factor, computed as
+  `end_to_end_latency / generated_audio_duration`.
+- `chunk_count`: number of transcript deltas observed, or `1` when only a final
+  transcript is returned.
+- `input_tokens`: whitespace token count of the final transcript.
+
+ASR-specific metrics:
+
+- `time_to_first_visible_text`: time from the first audio byte sent to the
+  first moment the assembled (concatenated then cleaned) transcript is
+  non-empty — when a user watching the live transcript would first see text —
+  in milliseconds. Usually equal to `ttfc`; the two differ only when cleaning a
+  single delta disagrees with cleaning the assembled transcript (e.g. a control
+  token split across deltas).
+- `time_to_first_partial`: time from the client sending EOF/commit for the audio
+  stream to the first non-empty partial transcript received after that point, in
+  milliseconds. This is only present when the provider emits such a partial.
+- `time_to_final_transcript`: time from the client sending EOF/commit for the
+  audio stream to the final transcript, in milliseconds.
+- `partial_transcript`: first non-empty post-EOF partial transcript used for
+  partial WER.
+- `final_transcript`: final transcript returned by the provider.
+- `expected_transcript`: reference transcript from the trace row.
+- `interactivity`: mean latency between when matched reference words finish in
+  the audio and when they first appear in the streamed client transcript, in
+  milliseconds. This requires `reference_word_timestamps` in the manifest and is
+  most meaningful with realtime audio pacing enabled.
+- `interactivity_word_count`: number of matched words used for the per-request
+  `interactivity` value.
+- `partial_wer`: WER for `partial_transcript` against `expected_transcript`.
+  Present when a partial transcript is available.
+- `final_wer`: WER for `final_transcript` against `expected_transcript`.
+  Present for every completed STT request with a reference transcript.
+
+## Replay Viewer
+
+For runs with `reference_word_timestamps` and `transcript_snapshots`, export a
+browser replay bundle:
+
+```bash
+.venv/bin/python tools/asr_replay_viewer/export_asr_replay.py \
+  benchmark_output/<run-dir>
+```
+
+Then serve the repo root and open the viewer:
+
+```bash
+.venv/bin/python -m http.server 8765 --bind 127.0.0.1
+```
+
+Open:
+
+```text
+http://127.0.0.1:8765/tools/asr_replay_viewer/?data=benchmark_output/<run-dir>/metrics/asr_replay.json
+```
+
+The viewer shows the ground-truth word timeline, the streamed transcript
+timeline, and an interactive latency graph. The graph loads ECharts from a
+pinned CDN URL at runtime, so the viewer needs network access when opened.
+
+## Partials and Finals
+
+Streaming providers may emit many transcript deltas before the audio stream is
+committed. Veeksha concatenates those deltas to track
+`time_to_first_visible_text` and the eventual transcript, but `partial_transcript` is specifically the first non-empty
+transcript observed after EOF/commit. `final_transcript` is the provider's final
+message when available, otherwise the concatenated deltas are used as a fallback.
+
+## Aggregate Metrics
+
+Aggregate metrics are emitted in the performance summary.
+
+General audio aggregates are percentile summaries over request-level values:
+
+- `ttfc (Mean/P50/P90/P99)`
+- `end_to_end_latency (Mean/P50/P90/P99)`
+- `generated_audio_duration (Mean/P50/P90/P99)`
+- `rtf (Mean/P50/P90/P99)`
+- `chunk_count (Mean/P50/P90/P99)`
+- `time_to_first_visible_text (Mean/P50/P90/P99)`, for STT runs when available
+- `time_to_first_partial (Mean/P50/P90/P99)`, for STT runs when available
+- `time_to_final_transcript (Mean/P50/P90/P99)`, for STT runs when available
+- `interactivity (Mean/P50/P90/P99)`, for STT runs with word-timestamped
+  references
+
+ASR WER aggregates:
+
+- `asr_final_sample_count`: number of scored ASR request samples.
+- `asr_final_sample_mean_wer`: unweighted mean WER across scored samples.
+- `asr_final_corpus_wer`: corpus-level WER from summed edit counts and summed
+  reference words.
+- `asr_final_duration_weighted_wer`: WER weighted by audio duration.
+- `asr_partial_*`: same aggregate modes for partial transcripts, over samples
+  where partial WER is available.
+- `asr_dataset_<dataset>_final_*` and `asr_dataset_<dataset>_partial_*`:
+  dataset-specific versions of the same metrics.
+
+For comparisons between serving engines, prefer the same aggregate metric across
+runs, with `asr_final_duration_weighted_wer` as the closest analogue to the
+duration-weighted WER used by public ASR leaderboards.
+
+## Manifest Fields
+
+Each manifest row represents one audio request. Key fields:
+
+- `session_id`: trace session identifier.
+- `audio_file`: WAV path relative to the manifest directory.
+- `dataset`: source dataset key, such as `aa_voxpopuli` or `aa_earnings22`.
+- `expected_transcript`: reference transcript used for WER.
+- `duration_s`: clip duration in seconds.
+- `sample_id`: unique row identifier.
+- `reference_word_timestamps`: optional list of reference word timings relative
+  to the request audio start, e.g.
+  `{"word": "hello", "start_ms": 120.0, "end_ms": 340.0}`.
+
+When the audio trace flavor sets `target_duration_s`, each row is trimmed at
+generation time: only the first `target_duration_s` seconds of audio are
+streamed (the STT client slices the decoded PCM using the
+`input_audio_start_ms` / `input_audio_end_ms` request metadata written by the
+generator), and `expected_transcript` / `reference_word_timestamps` are trimmed
+to the words that end within that prefix. Rows must provide
+`reference_word_timestamps`, and every clip must be at least
+`target_duration_s` long.
+
+## References
+
+- Artificial Analysis Speech to Text Methodology:
+  https://artificialanalysis.ai/speech-to-text/methodology
+- VoxPopuli-Cleaned-AA:
+  https://huggingface.co/datasets/ArtificialAnalysis/VoxPopuli-Cleaned-AA
+- Earnings22-Cleaned-AA:
+  https://huggingface.co/datasets/ArtificialAnalysis/Earnings22-Cleaned-AA

docs/getting_started/installation.rst

@@ -1,9 +1,10 @@
 Installation
 ============
 
-You can run Veeksha without explicit installation using ``uvx``.:
+You can run Veeksha without explicit installation using ``uvx``:
 
 .. code-block:: bash
+
    uvx -p 3.14t veeksha benchmark ...
 
 Veeksha requires free-threaded Python ``>=3.14``.

docs/index.rst

@@ -31,7 +31,7 @@ Key features
       as directed acyclic graphs with history inheritance, capturing real chat context accumulation
     - **Shared prefix testing**: Generate workloads with configurable prefix sharing to
       benchmark KV-cache efficiency
-    - **Trace replay**: Replay production traces (Claude Code, RAG, conversational) with
+    - **Trace replay**: Replay production traces (coding assistants, RAG, conversational) with
       preserved timing and token distributions
 
 **Flexible traffic generation**

docs/user_guide/index.rst

@@ -10,6 +10,7 @@ integrating with external tools.
    configuration
    output_files
    trace_flavors
+   tts_benchmarking
    capacity_search
    sweeps
    server_management