feat(voice): add duplex TTS benchmarking and validated metrics

[asmitks]

Excluding unit tests, the PR changes 22 production/documentation files. The core change is that Veeksha can now run the same client-observed TTS benchmark against Vajra, ElevenLabs, and Deepgram.

1. New ElevenLabs and Deepgram clients

Streaming WebSocket clients

veeksha/client/native_streaming_tts.py

Adds native protocol adapters for:

Client type | Provider endpoint | Input/output behavior -- | -- | -- elevenlabs_streaming_tts | ElevenLabs /stream-input | Paced text messages; base64 PCM responses deepgram_flux_streaming_tts | Deepgram /v2/speak | Paced Speak messages; binary PCM responses deepgram_aura_streaming_tts | Deepgram /v1/speak | Paced Speak, followed by Flush; binary PCM

trigger_to_first_playable_audio_ms is the intended steady-state TTFA. It excludes WebSocket setup and input time before the semantic synthesis trigger.

5. Fluidity and playback simulation

veeksha/evaluator/performance/audio_interactivity.py

The existing packet timeline is converted into fixed-duration playable PCM frames—20 ms by default. This removes dependence on how ElevenLabs, Deepgram, or Vajra happen to fragment WebSocket packets.

The Etalon-inspired algorithm:

1. Assigns the first playable frame a startup deadline.
2. Expects subsequent playable frames every 20 ms.
3. Lets early frames accumulate playback-buffer slack.
4. Lets late frames consume that slack.
5. Counts elapsed playback deadlines as misses if the frame is still late.
6. Resets slack after a miss.

fluidity = accepted playback deadlines / total playback deadlines

A score of 1.0 means the captured PCM supply could play continuously under the selected startup-buffer policy. It does not mean latency, quality, or naturalness was perfect.

The evaluator also computes:

• Stall count
• Total stall time
• Longest stall
• Minimum startup delay required to eliminate every underrun
• Stall-free fraction with 0/100/300 ms fixed delays
• Stall-free fraction after buffering 0/100/300 ms of audio
• Streaming RTF after the first audio chunk

User fluidity versus service fluidity

The implementation explicitly addresses input starvation.

• user_audio_fluidity_index: what the listener would experience, regardless of cause.
• tts_service_fluidity_index: emitted only when misses can reasonably be attributed to TTS.
• tts_service_fluidity_eligible: whether that attribution was valid.

In conservative mode, the service score is emitted only when all source text was available before playback began.

For controlled traces that guarantee sufficient synthesis-ready text, the benchmark can use:

fluidity_attribution_mode: source_oversupplied

This prevents the benchmark from blaming TTS for silence caused by an upstream LLM that did not produce text.

6. Duplex-overlap measurements

The evaluator now reports:

• audio_before_commit_ratio: fraction of audio received before the final text input.
• duplex_overlap_observed: whether a complete playable frame arrived before input completion.
• duplex_overlap_ms: how long playable audio delivery and text input overlapped.
• post_commit_audio_delivery_ms: audio delivery duration after the final input.

These answer two separate questions:

Did output actually overlap input?
How much useful overlap occurred?

7. Audio evaluator and output artifacts

veeksha/evaluator/performance/audio.py

The audio evaluator now:

• Parses the normalized provider timeline.
• Computes first-playable, duplex, buffering, stall and fluidity metrics.
• Aggregates them into percentile/CDF summaries.
• Retains provider/model/protocol labels.
• Writes the new metrics into request_level_metrics.jsonl.
• Optionally stores the complete audit timeline in audio_raw_timing.jsonl.
• Generates a stall-free-fraction-versus-startup-budget plot.
• Produces separate fluidity distributions for configured startup delays.

This means one captured request can be replayed locally under several playback-buffer policies without calling the provider again.

8. Configuration and registry plumbing

The following files make the new clients and metrics configurable from YAML:

• veeksha/config/client.py
• veeksha/client/registry.py
• veeksha/types/__init__.py
• veeksha/config/evaluator.py

They add:

• Five new client types.
• Provider model, voice, PCM, pacing and API-key settings.
• ElevenLabs chunk schedules, normalization, speed and auto mode.
• Deepgram Flux/Aura model and speed options.
• Fluidity frame duration.
• Fluidity startup buffer.
• Conservative/source-oversupplied attribution.
• Validation for invalid combinations.

9. SLO support

Files:

• veeksha/config/slo.py
• veeksha/slo/metrics.py
• veeksha/slo/slo.py

The new metrics can be used as benchmark pass/fail conditions.

Example:

- metric: trigger_to_first_playable_audio_ms
  percentile: 0.90
  value: 1000

For latency and stalls, lower is better. For the following metrics, higher is better:

• Fluidity
• Audio-before-commit ratio
• Duplex-overlap success
• Service-fluidity eligibility

This also fixes percentile direction. For example, a fluidity SLO should normally evaluate the lower tail, such as P1 ≥ 0.99, rather than P99.

10. Correctness validation changes

ASR WER aggregation

veeksha/evaluator/performance/asr.py

Clarifies and fixes empty-reference corpus WER:

• Empty reference + empty hypothesis → 0%
• Empty reference + inserted hypothesis words → 100%
• No samples → unavailable

The evaluator continues reporting:

• Sample-mean WER
• Corpus WER
• Duration-weighted WER

Corpus WER remains the primary ASR comparison.

TTS WER judge

Files:

• veeksha/config/verification.py
• veeksha/verification/audio.py

Whisper verification is now pinned with:

• Model
• Language
• Task=transcribe
• Beam size
• Device
• Compute type

This makes provider comparisons more reproducible and avoids automatic language detection changing the WER result.

With fail_on_threshold: true, verification now fails not only for bad WER but also for:

• Missing audio
• Transcription failure
• UTMOS failure
• Run-level verification errors

Previously, such failures could reduce the evaluated sample without necessarily failing the run.

11. Import/runtime cleanup

• librosa is loaded only when STT audio decoding is actually required.
• transformers is loaded only when a Hugging Face tokenizer is requested.
• Provider-native TTS benchmarks therefore avoid importing heavyweight model stacks merely by importing Veeksha.
• Pyright is explicitly configured for Python 3.14.

12. Documentation

docs/user_guide/tts_benchmarking.rst documents:

• Supported provider lanes
• All latency and fluidity definitions
• PCM frame normalization
• Buffering policies
• User versus service attribution
• Seed-TTS and ShareGPT trace sources
• Example YAML
• ASR versus TTS WER units
• UTMOS limitations
• The requirement to validate semantic duplex behavior

A small documentation cleanup also changes “Claude Code trace” to the generic “coding-assistant trace”; Veeksha does not bundle a Claude Code TTS trace.

Excluding unit tests, the PR changes 22 production/documentation files. The core change is that Veeksha can now run the same client-observed TTS benchmark against Vajra, ElevenLabs, and Deepgram.

1. New ElevenLabs and Deepgram clients

Streaming WebSocket clients

[veeksha/client/native_streaming_tts.py](https://github.com/project-vajra/veeksha/pull/203/files#diff)

Adds native protocol adapters for:

Client type	Provider endpoint	Input/output behavior
elevenlabs_streaming_tts	ElevenLabs /stream-input	Paced text messages; base64 PCM responses
deepgram_flux_streaming_tts	Deepgram /v2/speak	Paced Speak messages; binary PCM responses
deepgram_aura_streaming_tts	Deepgram /v1/speak	Paced Speak, followed by Flush; binary PCM

All three use the same shared measurement loop:

1. Open the WebSocket.
2. Send text at a configurable LLM-like rate such as 5 or 20 tokens/sec.
3. Receive audio concurrently with text transmission.
4. Timestamp every real text message and decoded PCM chunk using time.monotonic().
5. Normalize provider events into a common Veeksha result.
6. Preserve errors and rate limits instead of silently retrying them.

The adapters retain the provider-specific lifecycle:

• ElevenLabs begins with its required single-space initialization message, but that setup message is not considered the synthesis trigger.
• Deepgram Flux recognizes Connected, SpeechStarted, and SpeechMetadata.
• Deepgram Aura terminates the turn after Flushed.

API keys can come from the client configuration or:

• ELEVENLABS_API_KEY
• DEEPGRAM_API_KEY

Non-streaming HTTP clients

[veeksha/client/native_http_tts.py](https://github.com/project-vajra/veeksha/pull/203/files#diff)

Adds:

• elevenlabs_http_tts
• deepgram_flux_http_tts

These send the complete text in one HTTP request and receive the complete PCM response.

For these clients:

• TTFC = end-to-end response latency, because audio is unavailable until the full response arrives.
• The result contains one audio chunk.
• Fluidity is trivially perfect because all audio is already buffered; it should not be used to compare streaming quality.

2. Genuine duplex scheduling for Vajra

[veeksha/client/realtime_tts.py](https://github.com/project-vajra/veeksha/pull/203/files#diff)

Previously, the OpenAI Realtime-compatible client always did:

send all text → response.create → receive audio

It now supports two modes.

complete_text

Maintains the previous behavior:

text 1 → text 2 → ... → final text → response.create

duplex

Triggers synthesis after a configurable number of input tokens:

text 1 → response.create → text 2 → text 3 → ...
              │
              └──────────── audio may begin

Configuration:

input_output_mode: duplex
duplex_start_after_tokens: 1

The client records exactly when response.create was sent. It continues sending later text while the response is active.

Important: message ordering proves transport overlap, not semantic correctness. A server could speak only the prefix available at trigger time. Therefore, the documentation requires full-reference WER or a stronger coverage check before calling the result genuine duplex synthesis.

3. A common raw timing contract

[veeksha/core/audio_contract.py](https://github.com/project-vajra/veeksha/pull/203/files#diff)

The clients now expose the same provider-independent event timeline:

request start
WebSocket connected
session ready
first text sent
synthesis triggered
more text sent
first PCM bytes received
first playable PCM frame available
final text sent
audio completed
response completed

New raw fields include:

• provider
• provider_model
• provider_protocol
• response_trigger_offset_ms
• text_delta_timestamps
• audio_chunk_timestamps
• input_commit_offset_ms
• audio_done_offset_ms
• response_done_offset_ms

This lets the evaluator compute every provider’s metrics from client-side observations rather than trusting provider-reported latency.

4. First-playable-audio metrics

A provider packet is not automatically playable. Veeksha first converts the received bytes into audio duration.

For 24 kHz, mono, PCM16:

24,000 samples/sec × 2 bytes/sample = 48,000 bytes/sec
20 ms of audio = 960 bytes

Therefore, if the provider sends:

t=100 ms: 400 bytes
t=110 ms: 560 bytes

the first complete 20 ms frame becomes playable at 110 ms, when cumulative received PCM reaches 960 bytes.

New latency metrics:

Metric	Meaning
trigger_to_first_playable_audio_ms	Synthesis trigger to first complete playable PCM frame
first_input_to_first_playable_audio_ms	First real input text to first playable frame
request_start_to_first_playable_audio_ms	Connection initiation to first playable frame
ttfc	Request start to first wire audio chunk, which may not be playable yet

trigger_to_first_playable_audio_ms is the intended steady-state TTFA. It excludes WebSocket setup and input time before the semantic synthesis trigger.

5. Fluidity and playback simulation

[veeksha/evaluator/performance/audio_interactivity.py](https://github.com/project-vajra/veeksha/pull/203/files#diff)

The existing packet timeline is converted into fixed-duration playable PCM frames—20 ms by default. This removes dependence on how ElevenLabs, Deepgram, or Vajra happen to fragment WebSocket packets.

The Etalon-inspired algorithm:

1. Assigns the first playable frame a startup deadline.
2. Expects subsequent playable frames every 20 ms.
3. Lets early frames accumulate playback-buffer slack.
4. Lets late frames consume that slack.
5. Counts elapsed playback deadlines as misses if the frame is still late.
6. Resets slack after a miss.

fluidity = accepted playback deadlines / total playback deadlines

A score of 1.0 means the captured PCM supply could play continuously under the selected startup-buffer policy. It does not mean latency, quality, or naturalness was perfect.

The evaluator also computes:

• Stall count
• Total stall time
• Longest stall
• Minimum startup delay required to eliminate every underrun
• Stall-free fraction with 0/100/300 ms fixed delays
• Stall-free fraction after buffering 0/100/300 ms of audio
• Streaming RTF after the first audio chunk

User fluidity versus service fluidity

The implementation explicitly addresses input starvation.

• user_audio_fluidity_index: what the listener would experience, regardless of cause.
• tts_service_fluidity_index: emitted only when misses can reasonably be attributed to TTS.
• tts_service_fluidity_eligible: whether that attribution was valid.

In conservative mode, the service score is emitted only when all source text was available before playback began.

For controlled traces that guarantee sufficient synthesis-ready text, the benchmark can use:

fluidity_attribution_mode: source_oversupplied

This prevents the benchmark from blaming TTS for silence caused by an upstream LLM that did not produce text.

6. Duplex-overlap measurements

The evaluator now reports:

• audio_before_commit_ratio: fraction of audio received before the final text input.
• duplex_overlap_observed: whether a complete playable frame arrived before input completion.
• duplex_overlap_ms: how long playable audio delivery and text input overlapped.
• post_commit_audio_delivery_ms: audio delivery duration after the final input.

These answer two separate questions:

Did output actually overlap input?
How much useful overlap occurred?

7. Audio evaluator and output artifacts

[veeksha/evaluator/performance/audio.py](https://github.com/project-vajra/veeksha/pull/203/files#diff)

The audio evaluator now:

• Parses the normalized provider timeline.
• Computes first-playable, duplex, buffering, stall and fluidity metrics.
• Aggregates them into percentile/CDF summaries.
• Retains provider/model/protocol labels.
• Writes the new metrics into request_level_metrics.jsonl.
• Optionally stores the complete audit timeline in audio_raw_timing.jsonl.
• Generates a stall-free-fraction-versus-startup-budget plot.
• Produces separate fluidity distributions for configured startup delays.

This means one captured request can be replayed locally under several playback-buffer policies without calling the provider again.

8. Configuration and registry plumbing

The following files make the new clients and metrics configurable from YAML:

• veeksha/config/client.py
• veeksha/client/registry.py
• veeksha/types/__init__.py
• veeksha/config/evaluator.py

They add:

• Five new client types.
• Provider model, voice, PCM, pacing and API-key settings.
• ElevenLabs chunk schedules, normalization, speed and auto mode.
• Deepgram Flux/Aura model and speed options.
• Fluidity frame duration.
• Fluidity startup buffer.
• Conservative/source-oversupplied attribution.
• Validation for invalid combinations.

9. SLO support

Files:

• veeksha/config/slo.py
• veeksha/slo/metrics.py
• veeksha/slo/slo.py

The new metrics can be used as benchmark pass/fail conditions.

Example:

- metric: trigger_to_first_playable_audio_ms
  percentile: 0.90
  value: 1000

For latency and stalls, lower is better. For the following metrics, higher is better:

• Fluidity
• Audio-before-commit ratio
• Duplex-overlap success
• Service-fluidity eligibility

This also fixes percentile direction. For example, a fluidity SLO should normally evaluate the lower tail, such as P1 ≥ 0.99, rather than P99.

10. Correctness validation changes

ASR WER aggregation

[veeksha/evaluator/performance/asr.py](https://github.com/project-vajra/veeksha/pull/203/files#diff)

Clarifies and fixes empty-reference corpus WER:

• Empty reference + empty hypothesis → 0%
• Empty reference + inserted hypothesis words → 100%
• No samples → unavailable

The evaluator continues reporting:

• Sample-mean WER
• Corpus WER
• Duration-weighted WER

Corpus WER remains the primary ASR comparison.

TTS WER judge

Files:

• veeksha/config/verification.py
• veeksha/verification/audio.py

Whisper verification is now pinned with:

• Model
• Language
• Task=transcribe
• Beam size
• Device
• Compute type

This makes provider comparisons more reproducible and avoids automatic language detection changing the WER result.

With fail_on_threshold: true, verification now fails not only for bad WER but also for:

• Missing audio
• Transcription failure
• UTMOS failure
• Run-level verification errors

Previously, such failures could reduce the evaluated sample without necessarily failing the run.

11. Import/runtime cleanup

• librosa is loaded only when STT audio decoding is actually required.
• transformers is loaded only when a Hugging Face tokenizer is requested.
• Provider-native TTS benchmarks therefore avoid importing heavyweight model stacks merely by importing Veeksha.
• Pyright is explicitly configured for Python 3.14.

12. Documentation

docs/user_guide/tts_benchmarking.rst documents:

• Supported provider lanes
• All latency and fluidity definitions
• PCM frame normalization
• Buffering policies
• User versus service attribution
• Seed-TTS and ShareGPT trace sources
• Example YAML
• ASR versus TTS WER units
• UTMOS limitations
• The requirement to validate semantic duplex behavior

A small documentation cleanup also changes “Claude Code trace” to the generic “coding-assistant trace”; Veeksha does not bundle a Claude Code TTS trace.