feat(voice): add duplex TTS benchmarking and validated metrics - #1
Closed
asmitks wants to merge 46 commits into
Closed
feat(voice): add duplex TTS benchmarking and validated metrics#1asmitks wants to merge 46 commits into
asmitks wants to merge 46 commits into
Conversation
_stream_vajra now sends audio and receives deltas concurrently instead of
sending the whole (paced) clip before reading any response. With the old
sequential order the first delta could not be observed until the upload
finished, so ttft tracked clip length (~16s) rather than server latency.
Adds two per-request metrics that flow through the evaluator into
summary_stats.json / request_level_metrics.jsonl:
- first_partial: first audio frame -> first partial transcript (TTFB)
- final_latency: end-of-audio sentinel -> final transcript (server tail,
independent of clip length)
audio.py: new AudioRequestMetrics fields, CDFSketch sketches, extraction in
record_request_completed, and JSONL row output.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Conflict resolutions (8 files): - types: AUDIO=5, SEED_TTS_TEXT=6 (both trace flavors coexist) - session.py/trace: took theirs (char-mode ShareGPT superset + SeedTTS flavor), re-added STT AudioTraceFlavorConfig + AUDIO registry entry - audio evaluator metric naming (per review decision): keep BOTH ttft (STT first-token) and ttfa (TTS first-audio) as distinct fields; unify end-to-end as `e2e`. Re-added ttfa (dataclass/summaries/extraction/put/ JSONL) that auto-merge had dropped. - openai_chat/tts clients now emit ttfa+e2e for audio (was mislabeled ttft) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… as ttfc Keep end_to_end_latency as the request-level key (e2e stays the SLO alias), and collapse ttfa (TTS) / ttft (STT) into a single ttfc across clients, evaluators, SLO config, and analysis scripts. Fixes 3 stale unit tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Refactor STTClient into a base + per-provider subclasses (vajra, vllm_realtime) selected by provider via a __new__ factory, then trim to the two paced-WebSocket paths actually used: drop the vllm batch/SSE provider, vajra batch, httpx plumbing, tpot, and the mime/language/streaming config knobs. Fix the capacity gate: P95 RTF < 1.0 is unsatisfiable under 1x pacing (e2e >= clip length, so RTF >= 1 by construction), so gate on P99 ttfc instead and drop the dead TPOT SLO. Equate the two configs to engine-vs-engine (both serve Voxtral) leaving only provider/api_base/output_dir different, and strip verbose comments. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- stt.py: drop the __new__/__init_subclass__ provider registry for a plain
factory + dict; share one concurrent send/recv stream loop across both
providers (consistent live ttfc under 1x pacing); remove vajra-only
first_partial/final_latency timings.
- audio.py: drop dead tpot field; compute WER via the vendored Open ASR
Leaderboard EnglishTextNormalizer + jiwer instead of hand-rolled tables.
- asr_normalizer/: vendor normalizer.py + english_abbreviations.py from
open_asr_leaderboard (attributed) so WER matches the leaderboard.
- Merge prepare_audio_buckets.py into a single prepare_audio_traces.py.
- Remove the {datetime} YAML template-token feature; update configs/sweep.
- pyproject: add regex dependency.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-vajra/veeksha into users/mayank/stt_client # Conflicts: # pyproject.toml # scripts/utils.py # veeksha/client/tts.py # veeksha/config/slo.py # veeksha/evaluator/performance/audio.py
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…a#199) Added Artificial Analysis style datasets and metrics to exactly match the metrics they publish for ASR models
# Conflicts: # .gitignore # pyproject.toml # veeksha/config/core/flat_dataclass.py # veeksha/config/evaluator.py # veeksha/config/generator/session.py # veeksha/config/server.py # veeksha/config/utils.py # veeksha/generator/session/trace/__init__.py # veeksha/generator/session/trace/registry.py # veeksha/orchestration/__init__.py # veeksha/orchestration/registry.py # veeksha/types/__init__.py
…ect-vajra#200) * [stt] Add ASR interactivity traces Major changes: 1. Add word-timestamp interactivity scoring and STT transcript snapshots. 2. Refactor ASR trace preparation into source adapters plus NeMo alignment helper. 3. Keep ASR WER aggregation request-scoped and remove Earnings22 chunk regrouping. 4. Add max-duration trace selection with explicit underfilled-count failure. 5. Add mamba environment setup docs and focused unit coverage. * [stt] Consolidate ASR trace timestamping Fold the ASR trace dataset adapters and NeMo Docker alignment path into prepare_audio_traces.py so one command generates timestamped VoxPopuli, Earnings22, and AMI traces. Key changes: - Make word timestamping the default trace-generation path with a --without-word-timestamping escape hatch - Chunk timestamped long clips on word boundaries with a 30 second default max duration - Remove the separate align_audio_trace and asr_trace_sources modules - Update unit imports and ASR benchmarking notes for the consolidated flow * [stt] Automate AMI trace preparation Download AMI manual word annotations and Mix-Headset audio from the official AMI host when building ami_word_timed traces. This removes the local path setup requirement while keeping native AMI word timestamps separate from the NeMo Docker alignment path. Key changes: - Cache AMI annotations and meeting audio under benchmark_output/ami_cache - Derive trace output roots from the selected dataset family - Remove the public output-dir override from prepare_audio_traces * [stt] Harden ASR interactivity scoring Allow requests with missing or empty transcript snapshots to skip interactivity scoring instead of failing the completion worker. Expand normalized reference entries so timestamped words that split or disappear under normalization do not crash aligned-manifest runs. Key changes: - Return no interactivity metric when snapshots are absent or empty - Expand normalized reference words before matching snapshots - Add regression coverage for normalized references and empty snapshots * [stt] Add metadata for interactivity frontend * Added non negative clamp for interactivity * [stt] Add ASR replay viewer Add a standalone replay viewer for ASR request metrics so benchmark runs can visualize ground-truth word timing, streamed transcript arrival, and replay-derived latency over time. Key changes: - Export ASR request metrics into an asr_replay.json bundle - Add a self-contained browser viewer under tools/asr_replay_viewer - Load ECharts dynamically from a pinned CDN URL - Document how to export and open the replay viewer
…y tests (project-vajra#201) * Added a --target-duration flag to prepare_audio_traces.py to loop the dataset audio clips and transcripts to make them close to the passed duration * Made asr replay viewer more efficient for 10min workloads * Changed _maybe_pace->_maybe_pace_until in stt.py so that we sleep until a specified timestamp instead of a fixed time, this prevents time drifts in sending * Added stddev to prepare_audio_traces target-duration
…ming asr (project-vajra#202) Fix: Removed _audio_to_pcm16_bytes from ttfc calculation
CTM and AMI second->millisecond conversions leaked float artifacts (0.1 + 0.2 -> 300.00000000000006 ms) into trace manifests. Round to microsecond precision at parse time, matching the rounding already applied when re-basing word timings onto clips. Also folds in black formatting for two nonconforming statements. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VRhArSi52xyjzdiHvP4P7d
AudioTraceFlavorConfig.target_duration_s limits every trace row to a fixed streamed-audio prefix: reference_word_timestamps and expected_transcript are trimmed to the words that end within the prefix, and the generator records input_audio_start_ms / input_audio_end_ms slice bounds for the STT client to cut the decoded PCM before streaming. Rows without reference_word_timestamps, or whose trimmed transcript would be empty, fail fast at generation time; ground truth remains mandatory for WER scoring. The field uses Optional[float] (not `float | None`) so the flat-dataclass argparse bridge recognizes it as optional on Python 3.12, where `types.UnionType` is not `typing.Union`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VRhArSi52xyjzdiHvP4P7d
Replace the bespoke /stream provider ("vajra") with
"vajra_openai_realtime", which drives Vajra's OpenAI-compatible
/openai/v1/realtime?intent=transcription WebSocket:
transcription_session.created/update handshake, base64 PCM16
input_audio_buffer.append frames, manual commit as EOF, and
conversation.item.input_audio_transcription.delta/completed events.
The terminal ".failed" item event is mapped to an error so a failed
transcription fails the request immediately instead of waiting out the
request timeout.
Client-side additions shared by both providers:
- time_to_first_visible_text alongside ttfc. ttfc anchors to the first
delta whose own payload carries text after control-token cleaning;
time_to_first_visible_text anchors to the first moment the assembled
cleaned transcript is non-empty. Semantics are documented on
STTStreamResult.
- PCM slicing from input_audio_start_ms / input_audio_end_ms request
metadata (written by the audio trace generator's target_duration_s),
applied before the latency clock starts.
- Configurable WebSocket keepalive (ws_ping_interval_s /
ws_ping_timeout_s) so long paced streams are not dropped by the
default ping timeout.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VRhArSi52xyjzdiHvP4P7d
Plumb the new client-side metric through ASRRequestMetrics and the audio performance evaluator's latency CDF summaries. Refresh docs/ASR_BENCHMARKING.md: define ttfc vs time_to_first_visible_text precisely (per-delta content vs assembled visible transcript, both anchored at the first audio byte sent), add the aggregate summary entry, and document target_duration_s row trimming in the manifest section. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VRhArSi52xyjzdiHvP4P7d
…arget median Add target_duration_spread_s (hard clip half-width) and optional target_duration_sigma_s (default spread/2) to the audio trace flavor: each session streams Normal(target, sigma) re-drawn until inside [target - spread, target + spread], drawn deterministically from the run seed via a dedicated "audio_target_duration" stream. Symmetry keeps the median at target_duration_s (verified: 20k draws, median 300.2s for target 300/spread 120). Clips must be at least target + spread long (existing request-time failure covers shortfalls). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
None (new default) provisions one client thread per 8 target-concurrent sessions (floor 3) instead of a fixed 3: an under-provisioned pool serializes per-session sends and shows up as phantom server-side latency at high concurrency (ASR benchmarking doc's c160+ warning; measured at c128 as multi-second per-session interactivity offsets). The sweep planner already provisioned per-load; direct configs now get the same protection. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rmatting The spread help still described a uniform distribution from an earlier design; it is a clipped Gaussian. Sigma larger than the clip half-width now crashes at config time (rejection acceptance collapses past the bounds). black/isort applied. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ed normalizer, reused alignment ASR scoring (WER + per-word interactivity) ran fully serialized behind two nested evaluator locks on the completion path: 253.5s for a c128 wave's end-of-run burst. Scoring now runs outside the locks (channel delegation un-nested, accumulator self-locked) across num_completion_threads (default 2 -> 8). The normalizer's per-char unicodedata walk becomes memoized str.translate tables (~25x), the ~40 contraction regexes get a literal-containment prescan, and the number FSM a non-numeric fast path; interactivity alignment interns tokens, reuses one SequenceMatcher and matching blocks across identical snapshots. Worst-case post-run: ~258s -> ~26s. Outputs proven byte-identical on a captured 144-session production run (summary, SLOs, 12 CSVs, 202MB JSONL); per-stage timing logs added. New pinned regression fixtures for normalizer and interactivity. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…che clip assets off-loop, GIL guard Websocket permessage-deflate was silently negotiated by default and zlib-compressed every high-entropy base64 audio append (84% of send cost, 1.23 of 1.84 cores at c156). New ws_permessage_deflate config defaults off; ws_chunk_size help documents the CPU model (8192 recommended at high concurrency). Per-clip _ClipAssets cache (FIFO cap 64) holds decoded PCM + pre-encoded wire messages, built once per clip in an executor — removes 250 synchronous librosa decodes from the event loop (73ms avg pacing stalls) and per-session 19MB PCM copies via memoryview slicing. _warn_if_gil_enabled() at run start/end: without -Xgil=0 the first librosa.load lazily imports msgpack._cmsgpack and silently re-enables the GIL mid-run. Measured (mock-server isolated, identical workload): 1.843 -> 0.701 cores @2048 chunk, 0.427 @8192; RSS 4.83 -> 1.49 GB; ws_send 504 -> 80us/msg. Client tests 15/15, unit suite 136 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add provider-native streaming and HTTP adapters for ElevenLabs and Deepgram. Derive first-playable, overlap, stall, and Etalon-inspired fluidity metrics from client-side PCM timelines with conservative source attribution. Document reproducible Seed-TTS runs and cover protocols, SLOs, and config registration.
Owner
Author
|
Superseded by project-vajra#203, which targets users/ksukrit/tts_merge so the review contains only the two benchmarking commits. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Branch ancestry
This branch starts from
project-vajra/veeksha:users/ksukrit/tts_mergeat44ada942, then adds these two commits:183e7c56—feat(tts): add duplex fluidity benchmarking94a015cc—feat(voice): validate streaming latency and quality metricsBecause this draft targets
asmitks/veeksha:main, GitHub's full diff also includes the prerequisite voice/TTS work