Add veeksha voice support - #206
Open
AgrawalAmey wants to merge 76 commits into
Open
Conversation
_stream_vajra now sends audio and receives deltas concurrently instead of
sending the whole (paced) clip before reading any response. With the old
sequential order the first delta could not be observed until the upload
finished, so ttft tracked clip length (~16s) rather than server latency.
Adds two per-request metrics that flow through the evaluator into
summary_stats.json / request_level_metrics.jsonl:
- first_partial: first audio frame -> first partial transcript (TTFB)
- final_latency: end-of-audio sentinel -> final transcript (server tail,
independent of clip length)
audio.py: new AudioRequestMetrics fields, CDFSketch sketches, extraction in
record_request_completed, and JSONL row output.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Conflict resolutions (8 files): - types: AUDIO=5, SEED_TTS_TEXT=6 (both trace flavors coexist) - session.py/trace: took theirs (char-mode ShareGPT superset + SeedTTS flavor), re-added STT AudioTraceFlavorConfig + AUDIO registry entry - audio evaluator metric naming (per review decision): keep BOTH ttft (STT first-token) and ttfa (TTS first-audio) as distinct fields; unify end-to-end as `e2e`. Re-added ttfa (dataclass/summaries/extraction/put/ JSONL) that auto-merge had dropped. - openai_chat/tts clients now emit ttfa+e2e for audio (was mislabeled ttft) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… as ttfc Keep end_to_end_latency as the request-level key (e2e stays the SLO alias), and collapse ttfa (TTS) / ttft (STT) into a single ttfc across clients, evaluators, SLO config, and analysis scripts. Fixes 3 stale unit tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Refactor STTClient into a base + per-provider subclasses (vajra, vllm_realtime) selected by provider via a __new__ factory, then trim to the two paced-WebSocket paths actually used: drop the vllm batch/SSE provider, vajra batch, httpx plumbing, tpot, and the mime/language/streaming config knobs. Fix the capacity gate: P95 RTF < 1.0 is unsatisfiable under 1x pacing (e2e >= clip length, so RTF >= 1 by construction), so gate on P99 ttfc instead and drop the dead TPOT SLO. Equate the two configs to engine-vs-engine (both serve Voxtral) leaving only provider/api_base/output_dir different, and strip verbose comments. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- stt.py: drop the __new__/__init_subclass__ provider registry for a plain
factory + dict; share one concurrent send/recv stream loop across both
providers (consistent live ttfc under 1x pacing); remove vajra-only
first_partial/final_latency timings.
- audio.py: drop dead tpot field; compute WER via the vendored Open ASR
Leaderboard EnglishTextNormalizer + jiwer instead of hand-rolled tables.
- asr_normalizer/: vendor normalizer.py + english_abbreviations.py from
open_asr_leaderboard (attributed) so WER matches the leaderboard.
- Merge prepare_audio_buckets.py into a single prepare_audio_traces.py.
- Remove the {datetime} YAML template-token feature; update configs/sweep.
- pyproject: add regex dependency.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-vajra/veeksha into users/mayank/stt_client # Conflicts: # pyproject.toml # scripts/utils.py # veeksha/client/tts.py # veeksha/config/slo.py # veeksha/evaluator/performance/audio.py
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Added Artificial Analysis style datasets and metrics to exactly match the metrics they publish for ASR models
# Conflicts: # .gitignore # pyproject.toml # veeksha/config/core/flat_dataclass.py # veeksha/config/evaluator.py # veeksha/config/generator/session.py # veeksha/config/server.py # veeksha/config/utils.py # veeksha/generator/session/trace/__init__.py # veeksha/generator/session/trace/registry.py # veeksha/orchestration/__init__.py # veeksha/orchestration/registry.py # veeksha/types/__init__.py
* [stt] Add ASR interactivity traces Major changes: 1. Add word-timestamp interactivity scoring and STT transcript snapshots. 2. Refactor ASR trace preparation into source adapters plus NeMo alignment helper. 3. Keep ASR WER aggregation request-scoped and remove Earnings22 chunk regrouping. 4. Add max-duration trace selection with explicit underfilled-count failure. 5. Add mamba environment setup docs and focused unit coverage. * [stt] Consolidate ASR trace timestamping Fold the ASR trace dataset adapters and NeMo Docker alignment path into prepare_audio_traces.py so one command generates timestamped VoxPopuli, Earnings22, and AMI traces. Key changes: - Make word timestamping the default trace-generation path with a --without-word-timestamping escape hatch - Chunk timestamped long clips on word boundaries with a 30 second default max duration - Remove the separate align_audio_trace and asr_trace_sources modules - Update unit imports and ASR benchmarking notes for the consolidated flow * [stt] Automate AMI trace preparation Download AMI manual word annotations and Mix-Headset audio from the official AMI host when building ami_word_timed traces. This removes the local path setup requirement while keeping native AMI word timestamps separate from the NeMo Docker alignment path. Key changes: - Cache AMI annotations and meeting audio under benchmark_output/ami_cache - Derive trace output roots from the selected dataset family - Remove the public output-dir override from prepare_audio_traces * [stt] Harden ASR interactivity scoring Allow requests with missing or empty transcript snapshots to skip interactivity scoring instead of failing the completion worker. Expand normalized reference entries so timestamped words that split or disappear under normalization do not crash aligned-manifest runs. Key changes: - Return no interactivity metric when snapshots are absent or empty - Expand normalized reference words before matching snapshots - Add regression coverage for normalized references and empty snapshots * [stt] Add metadata for interactivity frontend * Added non negative clamp for interactivity * [stt] Add ASR replay viewer Add a standalone replay viewer for ASR request metrics so benchmark runs can visualize ground-truth word timing, streamed transcript arrival, and replay-derived latency over time. Key changes: - Export ASR request metrics into an asr_replay.json bundle - Add a self-contained browser viewer under tools/asr_replay_viewer - Load ECharts dynamically from a pinned CDN URL - Document how to export and open the replay viewer
…y tests (#201) * Added a --target-duration flag to prepare_audio_traces.py to loop the dataset audio clips and transcripts to make them close to the passed duration * Made asr replay viewer more efficient for 10min workloads * Changed _maybe_pace->_maybe_pace_until in stt.py so that we sleep until a specified timestamp instead of a fixed time, this prevents time drifts in sending * Added stddev to prepare_audio_traces target-duration
…ming asr (#202) Fix: Removed _audio_to_pcm16_bytes from ttfc calculation
…snapshots
Disconnected benchmark clients leave sessions decoding server-side
(the ws handler finalizes them, but the Talker keeps decoding to
eos/length_cap) -- ~122 zombies measured after a c=320 run. They
compete for decode capacity and silently skew subsequent runs.
New TTSZombieSessionProbe GETs {api_base}/debug/tts_worker_stats at
benchmark start and end (built only for Vajra endpoints that declare a
health_url) and the health check compares the server-side finished
delta (talker.finished.eos + talker.finished.length_cap) against the
requests this benchmark completed. A surplus fails the check with the
per-cause delta; a deficit is noted (sessions still decoding at the
end snapshot). A 404 / unreachable endpoint / missing Talker snapshot
skips the check with a note instead of failing the run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add provider-native streaming and HTTP adapters for ElevenLabs and Deepgram. Derive first-playable, overlap, stall, and Etalon-inspired fluidity metrics from client-side PCM timelines with conservative source attribution. Document reproducible Seed-TTS runs and cover protocols, SLOs, and config registration.
…e-tts-longform) New `veeksha score-tts-longform` command implementing the quality-ladder Tier-2 drift metrics for a single long-form waveform (PCM int16 or WAV): - WER(t): non-overlapping 28 s faster-whisper chunks (greedy, language pinned, VAD off -- seed-protocol-adjacent; backend CTranslate2 vs HF is the documented deviation), exact Seed-TTS-Eval normalization (punctuation minus apostrophe, single-pass double-space collapse, lowercase), then ONE global jiwer alignment of the concatenated transcript vs the reference. Per-chunk approximate WER by attributing global alignment ops to chunks via hypothesis-word spans; deletions split duration-proportionally with empty (silent) chunks at the same position so collapse chunks keep their omission mass. Per-chunk counts sum exactly to the global S/D/I. - UTMOS(t): balacoon TorchScript on 10 s chunks at 16 kHz, per-minute mean + min (min catches local collapse). - Repetition/omission: per-chunk ins/del split, duplicated-5-gram counts, zlib compression-ratio > 2.4 loop flag (Whisper convention). - Energy: RMS dBFS + silence fraction per 30 s bin, model-free numpy. - SIM(t): non-overlapping 3 s WavLM-SV cosine windows vs prompt (or audio-head) anchor; degrades to a skip-note when the UniSpeech wavlm_large_finetune.pth TorchScript export is not configured/present (download source documented in --help). Outputs: summary.json (global + per-chunk + per-bucket arrays), curves.csv (bucket, wer, utmos_mean, utmos_min, ins, del, dup5grams, rms, silence_frac), report.txt (global block, first-2 vs last-2 bucket drift deltas, per-bucket table). Every model track degrades to a note when its model is unavailable; the energy track always runs. Also: normalize_text now includes the seed protocol's double-space collapse, and the balacoon UTMOS helpers in verification/audio.py are parameterized by (hf_repo, jit_file, device) for reuse. 45 unit tests: seed-normalizer exactness, chunk boundaries, alignment attribution (planted ins/del/sub, silent chunks), bucket weighting conservation, loop detectors, energy on synthetic sine+silence PCM, fake-injected full pipeline, no-model graceful degradation, CLI registration and --help. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XaoSFTBvHyAP2JVrdeWpTo
Merge users/anirudha/tts_v2 with PR #203's duplex-fluidity work. - retain Vajra native streaming, cap/zombie health checks, and long-form scoring - add ElevenLabs/Deepgram clients plus duplex, playable-frame, and fluidity metrics - reconcile provider/timing contracts, client enums, verification, and ASR correctness - document feature decisions and validation in LOG.md Validation: - 135 changed-feature tests pass - 428 full-suite tests pass; 2 unchanged port-8000 tests deselected - Black/isort clean; pyright has no new diagnostics versus the common baseline
Unify provider-specific TTS and STT implementations behind transport-level public clients. Keep provider wire details internal so configuration, evaluation, and metrics remain provider agnostic. Key changes: - replace provider-specific TTS types with tts and streaming_tts - move TTS and STT wire behavior into internal protocol strategies - share authentication, WebSocket URL, and error handling utilities - update registry, evaluator, tests, and documentation
Add deterministic client-side aborts to Vajra streaming TTS so canary runs can exercise server cleanup and slot-reclamation paths. Treat intentional hang-ups as successful adversarial outcomes and keep their partial output from skewing service-quality aggregates. Key changes: - configure aborts by input progress, received audio, or wall time - mark aborted audio results and bucket them separately in evaluator exports - exclude aborted requests from quality and truncation aggregates - add protocol, configuration, evaluator, and integration coverage
Add a reproducible WebSocket benchmark matrix for hosted realtime TTS
providers across SeedTTS and ShareGPT traces.
Key changes:
- Add 100-request configs for five models at 50 words per second
- Report first-input TTFB, trigger-to-playable latency, TTFC, RTF,
and user-side fluidity
- Save generated WAV audio under third_party_numbers/{model}/{trace}
- Use X-API-Key, per-request UUIDs, Skylar, and 3000 ms buffering
for Cartesia
- Add protocol, configuration, and benchmark-matrix test coverage
ConcurrentTrafficScheduler latched its rampup reference at construction, but warmup and session pre-generation run between construction and the first dispatch. A setup phase longer than the rampup window consumed the whole ramp before any session was dispatched, so the full target concurrency was admitted at once and a configured 60-second rampup silently never fired. BaseTrafficScheduler already exposes reset_reference_time as the pre-dispatch hook and benchmark.py already calls it; RateTrafficScheduler already implemented it. Add the concurrent override so it re-anchors the monotonic reference and clears the latched ramp-complete flag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tions The rampup target is a function of time, but it was only ever sampled from schedule_session and notify_completion. A generator that schedules its whole backlog in one burst therefore samples the ramp only during that burst, and with sessions longer than the ramp window no completion occurs inside it, so the target is never re-read. Admission froze after the first few sessions and released the entire remainder at once when the pilot sessions finished — an arrival shape sharper than no ramp at all. Re-run admission on the dispatcher's wait path and bound that wait by the next ramp step while sessions are pending, so admission tracks the ramp whether or not any session has completed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Health checks previously ran only inline at the end of a benchmark and never affected exit status, so pipelines had no way to gate on a run's validity. Add a `veeksha health` subcommand that re-runs every health check against a finished benchmark output directory by reconstructing the BenchmarkConfig from the run's persisted config.yml. Key changes: - HealthCheckConfig (`veeksha health --run_dir <dir>`) with --output_file override and --strict (default) non-zero exit when any check fails, making it usable as a post-run pipeline stage - Carry over non-re-runnable sections (TTS zombie-session check) from the in-run report instead of silently dropping them; their recorded outcome still counts toward the exit code - load_benchmark_config_from_run_dir round-trips the persisted config.yml back into a BenchmarkConfig
Prepared ASR/TTS traces were rebuilt from scratch by every user, which made runs hard to reproduce: upstream HF datasets are unpinned, skipped rows shift clip selection, and NeMo forced alignment can jitter across GPUs. Publishing frozen trace bundles to a Hugging Face dataset repo (avartha/veeksha-voice-traces) fixes the references everyone measures against. Key changes: - scripts/audio_trace_hub.py: fetch/publish subcommands with a repo layout organized by direction (stt/, tts/) mapped onto the local traces/asr layout; pinned revisions, filtered manifest variants sharing one audio pool, validation before upload/after download - prepare_audio_traces.py now writes build_info.json provenance (git commit, argv, upstream dataset revision SHAs, NeMo model/image)
CI resolved dependencies fresh via `uv pip install -e .`, which ignores
uv.lock. Once numpy 2.5 shipped, that resolution took it (numpy is
unpinned), which excluded numba 0.66.0 (`numpy<2.5`). The resolver then
backtracked numba to 0.53.1 -- whose metadata declares no upper Python
bound -- and its llvmlite 0.36.0 hard-fails to build on 3.14:
RuntimeError: Cannot install on Python version 3.14.3;
only versions >=3.6,<3.10 are supported.
The failure only surfaces at build time, so a resolution dry-run looks
clean while CI dies.
uv.lock already pinned the working set (numpy 2.4.6, numba 0.66.0,
llvmlite 0.48.0 -- all with cp314t wheels), so pointing both install
paths at `uv sync --locked` fixes the build and makes CI reproducible
against future upstream releases. `--locked` rather than `--frozen` so a
pyproject.toml edit without a relock fails loudly instead of silently
installing a stale set.
librosa, added by this branch, is what first pulled numba into the
dependency tree and exposed the cap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This reverts commit 312b13d.
librosa, added by this branch, pulls in numba, and numba 0.66.0 requires
`numpy<2.5`. With numpy unconstrained, a fresh resolve takes numpy 2.5,
which rules out numba 0.66.0; the resolver then backtracks numba to
0.53.1, whose metadata declares no upper Python bound, and that pins
llvmlite 0.36.0:
RuntimeError: Cannot install on Python version 3.14.3;
only versions >=3.6,<3.10 are supported.
That guard only fires at build time, so resolution looks clean and the
install dies afterwards -- which is how CI failed while a dry-run passed.
Capping numpy holds numba at 0.66.0 / llvmlite 0.48.0, both of which
publish cp314t wheels and need no compilation at all.
This fixes the install for end users as well as CI: the documented entry
point is `uvx -p 3.14t veeksha`, which resolves fresh from PyPI and never
consults a lockfile, so pinning here is what keeps that working.
Lift the cap once numba supports numpy 2.5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ates TTFC was coerced to 0 ms whenever no first-content event was ever timed (`round(ttfc or 0.0, 3)`), so a request that produced no audio or an empty transcript reported an impossibly fast 0 ms and dragged every latency percentile down. TTFC is now None in that case, and the evaluator keeps None out of the sketch instead of inventing a sample. Excluding only TTFC left the aggregates inconsistent, though. StreamingTTSClient deliberately still attaches the AUDIO channel when a request emitted text deltas before failing, so a 502 "stream completed without audio" reached the evaluator and its end-to-end latency, chunk_count and RTF all landed in the sketches while its TTFC did not -- different denominators per metric. The evaluator now gates on `success`, the way it already gated on `aborted`, so a failed request keeps its row for forensics but feeds no aggregate. Rows carry `success` and `error_code`, and the summary reports `failed_requests_count`. An empty CDFSketch reported a fabricated 0 for mean and every percentile, which undid the fix one layer up: a run where everything failed would summarize as flawless 0 ms. It now reports no keys at all, matching what get_summary() already did for the ASR latency sketches. Also hardens the STT client: every provider handshake now decodes its first frame through _decode_stt_json_message rather than a bare json.loads, so a proxy error page or non-object JSON fails as a named protocol error instead of an opaque 520 carrying a JSONDecodeError. The _STTProviderProtocol.finish_messages return type becomes Sequence because list is invariant and every implementation returns list[str]. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This repository is public, so a few things that were fine internally are not fine shipped. Removes THIRD_PARTY_BENCH.md. It is an internal working note, not documentation: named-vendor result tables with explicit pass/fail SLO verdicts, an internal branch name and cluster path, and a "Before publishing" section listing four prerequisites that were never met (the ASR summary is recorded there as out of date, and the TTS latency numbers as unpaired with any intelligibility check). Its reproduction steps also reference scripts and result directories that are not in the repo. Replaces machine-local trace paths in the five ShareGPT streaming-TTS configs with a /path/to placeholder, matching the convention already used in managed_server.yml -- those configs could not have worked for anyone outside the original machine anyway. Renames output_dir from the internal experiment layout (third_party_numbers/<model>/<corpus>) to benchmark_output/, matching every other sample config, and updates the test that pinned the old value. Drops references to an internal product brief from the user guides and two configs, describing the metrics on their own terms instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds end-to-end voice benchmarking to Veeksha: streaming STT and TTS clients, audio latency/interactivity/WER metrics, trace preparation + HF hub tooling, managed Vajra server support, health checks, sample configs, and docs.
This is a large feature branch (multi-provider realtime audio) that unifies ASR and TTS under one measurement contract.
Architecture
client.type: stt→ WebSocket PCM-in / text-out (veeksha/client/stt.py)client.type: streaming_tts→ WebSocket paced text-in / audio-out (streaming_tts.py)client.type: tts→ HTTP complete-text TTS (tts.py)client/utils.py(text pacing, WS URL/auth/error mapping).core/audio_contract.py(AudioMetricKey).evaluator/performance/audio.py+asr.py).Metric contract (important)
nulland excluded from percentile sketches (never invent 0 ms)Additional TTS keys: duplex/fluidity/interactivity timestamps; STT: WER (Open ASR Leaderboard normalizer),
time_to_first_visible_text, partial/final latencies.How to run
Optional extras:
audio-verification(faster-whisper),utmos(MOS quality).Traces:
scripts/prepare_audio_traces.py+scripts/audio_trace_hub.py(HF fetch/publish).Other notable changes
reset_reference_time()after warmup so ramp is not consumed during setup.num_client_threads; GIL re-enable warning on free-threaded builds.start/stop, endpoint object); Vajra server support.veeksha health+ TTS zombie-session probe.Test plan
sanity_checkgreenFollow-ups (non-blocking)