fix(huddle): Pocket TTS quality — grouped chunking + FP32 model bundle#2224
Merged
Conversation
Co-authored-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@sprout-oss.stage.blox.sqprod.co> Signed-off-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@sprout-oss.stage.blox.sqprod.co>
…d speed knob Pocket TTS quality fix, part 1 of 2 (see the fp32 model swap commit for part 2; that commit is a single revertable unit — this one is int8-compatible on its own). Two changes: 1. Chunk grouping: upstream pocket-tts synthesizes chunks of up to MAX_TOKEN_PER_CHUNK=50 tokenizer tokens (typically multi-sentence); every generate() call is an independent generation with a cold FlowLM start, and each boundary is an exposed prosody seam (kyutai-labs/ pocket-tts#151 — the Kyutai team names chunk stitching as the reliability lever). Our sentence-per-call path created ~2-4x more seams than upstream. Sentences after the first now pack greedily up to ~200 chars (~60-100 tokens at the vocab's effective ~2-4 chars/token — deliberately at or above upstream's 50, since erring large means fewer seams and stays far below the 500-LM-step ceiling); the first sentence stays alone to preserve time-to-first-audio. 2. SYNTH_SPEED=1.05 deleted: sherpa-onnx's Pocket impl never reads GenerationConfig.speed (offline-tts-pocket-impl.h — zero references; the field is vits-only), and upstream has no speed knob. The constant documented an effect that does not exist. frames_after_eos remains untouched and the short-prompt max_frames cap stays, per the existing regression guards. Co-authored-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
Pocket TTS quality fix, part 2 of 2. Swap the pinned model manifest from csukuangfj2/sherpa-onnx-pocket-tts-int8-2026-01-26 to the fp32 sibling repo (same layout minus .int8 suffixes, commit 96d1e53). A direct same-runtime A/B (k2-fsa/sherpa-onnx#3172) found the int8 ONNX export audibly degraded output and that fp32 'significantly improved quality even at 1 step'. This commit is the complete fp32 change — models.rs manifest, pocket.rs FILE_* names, and doc text move together — so `git revert` of this one commit cleanly restores the int8 bundle if the blind A/B returns a null result. Cost: runtime bundle grows ~189 MB -> ~473 MB (lm_main.onnx is 303 MB, so MAX_TTS_FILE_BYTES rises to 400 MB). encoder, text_conditioner, both JSON tables, and LICENSE are byte-identical between the repos; only the three quantized sessions change. TTS_MODEL_VERSION bumps 2 -> 3 so existing int8 installs re-download. SHA-256 pins verified against local downloads of every file from the pinned HF commit. Co-authored-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
Co-authored-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
tlongwell-block
force-pushed
the
eva/pocket-tts-quality
branch
from
July 21, 2026 14:59
c6284a7 to
2f699d4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Pocket TTS in huddles "sounds janky." Audit (see
RESEARCH_POCKET_TTS_AB_PROTOCOL.mdon this branch) traced it to two root causes plus one piece of dead code:BATCH_SIZE = 1), so every sentence boundary is a prosody reset + audible seam. Upstream sherpa-onnx groups text into ~50-token chunks before synthesis for exactly this reason.speedparameter threaded through our synth path that the sherpa Pocket implementation never reads — a dead knob that only implied tunability we didn't have.What
Three changes (one commit each) plus a blind evaluation harness:
fix(huddle): group TTS sentences into upstream-sized chunks; drop dead speed knobtts.rs:BATCH_SIZE = 1→MAX_CHUNK_CHARS = 200(≈60–100 tokens at the vocab's effective ~2–4 chars/token, deliberately at-or-above upstream's 50 — Sami measured the ratio; we don't ship the SentencePiece tokenizer in Rust) with a newgroup_sentences_into_chunks(): the first sentence always goes alone to preserve time-to-first-audio, then greedy packing; oversized sentences stay unsplit. DeletedSYNTH_SPEED;pocket.rs::synth_chunkloses the never-readspeedparam, with a doc comment explaining why. Unit tests added for the grouping helper.fix(huddle): switch Pocket TTS to fp32 model bundlemodels.rs: newPOCKET_HF_BASEpinned tocsukuangfj2/sherpa-onnx-pocket-tts-2026-01-26@96d1e53ce3311ca6c2c6a35e2062d36b4cec6fa3; allTTS_FILE_HASHESswapped to FP32 SHA-256s (Mari verified by download+hash; I independently re-downloaded and re-verified — all match);TTS_MODEL_VERSION"2"→"3" to force re-download;MAX_TTS_FILE_BYTES200→400 MB (lm_main.onnxis ~303 MB); attribution/license text updated.test(tts): add blind Pocket quality corpus(Max)examples/pocket_quality_ab.rs+RESEARCH_POCKET_TTS_AB_PROTOCOL.md: deterministic blind INT8/FP32 × per-sentence/grouped corpus using production preprocessing, warm + genuinely fresh-engine cold cases, real-dormancy probes (--idle-minutes N --only ITEM), opaque filenames,key.json, generatedSCORING.md. Sami pre-registered the listening protocol and ship rules before any clips existed.Kept as-is deliberately:
steps=1(upstream design point), temperature at sherpa default 0.7 (correct for theenglish_2026-01weights),frames_after_eosuntouched (regression-guard tests still pass).Tradeoff
FP32 grows the model download 189 MB → 473 MB (one-time, per
TTS_MODEL_VERSIONbump). We're shipping it as the default rather than a variant because the upstream A/B attributes the worst artifacts directly to quantization. Merge gate: the pre-registered blind listening run must confirm — per Sami's decision rules, FP32 ships iff it wins on garble/post-idle items; a null result means reverting the FP32 commit (chunking stands on its own). Per Sami's review, the FP32 change is a single revertable unit —models.rsmanifest +pocket.rsFILE_*names + doc text all live in the one FP32 commit, verified by revert simulation (git revertof it compiles clean and restores a consistent int8 path).Testing
cargo check+ clippy clean; full desktop tauri suite green: 1509 passed / 0 failed / 13 ignored, plus 3/3 mixer tests.Credits
Review record
git revertfallback — fixed by re-splitting the branch (force-pushed; commit contents otherwise identical, plus his comment-wording fix onMAX_CHUNK_CHARS).