Skip to content

Perf(qwen3-tts): fuse CodePredictor RoPE and reuse decode masks - #204

Merged
NSagan271 merged 1 commit into
mstar-project:mainfrom
qaqjx:main
Aug 10, 2026
Merged

Perf(qwen3-tts): fuse CodePredictor RoPE and reuse decode masks#204
NSagan271 merged 1 commit into
mstar-project:mainfrom
qaqjx:main

Conversation

@qaqjx

@qaqjx qaqjx commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Optimizes the Qwen3-TTS autoregressive generation path.

  • Replaces the repeated PyTorch RoPE operations in CodePredictor with the fused Triton apply_rope_pos_ids kernel.
  • Reuses the static invalid-token mask during decode and passes only the per-request suppress_eos flag into the CUDA Graph.
  • Adds correctness coverage and a deterministic fixed-frame benchmark.

On an NVIDIA H20, CodePredictor CUDA Graph latency decreased by 35.8-36.9%, end-to-end latency decreased by 4.1-4.4%, and worker GPU memory decreased by about 1.5%.

How was it tested?

  • .venv/bin/ruff check .
  • .venv/bin/pytest -q test/modular/test_qwen3_tts_model.py (35 passed)
  • CUDA_VISIBLE_DEVICES=0 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 .venv/bin/pytest -q test/integration/test_qwen3_tts_real_weights.py (5 passed)
  • End-to-end fixed-frame benchmarks for 32, 64, and 128 frames, plus concurrency 2, 4, and 8 and natural-EOS validation.

Performance results

Measured with Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice on one NVIDIA H20:

Metric Baseline Optimized Change
CodePredictor CUDA Graph, BS=1 10.077 ms 6.359 ms -36.9%
End-to-end latency, 32 frames 333.1 ms 318.6 ms -4.3%
End-to-end latency, 64 frames 587.4 ms 561.7 ms -4.4%
End-to-end latency, 128 frames 1,146.4 ms 1,099.7 ms -4.1%

The CodePredictor improvement was 35.8-36.9% across BS=1/2/4/8. At concurrency 2, throughput improved by 2.9%; concurrency 4 and 8 were saturated and showed no statistically clear gain or regression. Three natural-EOS requests produced non-empty audio and stopped before talker_max_tokens.

Checklist

  • ruff check . passes
  • Added or updated tests / docs where relevant

@qaqjx qaqjx changed the title perf(qwen3-tts): fuse CodePredictor RoPE and reuse decode masks Perf(qwen3-tts): fuse CodePredictor RoPE and reuse decode masks Aug 3, 2026

@NSagan271 NSagan271 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, just one comment. Also @qaqjx #201 (the multi-sampler PR) got merged, so can you rebase off of main and re-request my review? That PR already migrated Qwen TTS to the new sampler configuration.

Comment thread mstar/model/qwen3_tts/components/talker.py Outdated

@NSagan271 NSagan271 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@NSagan271
NSagan271 merged commit 11eb366 into mstar-project:main Aug 10, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants