qwen3-tts (HIP): talker diverges from CPU at frame 0, runs away to context limit

[adam-schneider-dev]

Bug: qwen3-tts on HIP samples a different token at frame 0 than CPU, then runs to the context limit

One line: Same input, same weights. GPU picks a different first token than CPU. GPU then never stops — it runs until the KV cache is physically full (3796 frames, 303.76s of audio) instead of stopping normally (under 10s). No error, no non-zero exit code — just a log line.

---

Environment

Component	Value
CPU	AMD Ryzen 7 7800X3D, 8 cores / 16 threads
RAM	32GB physical. free -h shows "30Gi" — that's GiB vs GB, not less RAM. See note below.
GPU (compute)	AMD Radeon RX 7900 XTX, gfx1100, ~24GB VRAM
OS	Fedora Linux 43 (Workstation)
Kernel	7.1.5-101.fc43.x86_64
ROCm stack	6.4.4 (see package table below)
HIP compiler	clang 19.0.0git (rocm-clang)
CrispASR repo	https://github.com/CrispStrobe/CrispASR
CrispASR commit	1a9ee81eb76bf52b3ba4a589306b986539f42010
CrispASR version	0.8.25
ggml submodule	https://github.com/CrispStrobe/ggml.git @ 890278a8342c620197c90e702e1188bcab94f510
Backends compiled in	cpu,hip only

RAM note: at capture time the machine was under memory pressure (16Gi/30Gi RAM used, 7.4Gi/8.0Gi swap used). Doesn't affect the sampled tokens below — may affect CPU-path timing numbers.

GPU enumeration note: ggml sees 2 ROCm devices at startup — Device 0 (7900 XTX, gfx1100) and Device 1 (the CPU's integrated GPU, gfx1036, ~512MB). Build only targets gfx1100. HIP_VISIBLE_DEVICES was confirmed unset for every test. Whether the iGPU took part in any run was not measured — disclosed as an open variable, not ruled out.

Driver-version gotcha: rocm-smi --showdriverversion prints "7.1.5-101.fc43.x86_64" — that's the Fedora kernel/amdgpu driver, not a ROCm version. Don't read it as "ROCm 7.1.5."

ROCm packages installed

rocm-6.4.4-1.fc43.noarch, rocm-core-6.4.4-1.fc43, rocm-runtime-6.4.2-3.fc43, rocm-hip-6.4.2-2.fc43, rocminfo-6.4.0-2.fc43, rocm-smi-6.4.3-1.fc43, rocm-opencl-6.4.2-2.fc43

Build config

cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_HIP=ON -DGPU_TARGETS=gfx1100 -DAMDGPU_TARGETS=gfx1100

Flag	Value
GGML_HIP	ON
GGML_HIP_GRAPHS	OFF
GGML_HIP_MMQ_MFMA	ON
GGML_HIP_NO_VMM	ON
GGML_HIP_RCCL	OFF
GGML_HIP_ROCWMMA_FATTN	OFF

The two bolded flags matter later — they rule out two suspects (HIP graphs, flash-attention's rocWMMA kernel) before you even ask.

---

Models used

All from https://huggingface.co/cstr (talker + code_pred + speaker unified GGUF — not the split talker/predictor/ONNX format some other converters ship).

Label	Filename	URL	sha256
1.7B Base Q8_0	qwen3-tts-12hz-1.7b-base-q8_0.gguf	cstr/qwen3-tts-1.7b-base-GGUF	bbb93ab1...520d
1.7B Base F16	qwen3-tts-12hz-1.7b-base-f16.gguf	cstr/qwen3-tts-1.7b-base-GGUF	c7039d04...e00e
0.6B Base F16	qwen3-tts-12hz-0.6b-base.gguf	cstr/qwen3-tts-0.6b-base-GGUF	9b719418...af6dc
0.6B Base Q8_0	qwen3-tts-12hz-0.6b-base-q8_0.gguf	cstr/qwen3-tts-0.6b-base-GGUF	61626370...c1961
Tokenizer-12Hz codec	qwen3-tts-tokenizer-12hz.gguf	cstr/qwen3-tts-tokenizer-12hz-GGUF	70dc95db...19b

Full sha256 values and exact byte sizes: see the source report (crispasr-qwen3-tts-hip-bug-report.md), same folder.

Voice reference (Tests 1-8): redacted — real recording of a specific person, not shareable. Original available privately to maintainers on request.

Voice reference (Tests 9-10, public, use this one to reproduce): BBC recording of the late Daphne Park (MI6), from Wikimedia Commons, free license: https://commons.wikimedia.org/wiki/File:Daphne_Park_in_MI6_A_Century_in_the_Shadows_b00lv0bm.flac. Converted to 24000 Hz mono WAV (ffmpeg -i <flac> -ar 24000 -ac 1 daphne_ref.wav). Transcript, used verbatim as --ref-text: "I wanted to learn Russian. I wanted to serve in Russia. The enemy, the most interesting enemy. And also I was deeply interested by the Russians. I'd seen them on the streets of Vienna and how they behaved, and I felt anger about that. But also I felt an enormous interest in the country and the people as distinct from the philosophy, which I hated." No privacy concern — link it directly.

---

The bug, step by step

Every command below needs --i-have-rights (voice-cloning consent flag). Two test strings:

• Failing text: "Good evening. I am the voice assistant running on this workstation." (68 chars)
• Passing text: "Hello, this is a quick test of the new voice model." (51 chars)

1. Run on GPU with the failing text → it runs away

crispasr --backend qwen3-tts-1.7b-base \
  -m qwen3-tts-12hz-1.7b-base-q8_0.gguf \
  --codec-model qwen3-tts-tokenizer-12hz.gguf \
  --voice reference_voice.wav \
  --ref-text "<your ~81-token transcript>" \
  --i-have-rights \
  --tts "Good evening. I am the voice assistant running on this workstation." \
  --tts-output t1.wav

Result: talker kv cache full at frame 3796 (n_past=4095). 303.76s of audio. Exit code 0 — looks like success.

2. Same command, add --gpu-backend cpu → it's fine

Result: clean stop, ~40 frames, 3-8s of audio (varies a bit run to run — the point is it stops normally, not the exact length).

3. Same GPU run, swap to the passing text → also fine

Result: clean stop, 145 frames, 11.6s of audio.
Caveat: passing text is also shorter (51 vs 68 chars) — this test alone doesn't prove content matters more than length. See "Open question" below.

4. Swap model to 0.6B, keep failing text, GPU → still runs away

crispasr --backend qwen3-tts \
  -m qwen3-tts-12hz-0.6b-base.gguf \
  --codec-model qwen3-tts-tokenizer-12hz.gguf \
  --voice reference_voice.wav \
  --ref-text "<your ~81-token transcript>" \
  --i-have-rights \
  --tts "Good evening. I am the voice assistant running on this workstation." \
  --tts-output t4.wav

Note: 0.6B uses --backend qwen3-tts, not --backend qwen3-tts-1.7b-base.

Result: identical failure — frame 3796, 303.76s. Same as Test 1. Model size isn't the cause.

5. Same as Test 4, add --gpu-backend cpu → fine

Result: clean stop, 54 frames, 4.32s.

6. Dump per-frame tokens on both backends to find where they diverge

CRISPASR_QWEN3_TTS_EMBD_CHECK=1 crispasr --backend qwen3-tts-1.7b-base \
  -m qwen3-tts-12hz-1.7b-base-q8_0.gguf --codec-model qwen3-tts-tokenizer-12hz.gguf \
  --voice reference_voice.wav --ref-text "<your transcript>" --i-have-rights \
  --tts "Good evening. I am the voice assistant running on this workstation." \
  --tts-output t6_gpu.wav > t6_gpu.log 2>&1

Repeat with --gpu-backend cpu, output to t6_cpu.log. Then:

grep -oE "frame=[0-9]+ cb0=[0-9]+" t6_gpu.log | head -5
grep -oE "frame=[0-9]+ cb0=[0-9]+" t6_cpu.log | head -5

Result: see the divergence table below. GPU log reproduces the runaway independently; CPU log stops cleanly at frame 43.

7. Swap to F16 (not quantized), keep failing text, GPU → still runs away

Same as Test 1, -m qwen3-tts-12hz-1.7b-base-f16.gguf instead.

Result: identical failure. Quantization isn't the cause (Test 1 = Q8_0, this test = F16, same model size, same outcome).

8. Add --no-flash-attn, keep failing text, GPU → still runs away

Same as Test 1 plus --no-flash-attn.

Result: identical failure. Doesn't matter anyway — this build has GGML_HIP_ROCWMMA_FATTN=OFF compiled in, so the flag has nothing to disable. Confirms the flag has no effect, not that flash-attention was ever active.

9. Swap to a totally different voice reference, keep failing text, GPU → still runs away

Same as Test 1 but --voice daphne_ref.wav --ref-text "<Daphne Park transcript, see above>" — a completely unrelated speaker and recording.

Result: frame 3795 (vs Test 1's frame 3796 — same failure, essentially identical), 303.68s. Voice reference isn't the cause either.

10. Same as Test 9, add --gpu-backend cpu → fine

Result: clean stop, 43 frames, 3.44s. Matches the CPU pattern from every other test.

---

Where the two paths first disagree

First 5 sampled tokens (cb0 = codebook-0), same input, GPU vs CPU:

frame	GPU token	CPU token
0	1290	1543
1	2005	1365
2	2005	146
3	2005	532
4	882	129

They disagree from frame 0. Not a slow drift — wrong from the very first decode step. This pins down when it goes wrong, not why — the actual ggml/HIP op responsible hasn't been found.

---

Ruled out

Suspect	Verdict	Evidence
Quantization	❌ not it	Q8_0 (Test 1) and F16 (Test 7), same model size → same failure
Model size	❌ not it	1.7B (Test 7) and 0.6B (Test 4), same quantization → same failure
--no-flash-attn flag	❌ not it	Test 8, no change — also moot, rocWMMA fattn compiled off
HIP graph capture	❌ not it	GGML_HIP_GRAPHS=OFF in every test, including failing ones
Voice reference identity	❌ not it	Test 9, totally unrelated speaker/recording → same failure (frame 3795 vs 3796)
CLI --temperature / --seed	❌ not wired up	Talker's sampler hardcodes top_k=50, temp=0.9 in source (qwen3_tts.cpp:6945-6946) — CLI flags never reach it

---

Impact

GPU path: no error, exit code 0, valid WAV file written — but it silently ran ~30-90x longer than it should (303.76s of audio vs under 10s). The only signal is one informational stderr line: qwen3_tts: talker kv cache full at frame 3796 (n_past=4095). Any caller that isn't scraping logs sees a "successful" run.

[CrispStrobe]

Thank you — the paired tests ruling out quantization, model size, flash-attn, HIP graphs and the voice reference saved a lot of time, and the note about the sampler being hardcoded was the key. Two fixes are on main (f67abe3), and I have a result you'll want.

First: your frame-0 table was measuring through a confound

You spotted that qwen3_tts.cpp:6945-6946 hardcodes top_k=50, temp=0.9 and that the CLI flags never reach it. That's now fixed — qwen3_tts_set_temperature reached the code predictor but not the talker, so --temperature was inert on the decision that picks each frame.

But the deeper consequence: with top-k sampling, a CPU-vs-GPU token table can't distinguish a miscompute from noise. The RNG stream is identical across backends (fixed-seed xorshift), but the pick is a multinomial draw over a softmax of the top 50 logits, so any float difference — including the legitimate ones every backend has — can move it. So "they disagree from frame 0" was real but not yet evidence.

New lever: CRISPASR_QWEN3_TTS_GREEDY=1 forces top_k=1. The frame sequence becomes a pure function of the logits, so two backends agree if and only if their logits agree — and a divergence is a miscompute. (Same lever as CRISPASR_COSYVOICE3_GREEDY, which is what settled our native-Vulkan post-mortem in #304.)

Second: it is not HIP-specific — it reproduces on Metal

I ran your failing text on an M1 with the 0.6B q8_0 base and the baked default voice pack, greedy, both backends deterministic run-to-run (verified twice each):

	frames	first divergence vs CPU	continuation
CPU	92	—	varied tokens
Metal	107	frame 6	142, 142, 1657, 1657, 1657, 1657, 668, 668, 668, 668, 668, 668 …

Frames 0–5 are identical, then Metal diverges and immediately falls into a repetition attractor — the same shape your HIP run ends in. Metal doesn't run all the way to the ceiling (it stops at 107), so it's a milder presentation, but it's the same GPU-path divergence.

That matters because it means this is debuggable without AMD hardware. I'd stop treating it as a ROCm bug.

Differences from your setup, stated plainly: I used the baked voice pack rather than a WAV clone, q8_0 rather than F16, and Metal rather than HIP. So "same root cause" is an inference, not proof — but the signature lines up.

Third: the silent success

You're right that this is the worst part. Hitting the KV ceiling printed an informational line and returned a valid WAV with exit code 0. It now says explicitly that the output is a runaway and should be discarded, and points at the greedy lever. Making it a non-zero exit is still open — I didn't want to change the contract for callers who might be relying on truncated output without thinking about it first.

What would help most

If you can re-run Tests 1 and 2 with CRISPASR_QWEN3_TTS_GREEDY=1 on both backends, that tells us whether HIP diverges at frame 0 under greedy or later like Metal. Frame 0 under greedy would point at the prefill; a later divergence points at something accumulating in the decode loop.

Next steps on my side are recorded in PLAN.md: bisect where the logits first differ at the divergent frame (talker forward vs code predictor) via CRISPASR_QWEN3_TTS_DUMP_DIR, then work the usual GPU suspects — a weight-less first op on the scheduler, a non-contiguous get_rows index (CUDA and HIP assert on that where Metal tolerates it), and cached-graph input aliasing across steps.

[CrispStrobe]

Correction to my previous comment, and the actual fix (6b95c86).

I said the bug "reproduces on Metal, so it is not HIP-specific" and implied a GPU-path defect. I measured further and that reading was wrong. Sorry — I should have dumped the logits before saying it.

There is no miscompute

Tokens can't answer this question, so I added CRISPASR_QWEN3_TTS_DUMP_LOGITS=<dir> — raw per-frame talker logits, written before the repetition penalty and the suppress mask (the penalty reads the generated history, which itself diverges once the picks differ, and would confound every later frame). CPU vs Metal, greedy:

frame	cos(CPU, Metal)	max_abs	argmax
0	0.999924	0.44	same
3	0.990327	6.56	same
5	0.840977	19.6	flips
7	0.685161	25.3	different

cos 0.99992 at frame 0 is better than the 0.998–0.999 we treat as normal for GPU-vs-CPU. And neither --no-flash-attn (bit-identical output) nor CRISPASR_KV_QUANT_{K,V}=f32 (moves the 5th decimal) changes it — so there's no precision knob to turn and no op to blame.

What you're seeing is ordinary backend arithmetic amplified by the autoregressive loop: identical at frame 0, argmax flips around frame 5, and the two backends then follow different — but individually plausible — trajectories. Yours happened not to terminate. That also explains why every suspect you tested came back negative: none of them was ever the cause.

The obvious fix was wrong too

core_repeat::tail_is_repetition exists here, three other backends use it, qwen3-tts never adopted it, and your failure repeats visibly. I nearly shipped it. Then I measured the healthy arm:

healthy CPU  ("quick brown fox"): period-1 [1657] repeated 7x, mid-utterance
degenerate Metal                : period-1 [668]  repeated 8x, mid-utterance

Structurally identical. That detector was written for text tokens; at 12 Hz a held sound legitimately repeats a codec frame. Enabling it would have truncated good audio — worse than the bug.

What actually landed

max_frames was the KV ceiling, so any input was allowed 4096 frames (340 s). That is the defect: not that the trajectory went bad, but that a bad trajectory could run 40× longer than the text could justify and report success.

It's now bounded by the input text — max(240, codepoints × 12) — the same max_token_text_ratio idea upstream TTS models carry. Sized from measurement, not taste: five utterances on this model ran 1.35–2.61 frames per codepoint, so 12 is ~5× the worst observed; all five are untouched (79–124 frames against caps of 528–804). max_codec_steps and CRISPASR_QWEN3_TTS_MAX_FRAMES still override, and it only ever tightens the ceiling.

Hitting either bound now prints an explicit error saying the output is a runaway and should be discarded. Plus, from before: --temperature now actually reaches the talker, and CRISPASR_QWEN3_TTS_GREEDY=1 gives you argmax.

One thing I'd still like from you

Your HIP run diverged at frame 0 under sampling. Does it also diverge at frame 0 under greedy? Metal doesn't — frames 0–4 match exactly. A frame-0 greedy divergence on HIP would be a different story from what I measured, and would point at the prefill rather than the loop. CRISPASR_QWEN3_TTS_GREEDY=1 CRISPASR_QWEN3_TTS_DUMP_LOGITS=/tmp/lg on both backends and the frame-0 cosine would settle it.

[CrispStrobe]

Settled properly (cce424ac). My earlier "no miscompute" call was the right conclusion reached with insufficient evidence — here is the version that actually establishes it.

Why the earlier comparison couldn't decide anything. Free-running, once the argmax flips at one step, every step after is conditioned on different history. So a CPU-vs-GPU logits diff past that point measures the trajectory divergence, not the arithmetic. Frame 0 was clean (cos 0.99992) and I extrapolated from it.

Teacher forcing settles it. New CRISPASR_QWEN3_TTS_REPLAY_CODES=<file> takes 16 codec ids per frame and uses them instead of sampling, so both backends walk an identical trajectory while still computing every logit. CPU vs Metal, 49 steps, all 16 codebooks pinned:

worst cos = 0.999870   mean = 0.999940   argmax disagreements = 0/49

Given identical history the two backends agree at every single step. There is no miscompute — not at frame 0, not anywhere. The divergence you saw is entirely trajectory divergence seeded by ~1e-4 arithmetic differences, which is what the frame bound in v0.8.26 addresses. There is no op to chase, on HIP or anywhere else.

One trap worth passing on, since you may try this: replaying codebook 0 alone is not teacher forcing. code_pred_generate_15 must sample — greedy there is documented to produce silent output — so with only cb0 pinned the other 15 codebooks still diverge, the next frame's input embedding differs, and the diff bottoms out at cos 0.849 measuring nothing. I had that artefact on screen before spotting it. Pin all 16 (REPLAY_CODES) or measure nothing; REPLAY_TOKENS is the cb0-only form and is for forcing a trajectory, not for diffing.

If you want to run the same check on HIP:

# CPU: capture the trajectory and the logits
CRISPASR_QWEN3_TTS_GREEDY=1 CRISPASR_QWEN3_TTS_DUMP_DIR=codes \
CRISPASR_QWEN3_TTS_DUMP_LOGITS=lg_cpu crispasr ... --gpu-backend cpu --tts "..."
# reshape codes/generated_codes_s00.bin (int32) to 16 per line -> codes16.txt
# GPU: replay it
CRISPASR_QWEN3_TTS_GREEDY=1 CRISPASR_QWEN3_TTS_REPLAY_CODES=codes16.txt \
CRISPASR_QWEN3_TTS_DUMP_LOGITS=lg_gpu crispasr ... --tts "..."

Compare talker_s00_*.f32 pairwise. Note the _s00 — one --tts run generates twice (your utterance, then the spoken AI disclaimer), and the dumps used to be frame-only names so the second overwrote the first. That's fixed in the same commit; it had already produced one bogus table for me.

If HIP shows the same near-1.0 agreement under full replay, your bug is closed by the v0.8.26 frame bound. If it does not — if HIP disagrees with CPU under identical history — that would be a genuine miscompute and I'd want to see it.

[adam-schneider-dev]

I will look into this after I get off work. Thank you.

[adam-schneider-dev]

Ran the greedy/replay comparison on this box (Ryzen 7800X3D + RX 7900 XTX, gfx1100). Built fresh at v0.8.28 (8b91f4c5) to get the diagnostic env vars.

Result: this does not match the CPU-vs-Metal control.

	your CPU-vs-Metal (n=49)	CPU-vs-HIP here (n=60)
worst cos	0.999870	0.449509 (frame 0)
mean cos	0.999940	0.916888
argmax disagreements	0/49	30/60

Frame 0 is the worst frame, not a middle one — under full 16-codebook teacher forcing that points at the prefill, not something accumulating in the decode loop.

Two things I checked and ruled out

This box has two ROCm devices visible (7900 XTX + an iGPU), so I first suspected a multi-GPU artifact, not a real backend bug. Neither of these two changes moved the numbers at all (bit-identical to 6 decimals):

• Pinned to one device: HIP_VISIBLE_DEVICES=0 (matches how this project already pins its LLM server for the same reason) — same result.
• --no-flash-attn — same result.

Positive controls (methodology check)

Before trusting the numbers above, I checked the comparison pipeline itself doesn't lie:

• Replayed the CPU-captured codes back through --gpu-backend cpu: bit-exact match (cos 1.000000, max_abs 0.0, 0/60 disagreements) across all frames.
• Frame 0's step00 embed (the first thing computed) also reproduces CPU-vs-itself exactly (cos 1.0).

One caveat on the finer-grained DUMP_DIR embed comparison: steps 1–14's embed tensors do not reproduce even CPU-vs-CPU on a repeat run (cos ~0.24–0.34) — so I'm not treating those as evidence either way, just flagging it in case it's useful to you (possibly expected, possibly worth a look separately from this issue).

What is clean: frame 0 / step00's embed, which reproduces exactly on CPU, diverges to cos 0.129 CPU-vs-HIP. Same conclusion as the logits comparison — points at the prefill.

Steps to reproduce

# 1. CPU capture (greedy, dump codes + logits)
CRISPASR_QWEN3_TTS_GREEDY=1 \
CRISPASR_QWEN3_TTS_DUMP_DIR=codes_cpu \
CRISPASR_QWEN3_TTS_DUMP_LOGITS=lg_cpu \
crispasr --backend qwen3-tts \
  -m qwen3-tts-12hz-0.6b-base-q8_0.gguf --codec-model qwen3-tts-tokenizer-12hz.gguf \
  --gpu-backend cpu \
  --voice voice.wav --ref-text "<voice ref text>" --i-have-rights \
  --no-spoken-disclaimer --accept-marking-responsibility \
  --tts "Hello, this is a diagnostic test of the text to speech system." \
  --tts-output cpu_output.wav -v

# 2. Reshape the generated codes to 16-per-line
python3 -c "
import numpy as np
codes = np.fromfile('codes_cpu/generated_codes_s00.bin', dtype='<i4').reshape(-1, 16)
np.savetxt('codes16.txt', codes, fmt='%d')
"

# 3. Replay on HIP, single device pinned, logits dumped
HIP_VISIBLE_DEVICES=0 \
CRISPASR_QWEN3_TTS_GREEDY=1 \
CRISPASR_QWEN3_TTS_REPLAY_CODES=codes16.txt \
CRISPASR_QWEN3_TTS_DUMP_LOGITS=lg_gpu \
crispasr --backend qwen3-tts \
  -m qwen3-tts-12hz-0.6b-base-q8_0.gguf --codec-model qwen3-tts-tokenizer-12hz.gguf \
  --gpu-backend auto \
  --voice voice.wav --ref-text "<same voice ref text>" --i-have-rights \
  --no-spoken-disclaimer --accept-marking-responsibility \
  --tts "Hello, this is a diagnostic test of the text to speech system." \
  --tts-output gpu_output.wav -v

# 4. Compare per-frame talker logits
python3 - <<'EOF'
import numpy as np, glob, os

def load(d): return sorted(glob.glob(os.path.join(d, "talker_s00_*.f32")))

cpu_files, gpu_files = load("lg_cpu"), load("lg_gpu")
n = min(len(cpu_files), len(gpu_files))
worst = (1.0, -1)
disagree = 0
for i in range(n):
    a = np.fromfile(cpu_files[i], dtype="<f4")
    b = np.fromfile(gpu_files[i], dtype="<f4")
    cos = float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-12))
    if cos < worst[0]: worst = (cos, i)
    if np.argmax(a) != np.argmax(b): disagree += 1
print(f"worst cos = {worst[0]:.6f} at frame {worst[1]}")
print(f"argmax disagreements = {disagree}/{n}")
EOF

Expected output here: worst cos = 0.449509 at frame 0, argmax disagreements = 30/60.

Happy to run any other env var combination you want tested. Can also share the raw talker_*.f32 / *_embed.bin logit dumps directly if useful — but not generated_codes_s00.bin or the .wav outputs, since those are the actual synthesized audio in a voice-cloned reference speaker's voice and I don't have consent to distribute that publicly. If you want dumps from a run with no voice-cloning concern at all, say so and I'll redo it with a synthetic/non-cloned voice.