fix(ROCM): AttnQkNormRopeGate output-dtype dispatch -- the 0.8B GDN forward divergence root-caused and fixed (RED anchor turns GREEN) - #570
Draft
VikashLoomba wants to merge 5 commits into
Conversation
…lden lane in the Qwen3-0.6B near-tie gate Issue mudler#41 M4 milestone. The oracle is upstream vLLM AT THE PIN (555967922, runtime 0.23.1rc1.dev1511+g555967922 -- verified == the parity-pin block) built from source as a ROCm wheel for gfx1100 in rocm/vllm-dev:base and committed as the image vllm-rocm-oracle:555967922-gfx1100 (recipe: .agents/specs/rocm-m4-oracle.md). Gate result on this box (4x RX 7900 XTX, ROCm 7.14, enforce_eager, batch=1, flock ~/gpu.lock): oracle K=10 per-prompt greedy deterministic in every cell; our ROCm engine 16/16 prompts PASS -- 11/16 strict token-exact vs base greedy, 5/16 near-tie band, max teacher-forced gap 0.125 nats, 0 forward-divergent, 125/125 assertions, backend proof all 8 Qwen3-dense ops selections>0 and declines==0 on device 5. The p0 France/Italy flip is a literal 0.0000-nat tie in the oracle's own logits (the Metal M3b signature, reproduced on ROCm). Change shape: the parity gate's Metal/Tenstorrent device-golden lane gains the kROCM arm (identical anchor+band logic, ROCm's own oracle-backed golden pair); dgx base goldens byte-untouched; the *_rocm goldens are additive. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]
…e, committed as the RED anchor of the forward-divergence fix Issue mudler#41 M4, second model. The GDN sibling of the Qwen3-dense gate: test_qwen35_paged_engine drives the standard 16-prompt battery through the full paged engine and holds our ROCm tokens against the pinned vLLM-ROCm oracle (555967922 / 0.23.1rc1.dev1511+g555967922, same container as the 0.6B lane) via the anchor + <=0.5-nat near-tie band, with the GDN backend proof (all 15 dispatched ops selections>0 and declines==0). VERDICT AT LANDING: RED BY DESIGN -- the gate FAILS 13/16 prompts (0/16 strict, 3/16 near-tie band, max teacher-forced gap 14.125 nats; first-token divergences on 6 prompts). This is the known 0.8B CPU/ROCm divergence, now oracle-measured: a REAL forward divergence, not a distributional near-tie. The oracle K=10 capture is deterministic in every cell. Backend proof is GREEN (kPagedAttention 1,536 / kGdnDecode 4,320 selections, 0 declines) -- the divergence is numerical, not dispatch. This commit is the RED test of the fix lifecycle: the gate goes GREEN when the 0.8B forward lands. Goldens carry a manifest.json (oracle identity + the real HF revision 2fc06364...); the gate resolves the checkpoint through parity::HfSnapshot at that pinned revision (check-snapshot-pins clean). hf_snapshot.h gains the kQwen35_08BRevision pin + resolver, and a forward declaration fixing the call-before-declaration order hipcc rejects (mudler#555 carries the canonical main-side fix). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]
…RED anchor at landing, 13/16 forward-divergent, max gap 14.125 nats) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]
…he 0.8B GDN forward divergence, root-caused and fixed; the RED anchor turns GREEN Issue mudler#41, the M4 RED anchor's fix. The ROCm dispatcher for the fused full-attention preamble keyed its template on the SOURCE dtype (qgate.dtype) instead of the OUTPUT dtype like the CUDA lane (LaunchAttnPreambleOut switches on q_out.dtype). The 0.8B bf16 model runs bf16 QKV-projection output + f32 q/k/gate outs (the f32-attention path; no FA-2 on ROCm), which the src-keyed dispatch mis-launched as all-bf16 -- writing bf16 bits through the f32 out pointers. In-context evidence (VT_DUMP_ATTN per-stage dumps): fa0_q rms-rel 1.196 vs CPU with fa0_qkv/fa0_gate clean; layer-3 (first full-attn layer) block_out rms-rel 4.4e-1 with layers 0-2 clean; VT_ROCM_ATTN_CPU_REF=1 exonerated the paged-attention core. Fix: dispatch on q_out.dtype; bf16-out requires bf16-src (VT_CHECK). Teeth: the cross-device AttnQkNormRopeGate case gains the bf16-src -> f32-out combo at the real 0.8B dims (Hq=8, Hkv=2, Dh=256, rot=64); mutation-proven (fails with the fix reverted, passes with it). Gates (gfx1100, flock ~/gpu.lock): - test_qwen35_paged_engine: 16/16 PASS (15/16 strict token-exact vs the pinned vLLM-ROCm oracle, 1/16 near-tie band, max gap 0.125 nats, 0 forward-divergent) -- the RED anchor from the parent commit now GREEN - test_qwen3_paged_engine (0.6B regression): 16/16 PASS unchanged - test_backend_cross_device: 19/19 (346 assertions) incl. the new combo - full HIP ctest: zero-delta vs the parent (the box-baseline failures -- async-scheduling-disabled gates, safetensors RSS mapping, op-63-less bench model, server/embedding ROCm gaps -- are identical on the parent); test_op_parity repaired (the golden manifest's op name joins PendingRunnerOps like the qwen36_gguf_greedy precedent) - e2e: 'The capital of France is' -> ' Paris.\nThe capital of France is', matching the pinned oracle AND our CPU backend token-for-token Also: three pre-existing hipcc/clang-only build breaks fixed (they block any full HIP build on this toolchain): same-line double CAPTURE in test_qwen3_5_gdn_spec_routing.cpp and test_ops_quant_repack.cpp (doctest redefinition), and -Wunused-const-variable on kGdnLayers in test_qwen27n_fp8_tower_paged_engine.cpp (its uses compile out on HIP). Instrumentation shipped env-gated and off by default (VT_DUMP_ACT / VT_DUMP_ACT_SUB / VT_DUMP_ATTN; deepseek_v4 VT_DUMP_ACT precedent; env-doc-allowlist updated). The 0.8B goldens (our_ids.npy + neartie_gap_mnats.npy) are re-derived from the FIXED engine and re-teacher-forced by the pinned oracle (manifest updated); docs/FEATURES.md + docs/STATUS.md updated in the same change; spec .agents/specs/rocm-gdn-08b-fix.md carries the full W1-W4 evidence. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]
…tch root cause, evidence chain, gates) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]
This was referenced Aug 13, 2026
This was referenced Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The 0.8B GDN forward divergence — root-caused and fixed; the M4 RED anchor turns GREEN (issue #41)
Stacked on #559 (the 0.8B gate) ← #549 (the 0.6B lane + oracle). This PR fixes the divergence that #559's gate pinned RED.
Root cause
AttnQkNormRopeGateKernelRocm(the fused full-attention preamble) dispatched its kernel template on the source dtype (qgate.dtype) instead of the output dtype — the CUDA lane (LaunchAttnPreambleOut) switches onq_out.dtype. The 0.8B bf16 model runs bf16 QKV-projection output + f32 q/k/gate outs (the f32-attention path; no FA-2 on ROCm), which the src-keyed dispatch silently mis-launched as all-bf16 — writing bf16 bits through the f32 out pointers.Evidence chain (each step on gfx1100, GPU-locked)
VT_DUMP_ACTper-layer residual dumps, CPU vs ROCm): layers 0–2 clean (rms-rel ≤ 9.6e-3), layer 3 — the first full-attention layer — ignites to 2.1e-1.VT_DUMP_ACT_SUB):post_input_normclean (6.8e-3),block_out4.4e-1 → inside FullAttnBlockPaged.VT_DUMP_ATTN):fa0_qkvclean (3.2e-3),fa0_gateclean (3.1e-3),fa0_qrms-rel 1.196 — 100% wrong → the fused preamble's q/k path.VT_ROCM_ATTN_CPU_REF=1(CPU-oracle attention core) changed nothing → attention core exonerated.Fix
Dispatch on
q_out.dtype(bf16-src + f32-out is now a first-class combo;VT_CHECKbf16-out requires bf16-src).Teeth (mutation-proven)
The cross-device
AttnQkNormRopeGatecase gains the bf16-src → f32-out combo at the real 0.8B dims (Hq=8, Hkv=2, Dh=256, rot=64). With the fix reverted it FAILS (2 assertions); with the fix it passes.Gates
test_qwen35_paged_engine: 16/16 PASS (15/16 strict token-exact vs the pinned oracle, 1/16 band, max gap 0.125 nats, 0 forward-divergent) — the RED anchor GREENtest_qwen3_paged_engine(0.6B): 16/16 unchangedtest_backend_cross_device: 19/19 (346 assertions)Paris.\nThe capital of France isAlso in this PR (each needed to run the full HIP suite here): three pre-existing hipcc/clang-only build breaks fixed (same-line double
CAPTURE×2,-Wunused-const-variableon a CUDA-gated constant), and the env-gated dump instrumentation that found the bug (VT_DUMP_ACT/VT_DUMP_ACT_SUB/VT_DUMP_ATTN, off by default, allowlisted).