Canonical grids measure vLLM WITHOUT --language-model-only: the oracle runs the UNFUSED QK-norm+RoPE+gate path while we run the fused one — our published ratios are FLATTERED

[localai-bot]

Found auditing vllm.ai 2026-08-06 "25K tok/s/GPU" against our tree. This is a denominator defect, and it runs in our favour — fixing it makes our published parity ratios WORSE.

The article's lever

--language-model-only "not only disables multimodal inputs but also unlocks the fused QK-norm + RoPE + gate path in the attention layers"

C10 captured three levers from this article and did not capture this one.

Why it applies to us, unlike the GDN prefill lever

The fused kernel has NO compute-capability gate — the predicate is current_platform.is_cuda() and nothing more:

• flag: vllm/engine/arg_utils.py:1276
• predicate: vllm/model_executor/models/qwen3_next.py:322-330
• kernel: vllm/model_executor/layers/fused_qk_norm_rope.py:117-201
• Qwen3.5 reaches it via qwen3_5.py:146 -> Qwen3NextAttention

GB10 sm_121 is fully eligible. Contrast lever (a), the Blackwell GDN prefill, which is family(100)-gated and hardware-blocked here.

But it is OFF by default for our checkpoints

Our gate models load as Qwen3_5ForConditionalGeneration / Qwen3_5MoeForConditionalGeneration, so multimodal_config is non-None and language_model_only defaults False (vllm/config/multimodal.py:78), leaving text_only == False and the fusion disabled.

And our canonical driver never passes the flag

scripts/dgx-online-serving.sh:468-478 -- the vLLM serve arm passes --gpu-memory-utilization, --max-num-seqs, --max-num-batched-tokens, --no-enable-prefix-caching, --mamba-ssm-cache-dtype float32, --port. No --language-model-only.

Meanwhile tools/bench/run_serve_low.py:549 does pass it. Across the repo the flag appears exactly twice. Our two harnesses disagree about the oracle's configuration.

Meanwhile our arm runs the fusion

FuseAttnPreambleOn() is default ON (src/vllm/model_executor/models/qwen3_5.cpp:1679), dispatching ONE launch through vt::FusedChain (:4660, recipe vt::kAttnQkNormRopeGate, include/vt/recipes.h:247, CUDA src/vt/cuda/cuda_ops.cu:1316) where the fallback issues four ops: AttnGateSplit -> RmsNorm(q) -> RmsNorm(k) -> RopeNeox (:4668-4682).

Consequence

The canonical 27B 0.9371x-0.9561x and 35B 0.918x-0.972x were measured with our arm FUSED and the oracle's arm UNFUSED, in every full-attention layer.

The kernel's grid is (n_tokens, Hq+Hkv), so its absolute cost scales with prompt tokens -- it is prefill-weighted, and prefill is exactly where our deficit sits (mean TTFT 0.872-0.972x, docs/BENCHMARKS.md:132).

AGENTS.md: "The honest denominator is vLLM's production configuration." An oracle running four ops where its own production config would run one is not that.

Direction of the error is AGAINST us. Repairing it makes the oracle faster and our ratio lower. This is debt to settle, not a win to harvest.

Owed

1. Add --language-model-only to the vLLM arm of scripts/dgx-online-serving.sh and re-run the 27B and 35B canonical grids. Reconcile with run_serve_low.py:549 so the two harnesses agree.
2. Update docs/BENCHMARKS.md / STATUS / NOW with the corrected ratios in one change, and record WHY they moved -- a ratio that drops for a denominator repair must not read as a regression.
3. Re-read our kernel's deliberate omission of upstream's bf16 round-trip of normed q/k before RoPE (fused_qk_norm_rope.py:67 vs the NOTE at src/vt/cuda/cuda_ops.cu:1307-1314). Documented as op-level bit-identical, but it flipped a 27B near-tie once, and this kernel is now known to sit on the critical prefill path.

Also from the same audit, for the record

• --async-scheduling (#48481 -> 530852f95, #45357 -> d467a2a7f, both verified pin ancestors) is already default-ON at the pin for our config (vllm/config/vllm.py:1064-1112) and on our side. C10's "pin move required" is false for this too.
• --max-num-batched-tokens: the article recommends 2x ISL; our driver already passes 8192 (35B) and 2048 (27B) at ISL 1024 on both arms. Already met.
• --mamba-ssm-cache-dtype bfloat16: NOT applicable -- our checkpoints pin float32, and the bf16 stochastic-rounding mitigation is family(100)-gated and hard-raises on sm_121 (vllm/config/mamba.py:66-77). Adopting it would break token-exactness.
• --max-cudagraph-capture-size, --stream-interval, --api-server-count: not applicable at c1-c32 on one GPU; --stream-interval would actively corrupt our latency axes.