We emit F32 where vLLM emits bf16 on the biggest prefill buffers (GDN in_proj, attn gate, q/k/v) — 2x traffic, per-token, flat across concurrency; and the canonical harness ENFORCES the 0.25.0 rollback oracle

[localai-bot]

Two findings from a full audit of vLLM's Qwen paths at the pin vs ours.

FINDING 1 — F32-vs-bf16 output dtype on the fp8 path

buffer	ours	vLLM	layers
GDN in_proj_qkvz out (mixed_qkv, and hence convdt, conv1d in/out, post-conv read)	F32, hardcoded DType::kF32 at qwen3_5.cpp:3292 (merged) and :3409 (split); consumed :4155 convdt = mixed.dtype	bf16 — ModelOptFp8LinearMethod uses out_dtype = torch.get_default_dtype() (modelopt.py:458); conv runs bf16 (qwen_gdn_linear_attn.py:1285-1295)	48/64 (27B), 40/40 (35B)
attention gate buffer gatef {T,Hq,Dh}	F32 (qwen3_5.cpp:4652, consumed :4754-4755) — ~25.2 MB/layer at 27B prefill	bf16 slice of the qkv projection (qwen3_next.py:367-375,397-398)	16 (27B)
q/k/v projection outputs, fp8 branch	F32 (qwen3_5.cpp:2322-2323, :2331-2336) — Bf16GemmOutEnabled() only guards the fp4 branch	bf16 (modelopt.py:458)	16 (27B), 10 (35B)

One root cause: on FP8-tower checkpoints our GEMM out-dtype is hardcoded F32 where vLLM's is bf16. That is 2x memory traffic on the largest activation buffers in the model.

It matches the gap signature exactly

Per-token cost -> prefill-dominant and flat across c1-c32, which is the property that ruled out every launch-count lever this campaign chased. Our measured deficit is prefill (27B 0.9371-0.9561x canonical; decode ~0.99x on a decode-dominant harness).

The lever already exists and is UNREACHABLE

VT_GDN_IN_BF16 is implemented and default ON, with recorded measurements of conv -31.5% and post-conv -17.6%. It cannot fire on the nvidia 27B or the 35B because the fp8 GEMM's output dtype is hardcoded, so mixed_qkv is F32 regardless.

This is the same axis as the 35B both-arms trace, where ours emitted nvjet_qqsss (f32-out) and vLLM nvjet_qqtst (bf16-out) at an identical 192x24x128 shape, theirs 3.9% faster. That was the cleanest same-shape dtype contrast measured this campaign and it now has a mechanism.

Ranked follow-ons from the same audit

• Full-attention QKV is three FP8 GEMMs, not one. MergedFp8QkvEnabled() is default OFF (qwen3_5.cpp:1512-1518) and additionally demands bit-exact input_scale equality across q/k/v (:1530-1534). vLLM's QKVParallelLinear is always one physical projection and requantizes mismatched shards to max_w_scale (modelopt.py:519-529) rather than declining. Note the parity implication: vLLM's requantization is lossy where ours is exact, so tokens can legitimately diverge on such a checkpoint.
• No quantize-once / fused RMSNorm->fp8-quant on the DENSE (27B) path. InputLayernormFp8 is MoE-only (qwen3_5.cpp:5836-5866); RunDenseLayerPaged (:6129-6142) passes no h_fp8, so every fp8 projection re-quantizes the whole [T,H] activation. Independently corroborates 27B: a STALE COMMENT keeps the dense path off an already-built, default-ON, byte-exact quant fusion — 112 -> 16 QuantFp8Static launches/step (and the epilogue-fusion premise is REFUTED) #402's Lever A, which found the same thing via a stale comment at :1577.

FINDING 2 — the canonical harness ENFORCES the rollback oracle

tools/bench/online_gate.py:53-54
VLLM_ORACLE_VERSION = "0.25.0"
FLASHINFER_VERSION  = "0.6.13"

and it raises on drift (:3505-3513, HarnessError: vLLM oracle version drift).

The recorded parity pin is 555967922 = 0.26.0.dev0 + FlashInfer 0.6.15.post1 (.agents/upstream-sync.md:7-9).

So #375 is worse than "a symlink pointed at a rollback": the benchmark itself demands 0.25.0 and would REJECT the pin. Every canonical ratio -- 27B 0.9371-0.9561x, 35B 0.918-0.972x -- was produced against an oracle the harness enforces and the pin record contradicts.

Measured today for what it is worth: the 0.25.0 rollback and the rebuilt 0.26 pin are equivalent in speed on the 27B at c1 (rollback/pin 0.9983 mean, 0.9996 median, OVERLAPPING, n=3). So this is a provenance defect, not a numbers defect -- but advancing the harness constant is a deliberate pin-advance action requiring reconciliation per AGENTS.md, not a one-line edit.

Correction to a claim in the audit that produced this

The audit asserted vLLM does NOT run fp8 KV, on the grounds that cache_dtype is CLI-only. Refuted by runtime: the pinned oracle's own startup config prints kv_cache_dtype=fp8_e4m3 and kv_cache_dtype=torch.float8_e4m3fn, and get_kv_cache_quant_algo_string exists at vllm/utils/torch_utils.py:310-341 reading kv_cache_quant_algo from the checkpoint. #415 stands. Recording it because this is the fourth time in this campaign a confident source reading has lost to a runtime log.