GLM-5.2 v14: ~41 tok/s cc1 instead of ~126-136 on 8x RTX PRO 6000 (EPYC 9565, PEX89144) — fixed ~12-16 ms of spin per forward pass, SM=100% at only ~140 W

[repandv]

Following the v14 recipe as closely as possible, we get ~3x lower single-stream decode throughput than the published numbers, with a clear spin-wait signature. Full diagnostics below — the box is available for any experiments you suggest.

Environment

• Host: 2x AMD EPYC 9565 (Turin, 72c), 2 NUMA nodes, 768 GB RAM, Ubuntu 24.04, kernel 6.8.0-1045-nvidia, driver 595.71.05 / CUDA 13.2, iommu=pt, pcie_aspm=off, processor.max_cstate=1
• GPUs: 8x RTX PRO 6000 Blackwell Server Edition 96 GB, behind 2x Broadcom PEX89144 (4 GPUs per chip; each PEX partitioned into 2 virtual switches x 2 GPUs)
• PCIe fabric verified healthy: p2pBandwidthLatencyTest shows P2P-enabled latency 0.44-0.53 us uniformly across all 28 GPU pairs (incl. cross-NUMA), unidirectional 48-55 GB/s
• Image: voipmonitor/vllm:dev-eldritch-enlightenment-vllmc382f1d-fp8d005934-b12xe44cb77-it85e7c5f-cu132-20260706 (running digest sha256:2dfb16d1... with the embedded launcher; pinned e15f3946... also verified)
• Checkpoint: lukealonso/GLM-5.2-NVFP4, all 90 shards incl. layer-78 MTP
• Config: TP=8, DCP=1, MTP=3, MOE_MODE=a4, MAX_NUM_SEQS=64, GRAPH=256, fp8 KV, B12X_MLA_SPARSE, b12x MoE backend, 78-char index_topk_pattern
• LOAD_FORMAT=fastsafetensors — InstantTensor open throws RuntimeError: std::exception on all workers on this host (io_uring is enabled); separate issue, shouldn't affect inference speed

Symptom

• cc1 decode: 41-44 tok/s vs your 125.9-136.5 (MTP3, a4)
• MTP acceptance is healthy (mean length 2.8-3.5, per-position 0.85-0.95 / 0.6-0.8 / 0.4-0.7) — matching your benchmark values, so the draft path is fine
• cc2 scales linearly to ~76 tok/s → fixed per-step cost, not compute-bound
• During decode: SM=100% on all 8 GPUs but only ~140 W/GPU, mem-util 10-11%, pclk 2280-2340 MHz (no throttling, temps 38-41 C) — spin-wait signature, not real compute
• Startup log is clean: 57 indexer-skip lines, Configured b12x PCIe crossovers: oneshot/fused max=86016, DMA min=6291456, ['B12X_PCIE_ONESHOT_DMA', 'PYNCCL'] for tp:0, PYNCCL for ep:0, AOT cache hit, JIT monitor silent during inference

What we ruled out (each a separate restart, 2 measurements each)

1. NCCL tuning (graph XML, P2P_LEVEL=SYS, MIN_NCHANNELS etc.): no change — expected, decode allreduce dispatches to B12X_PCIE_ONESHOT_DMA
2. DCP_BACKEND=ag_rs: 43 tok/s, no change
3. Pure image defaults (no host env overrides at all): 41/44 tok/s
4. VLLM_USE_B12X_DCP_A2A=0 (the launcher exports =1 unconditionally at line 83, and envs.py does not flag this variable as unknown, so the code reads it): 42/42 tok/s
5. Clocks / power / thermal / IOMMU: all healthy (see above), iommu=pt, P2P matrix perfect
6. MTP=0 baseline: 37 tok/s vs your 87.99 (i.e. ~27 ms per forward vs ~11.4 ms implied by your tables)

Key observation

Combining (6) with the MTP=3 numbers: at MTP=3 we see ~71 ms per engine step at acceptance ~3.0, and each step is 4 model forwards (target + 3 MTP drafts). At MTP=0 we see ~27 ms per step = 1 forward. In both cases the excess over your numbers works out to ~12-16 ms per model forward pass — proportional to the number of forwards, not per scheduler step. So whatever is spinning executes on every forward (the per-layer allreduce chain / b12x oneshot completion barriers / decode graph launch path), and is not related to MTP logic, sampling, or the scheduler.

Question

What differs between this setup and your bench environment that could add ~12-16 ms of spin per forward? Anything topology-specific we should check or flip (e.g. barrier behavior of the oneshot path on PEX89144 virtual-switch partitions, PCIe switch config expectations)? Happy to run nsys traces or any other diagnostics — turnaround on this box is fast.

[repandv]

Update: reproduced on the latest v19 image (voipmonitor/vllm:fathomless-firmament-v19-vllm0d1ad03-b12x90172a5-cu132-20260709, vLLM 0d1ad03, b12x 90172a5).

• MTP=0, cc1: 37 tok/s — identical to the v14 image (vs your 87.99). The fixed ~15 ms per forward pass is unchanged.
• Note the run config differed slightly (MAX_NUM_SEQS=1, GRAPH=4 vs 64/256 on v14) with no effect on the number — so the overhead is also independent of graph capture size and scheduler limits.
• Side note: InstantTensor now works on this host with v19 (it threw RuntimeError: std::exception on v14), loads in ~42 s vs ~60 s for fastsafetensors. So that part of the report is resolved.
• v19 log shows the allreduce crossover thresholds changed (oneshot max=65536, fused max=86016 vs unified 86016 in v14) — whatever changed there did not affect our fixed per-forward cost.

So the issue is not version-specific between 20260706 and 20260709 builds. Happy to run nsys on either image.

[repandv]

Update — reproduced on the v19 image with your own bench tooling; DCP sweep attached.

Image: voipmonitor/vllm:fathomless-firmament-v19-vllm0d1ad03-b12x90172a5-cu132-20260709 (the exact image from the glm5.2_v15.md page). Checkpoint: lukealonso/GLM-5.2-NVFP4. Benchmark: your llm_decode_bench.py via scripts/bench-glm52-v15-reduced.sh (MTP0, cc1, ctx0, SKIP_PREFILL, MAX_NUM_SEQS=1, GRAPH=6 — i.e. the reduced validation command from the v15 page).

DCP	decode tok/s	ITL avg (ms)	KV pool (tokens)
1	39.6	25.3	575,488
2	33.2	30.1	1,141,632
4	34.0	29.4	2,283,264
8	27.0	37.0	4,566,528

Your v15 page states the MTP0 sanity sweep on this image "shows no regression" vs the v14 cc1 table (87.99) on your bench host — we get 39.6 on the same image + same checkpoint + same bench. So: not build-specific (v14 image gives the same 37-42), not methodology (your bench reproduces it), not DCP-path-specific (DCP>1 is worse, as expected), not MTP/scheduler/graph-size (reproduced across MTP0/MTP3, MAX_NUM_SEQS 1/64, GRAPH 4/6/256). The fixed ~14 ms per forward sits in the common TP forward/allreduce path on this platform.

Also confirming from earlier: InstantTensor works fine on v19 here (~42 s load), while it threw RuntimeError: std::exception on the v14 image — that part is resolved.

Platform reminder: 2x EPYC 9565 (Turin), 8 GPUs behind 2x PEX89144 (2 virtual switches x 2 GPUs each), p2pBandwidthLatencyTest 0.44-0.53 us / 48-55 GB/s across all 28 pairs. Your bench host appears to be a 16-GPU box (v15 page mentions GPUs 8-15) — could the difference be in how the b12x oneshot barriers behave on our switch partitioning vs yours? Happy to run nsys or any targeted experiment; full decode.json files with startup_diagnostics available on request.

[koenfaro90]

Tell your clanker you use to write these messages to read more of the repository; especially about enabling P2P and some kernel options regarding ACS and IOMMU.

[repandv]

Thanks — checked ACS and IOMMU as suggested.

Setup: ASUS ESC8000A-E13P, 8× RTX PRO 6000 Blackwell SE (SM120a),
2× PEX89144 (each split by firmware into 2 virtual switch domains, 2 GPUs each).
Kernel already had iommu=pt, pcie_aspm=off, processor.max_cstate=1.

ACS was indeed enabled on all downstream ports (ACSCtl: SrcValid+ ReqRedir+ CmpltRedir+ UpstreamFwd+, ECAP_ACS+6.w = 001d), and IOMMU groups confirmed the
isolation — one group per GPU pair. nvidia-smi topo -m reports NODE between
domains, PIX within a pair.

I cleared ACS at runtime on the two switches backing GPUs 4-7
(setpci -s <bdf> ECAP_ACS+6.w=0000) and measured P2P before/after.
Result: no measurable difference.

pair	topo	64KB (µs)	256KB (µs)	64MB (GB/s)
4→5	PIX	10.3	16.2	54.3
6→7	PIX	10.1	16.3	54.5
4→6	NODE	10.2	15.2	54.4
4→7	NODE	10.2	16.1	54.3
5→6	NODE	9.9	17.5	54.4

NODE and PIX are indistinguishable on both latency and bandwidth — crossing
virtual switch domains costs nothing here. ACS restored afterwards.

So for us the PCIe path is not the bottleneck. A torch profile of MiniMax-M3
NVFP4 (TP4, 60 layers) shows the fixed ~11.3 ms/step budget splits roughly as:

• 26.7% ncclDevKernel_AllReduce_Sum_bf16_RING_LL — but given the numbers above,
  this is launch/barrier overhead across 120 small collectives per step, not link cost
• 22.7% cutlass_80_wmma_* bf16 GEMMs — legacy SM80 kernels in the attention path
• ~22% MoE FP4 (cutlass GroupGemm + trtllm kernels) — correct, fast path
• 4.2% GPU idle

If anyone has managed to get the small-collective overhead down on a PCIe-only
4-GPU setup (mscclpp? allreduce fusion?), I'd be interested. Posting the numbers
in case they save someone else the ACS detour.

[yumenoshizuku]

Experiencing the same issue here.

We've only tested tp=8, dcp=8 so far, which is giving us a decode speed of around 27 t/s.

Interestingly, when we use the image from kimi-k27-code.md to run Kimi K2.7 Code (using the exact same tp=8, dcp=8), the decode speed jumps to over 50 t/s. Given that they have a similar number of active parameters, the performance shouldn't be that far apart.

Is there some extra stuff in the GLM 5.2 image or a vLLM configuration difference that is slowing things down so much? Or are we missing some optimization flags?

[repandv]

Same setup here (8× RTX PRO 6000 Blackwell SE, dual PEX89144, TP=8), so I can
add a DCP sweep that may help narrow this down.

Decode scales inversely with DCP on GLM-5.2 for us:

DCP	single-stream decode
1	39.6 t/s
2	33.2 t/s
4	34.0 t/s
8	27.0 t/s

So dcp=8 is the worst point, not the best. Under real concurrent load the gap
is even wider: DCP=4 sustains up to ~227 t/s aggregate, DCP=8 tops out around
~55 t/s. We've settled on DCP=4 in production (also gives a ~2.35M-token KV pool
at util 0.93, ctx 262144).

On the GLM-vs-Kimi gap: I don't think active-parameter count is the right
lens here. GLM-5.2 runs the DSA sparse-MLA indexer (top-k recomputed per step),
which Kimi K2.7's plain MLA doesn't — so GLM is doing strictly more work per
decode step, and it gets worse as the DCP A2A crosses the PEX switch boundary.
That points at the indexer + DCP collective path rather than a general slowdown.

Worth noting: the jarrelscy/vllm-glm52-sm120 fork exposes
VLLM_MTP_INDEX_SHARE, which lets MTP draft steps reuse the verify-anchored
indexer top-k instead of recomputing it. That flag isn't present in the
fathomless-firmament build. If the indexer recompute is the dominant cost with
MTP enabled, porting that into this image could be the fix — happy to benchmark
it here if useful.

One question back: was your Kimi run also using MTP/speculative decode, or plain
decode? That would help confirm whether the delta is indexer-driven or MTP
interaction.

[yumenoshizuku]

One question back: was your Kimi run also using MTP/speculative decode, or plain decode? That would help confirm whether the delta is indexer-driven or MTP interaction.

plain decode, haven't got a chance to try any Kimi-K2.7-Code-DFlash yet

[repandv]

Ah, that's an important asymmetry — your Kimi run was plain decode, but the
GLM numbers (mine and presumably yours) are with MTP=3. So the 27 vs 50 t/s
isn't a clean model-vs-model comparison; part of it is the MTP path on GLM.

That actually cuts both ways: MTP should speed up decode (multiple tokens per
forward), but if the DSA indexer top-k is recomputed on every draft forward
(×3), MTP may be eating its own gain on GLM — which is exactly what
VLLM_MTP_INDEX_SHARE targets. So the delta is likely a mix of "indexer
recompute" and "MTP interaction", not one or the other.

To pin it down cleanly you'd want two points neither of us has yet:

1. GLM MTP=0 (plain) vs Kimi plain → like-for-like architecture delta
2. GLM MTP=3 vs GLM MTP=0 → whether MTP is a net win or net loss on GLM

Our box is in production so I can't pull it for a controlled MTP=0 run right
now, but the maintainers' v15 benchmark page does have MTP0/MTP3 columns that
should at least indicate the direction (different host topology, so treat as
relative). If you get a chance to run Kimi with MTP=3, that would be the other
half — a same-mode GLM-vs-Kimi comparison would tell us a lot.

[repandv]

Update — tested MTP=5 (GLM-5.2 natively supports up to 5 draft tokens) on our
box (same 8× RTX PRO 6000 Blackwell SE, TP=8, DCP=4). Result: MTP=5 is net
slower than MTP=3 for us, and the per-position data shows exactly why.

Per-position draft acceptance at depth=5 (median over ~40 single-stream windows,
real coding/reasoning prompts via OpenWebUI):

draft position	acceptance
1	~0.85
2	~0.60
3	~0.40
4	~0.25
5	~0.12

Mean acceptance length stayed ~3.3 (same range as MTP=3's 3.0–4.0), i.e. adding
draft positions 4–5 barely moved accepted length, while single-stream decode
dropped from ~34–40 t/s (MTP=3) to a ~19–27 t/s median (MTP=5). Positions 4–5
are almost always rejected on real (non-boilerplate) text, so we pay for two
extra MTP-head forwards per step — each recomputing the DSA indexer top-k — and
get almost nothing back.

This lines up with the earlier hypothesis: the cost scales with per-draft
indexer recompute, not with draft count itself. If the MTP draft steps could
reuse the verify-anchored indexer top-k (what VLLM_MTP_INDEX_SHARE does in the
jarrelscy fork, and what GLM-5.2's IndexShare is supposed to enable), deeper MTP
would be much cheaper and positions 4–5 might actually pay off. As-is on the
fathomless-firmament build, MTP=3 is the sweet spot and we've reverted to it.

Would porting the indexer-share path for MTP into this image be on the table?
Happy to re-run the MTP sweep against a build that has it.