glm52: serve MLA absorbed projections from the kv_b mxfp8 pack (qbmm)#157
glm52: serve MLA absorbed projections from the kv_b mxfp8 pack (qbmm)#157MadeBy561 wants to merge 1 commit into
Conversation
process_weights_after_loading notes there are no quantized bmm's for W_UV/W_UK_T -- and the quantized-bmm paths that do exist (aiter fp8/fp4) are ROCm-only. With sparkinfer's qbmm_absorb kernel there now is an sm120 one: VLLM_B12X_ABSORB_QBMM=1 (default off, registered in envs.py) serves both decode-path bmm's directly from kv_b_proj's mxfp8 pack and never materializes the BF16 pair, reclaiming the pair plus its load-time dequant scratch churn (~724 MiB/GPU measured at GLM-5.2 TP4 geometry) at identical decode numerics. The elide is geometry- and pack-checked with a warned fallback to the BF16 pair. Graph-visible batches (B <= 32) run the kernel; eager batches beyond the envelope and the DCP gather lane dequant the pair transiently via the e8m0-viewed expression (the view is load-bearing: a raw uint8.to(bf16) converts exponent bytes as integers). Load-finalize warmup precompiles every graph-visible batch size before capture, including the speculator's prefill-capture batch (num_tokens - num_reqs + 1); a compile miss during capture raises. Measured on GLM-5.2 NVFP4+NF3 hybrid (4x96GB sm120, TP4+DCP4, MTP3): max_model_len 700k -> 775,000, +724 MiB/GPU post-capture, decode 105.6-107.3 t/s vs 104.8-104.9 baseline, MTP3 acceptance windows to 92.1%, 157,932-token prefill 49 s, teacher-forced prefill KLD 0.1351+-0.0037 vs the standing 0.1356+-0.0054 cert (5/5 in band). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
|
Warning Review limit reached
Next review available in: 59 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Closing as superseded, not abandoned. The maintained implementation now lives directly in the canonical
The canonical implementation uses |
2 participants
VRAM: +724 MiB/GPU measured; max_model_len 700k → 775,000 on 4×96 GB.
process_weights_after_loadingnotes "we currently do not have quantizedbmm's which are needed for
W_UVandW_UK_T" — and the quantized-bmmpaths that do exist (aiter fp8/fp4) are ROCm-only. With the sparkinfer
qbmm_absorb kernel there now is an sm120 one:
VLLM_B12X_ABSORB_QBMM=1(default off, registered in envs.py) serves bothdecode-path bmm's directly from kv_b_proj's mxfp8 pack and never materializes
the BF16 pair — reclaiming the pair and its load-time dequant scratch churn
(~724 MiB/GPU measured at GLM-5.2 TP4 geometry) at identical decode numerics
(the kernel's dequant is bitwise the absorb expression; its fp64 error
envelope equals cuBLAS bf16 bmm's to 4 significant digits per cell).
Wiring:
qbmm_absorb_supported), fallsback to the BF16 pair with a warning when unsupported. (If the kv_b
pack-free from open glm52: free dead kv_b pack post-absorb + share draft rope tables (stacks on #145) #146 lands, it must stay skipped under qbmm — the
pack is the decode weight source.)
qbmm_absorb_ukt/_uvfor graph-visible batches (B ≤ 32);eager batches beyond the envelope (profiler dummy runs) and the DCP gather
lane dequant the pair transiently via the e8m0-viewed torch expression —
bitwise the same values the kernel produces.
warmup_qbmm_absorbprecompiles every graph-visible batchsize before capture — including the speculator's prefill-capture batch
(
num_tokens - num_reqs + 1); a compile miss during capture raises (agraph captured around a JIT replays corrupt output, and mid-capture cache
writes poison the persistent compile cache tree).
Measured (GLM-5.2 NVFP4+NF3 hybrid, 4×96 GB sm120, TP4+DCP4, fp8-rope nvfp4
KV + calibrated outer scales, MTP3, 2048-token batches):
🤖 Generated with Claude Code