[Quant] Add EXL3 Trellis backend for rank-sliced GLM/DeepSeek MoE#139
[Quant] Add EXL3 Trellis backend for rank-sliced GLM/DeepSeek MoE#139brandonmmusic-max wants to merge 15 commits into
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughEXL3 quantization support is added for dense and routed-expert layers, including metadata handling, checkpoint loading, CUDA execution, Trellis dispatch, and tests. Structured-output speculative advancement, DeepSeek attention gating, MLA flattening, and optional Docker cache installation are also updated. ChangesEXL3 quantization
Structured-output advancement
DeepSeek attention gating
Build installation control
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant ModelLoader
participant Exl3Config
participant Exl3MoEMethod
participant SparkinferTrellis
participant exllamav3_ext
ModelLoader->>Exl3Config: normalize rank-sliced checkpoint names
Exl3Config->>Exl3MoEMethod: configure EXL3 MoE tensors
Exl3MoEMethod->>SparkinferTrellis: plan rank-sliced dispatch
SparkinferTrellis->>exllamav3_ext: execute Trellis or parity path
exllamav3_ext-->>Exl3MoEMethod: return routed expert outputs
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@vllm/model_executor/layers/quantization/exl3.py`:
- Around line 464-472: The _moe_prefix_is_exl3 method currently hardcodes
projection names, so remapped MoE checkpoints are not detected. Update this
check to use the RoutedExperts layer’s ckpt_gate_proj_name, ckpt_up_proj_name,
and ckpt_down_proj_name values, matching the names consumed by
_validate_codebooks, and thread those names through the method as needed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 87dd2ded-af12-4ca3-992c-dad738c0d7be
📒 Files selected for processing (3)
vllm/model_executor/layers/fused_moe/routed_experts.pyvllm/model_executor/layers/quantization/__init__.pyvllm/model_executor/layers/quantization/exl3.py
3f575f5 to
60c82d9
Compare
brandonmmusic-max
changed the title
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@vllm/v1/worker/workspace.py`:
- Around line 36-43: Update the use_workspace_lane() docstring with Google-style
Args: documentation for lane and a Raises: section documenting the ValueError
for negative lanes, while preserving the existing behavior and explanatory text.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 1692d79f-0266-4848-898d-b40a56b3e55b
📒 Files selected for processing (9)
tests/v1/structured_output/test_reasoning_structured_output.pytests/v1/worker/test_workspace.pyvllm/models/deepseek_v32/nvidia/attention.pyvllm/v1/core/sched/scheduler.pyvllm/v1/structured_output/__init__.pyvllm/v1/worker/gpu/model_runner.pyvllm/v1/worker/gpu/warmup.pyvllm/v1/worker/gpu_worker.pyvllm/v1/worker/workspace.py
Builds on the rank-sliced EXL3 (ExLlamaV3 trellis) support added in local-inference-lab#139 by @brandonmusic, paired with the trellis3_t256 planned MoE from local-inference-lab/sparkinfer#49. No change to that work's weight loading, formats, or decode path; this commit only extends its batch dispatch. The vllm-project#139 backend serves every batch with m > 32 through the eager exllamav3_ext parity path: fp16-in/fp32-out staging (~25 GB extra HBM traffic per 4096-token step at GLM-5.2 geometry), python chunk loops, and one expert re-stream per 128-row chunk. That covers 100% of prefill, so prefill throughput is capped at ~2.0-2.3k tok/s while the same weights decode at full speed through the planned Trellis window. Build a second trellis_moe plan at max_tokens=max_num_batched_tokens with block_size_m=64 (env VLLM_EXL3_PREFILL_BLOCK_M, allowed {8,16,32,48,64}) beside the existing decode plan (32/8), and dispatch three ways in _apply_rank_sliced: * m in [min, max] -> decode plan (unchanged) * max < m <= capacity -> prefill plan (sparkinfer bind already accepts any tokens in [1, max_tokens]; zero kernel changes needed) * m < min -> parity path, whose persistent staging (xh/out32/token_sorted/weight_sorted) shrinks from capacity rows to one chunk while the prefill plan is live (~110 MiB/GPU returned) VLLM_EXL3_PREFILL_TRELLIS=0 restores the exact single-plan parity behavior (one-variable serving A/B lever). The parity branch now raises during CUDA graph capture instead of silently recording eager ext calls. The prefill arena (~1.05 GiB/GPU at capacity 3072) allocates during the profile pass, so vLLM's memory profiler sees it and the KV pool auto-shrinks instead of risking request-time OOM. Matched A/B serving brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw on the published verdictai/glm52-exl3-sparkinfer runtime image (4x RTX PRO 6000 Blackwell, TP4/DCP4/MTP3, identical boot geometry, only the kill-switch flipped, fresh JIT cache both arms; llm_decode_bench prefill-only, exact /tokenize, C1, 60 s/context): ctx parity trellis delta 8k 2,287 3,742 +63.6% 32k 2,218 3,557 +60.4% 64k 2,078 3,381 +62.7% 128k 1,920 3,032 +57.9% Decode C1 unchanged-to-better; 57k-token needle retrieval exact.
Optional companion to the prefill-plan commit; mocked plan/ext APIs, no GPU, sparkinfer, or exllamav3_ext required. Covers plan construction order, the three-way dispatch boundaries, chunk-capped parity staging, the VLLM_EXL3_PREFILL_TRELLIS=0 kill-switch, VLLM_EXL3_PREFILL_BLOCK_M override, elastic re-plan above scheduler capacity, and the parity-path capture guard. Follows the existing CPU-test pattern of tests/quantization/test_exl3.py from local-inference-lab#139.
Builds on the rank-sliced EXL3 (ExLlamaV3 trellis) support added in local-inference-lab#139 by @brandonmusic, paired with the trellis3_t256 planned MoE from local-inference-lab/sparkinfer#49. No change to that work's weight loading, formats, or decode path; this commit only extends its batch dispatch. The vllm-project#139 backend serves every batch with m > 32 through the eager exllamav3_ext parity path: fp16-in/fp32-out staging (~25 GB extra HBM traffic per 4096-token step at GLM-5.2 geometry), python chunk loops, and one expert re-stream per 128-row chunk. That covers 100% of prefill, so prefill throughput is capped at ~2.0-2.3k tok/s while the same weights decode at full speed through the planned Trellis window. Build a second trellis_moe plan at max_tokens=max_num_batched_tokens with block_size_m=64 (env VLLM_EXL3_PREFILL_BLOCK_M, allowed {8,16,32,48,64}) beside the existing decode plan (32/8), and dispatch three ways in _apply_rank_sliced: * m in [min, max] -> decode plan (unchanged) * max < m <= capacity -> prefill plan (sparkinfer bind already accepts any tokens in [1, max_tokens]; zero kernel changes needed) * m < min -> parity path, whose persistent staging (xh/out32/token_sorted/weight_sorted) shrinks from capacity rows to one chunk while the prefill plan is live (~110 MiB/GPU returned) VLLM_EXL3_PREFILL_TRELLIS=0 restores the exact single-plan parity behavior (one-variable serving A/B lever). The parity branch now raises during CUDA graph capture instead of silently recording eager ext calls. The prefill arena (~1.05 GiB/GPU at capacity 3072) allocates during the profile pass, so vLLM's memory profiler sees it and the KV pool auto-shrinks instead of risking request-time OOM. Matched A/B serving brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw on the published verdictai/glm52-exl3-sparkinfer runtime image (4x RTX PRO 6000 Blackwell, TP4/DCP4/MTP3, identical boot geometry, only the kill-switch flipped, fresh JIT cache both arms; llm_decode_bench prefill-only, exact /tokenize, C1, 60 s/context): ctx parity trellis delta 8k 2,287 3,742 +63.6% 32k 2,218 3,557 +60.4% 64k 2,078 3,381 +62.7% 128k 1,920 3,032 +57.9% Decode C1 unchanged-to-better; 57k-token needle retrieval exact. (cherry picked from commit d57d775)
2fdc6d3 to
31ff5a2
Compare
Optional companion to the prefill-plan commit; mocked plan/ext APIs, no GPU, sparkinfer, or exllamav3_ext required. Covers plan construction order, the three-way dispatch boundaries, chunk-capped parity staging, the VLLM_EXL3_PREFILL_TRELLIS=0 kill-switch, VLLM_EXL3_PREFILL_BLOCK_M override, elastic re-plan above scheduler capacity, and the parity-path capture guard. Follows the existing CPU-test pattern of tests/quantization/test_exl3.py from local-inference-lab#139. (cherry picked from commit f80a56d)
Rebased onto Gilded Gnosis v20 canonical (
|
| Test | v20 | Prior base |
|---|---|---|
| Code generation (Python/HTML, temp 0) | 94–100% valid — reported ~50% syntax-error symptom eliminated | ~50% (reported) |
| LAVD r10/c5 | 10/10 (6 exact, 4 near, 0 fail) | 8/10 (2 fail) |
The LAVD fail count 2 → 0 reflects the v20 canonical MTP fixes plus the flattening fix. Tested image: verdictai/glm52-exl3-sparkinfer:v20-gg6722c1d-si1a88b38-cu132-sm120a.
Re #163: the dual-plan Trellis prefill from #163 is incorporated in this rebase, so #163 can be closed or rebased onto v20 as preferred.
Full results on current base for a full validation sweet a/b for kv cache, set on reasoning high for the PR’s serve.sh, docker compose |Test |nvfp4_ds_mla |fp8 |
|---------------------------|--------------------|--------------------|
|Estonia (133K needle) |PASS 30 / FAIL 0|PASS 30 / FAIL 0|
|LAVD (ledger) |18E/11N/1F — 97%|15E/13N/2F — 93% |
|Hotel-lights (low tier)|15E/15F — 50% |18E/12F — 60% |
|KLD vs BF16 |0.116138 |0.101198 |
|Decode C1 / C8 |87.5 / 308.1 |86.6 / 312.2 |
|Prefill 8k / 128k |2,551 / 1,833 |2,660 / 1,771 |
Full server.sh, docker compose yaml and image and full results. Al run on four in pro 6ks, capped at 300 watts each, so on full workstation or higher power, decode and prefill would be faster. here https://huggingface.co/brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw/blob/main/RELEASE_TEST_SUITE.md
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@vllm/v1/attention/backends/mla/indexer.py`:
- Around line 399-402: Update the B12X activation condition in the relevant
build path around the sparse-indexer capability check to use the canonical
backend-aware predicate, including attention_config.backend == "B12X_MLA_SPARSE"
when the environment variable is unset. Preserve the existing SM120 and next_n
behavior while ensuring the native (B, next_n) path remains disabled for all
supported B12X configurations.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: e1796822-6dd0-4a09-a332-cead2920cc76
📒 Files selected for processing (14)
docker/Dockerfiletests/quantization/test_exl3.pytests/quantization/test_exl3_prefill_plan.pytests/v1/structured_output/test_reasoning_structured_output.pyvllm/config/model.pyvllm/model_executor/layers/fused_moe/routed_experts.pyvllm/model_executor/layers/quantization/__init__.pyvllm/model_executor/layers/quantization/exl3.pyvllm/model_executor/models/deepseek_v2.pyvllm/model_executor/models/glm4_moe.pyvllm/models/deepseek_v32/nvidia/attention.pyvllm/v1/attention/backends/mla/indexer.pyvllm/v1/core/sched/scheduler.pyvllm/v1/structured_output/__init__.py
🚧 Files skipped from review as they are similar to previous changes (5)
- vllm/v1/core/sched/scheduler.py
- vllm/model_executor/layers/fused_moe/routed_experts.py
- vllm/model_executor/layers/quantization/init.py
- vllm/models/deepseek_v32/nvidia/attention.py
- vllm/v1/structured_output/init.py
|
Credit goes to davidsyoung for prefill improvement |
|
Note on the incorporated prefill work (cross-ref #163): As of the v20 rebase, this branch carries the dual-plan Trellis prefill dispatch verbatim from #163:
so reviewing/merging this PR covers that change completely. #163 stays open only as the isolated review scope against the pre-rebase base ( Validation data for the incorporated change, measured on the pre-v20 stack (pinned digest
The public test image I published earlier ( |
31ff5a2 to
6a1a9f9
Compare
|
@coderabbitai review Fixes are in — the flattening gate now uses the canonical backend-aware |
🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Target module outline =="
ast-grep outline vllm/v1/attention/backends/mla/indexer.py --items all --type class,function --view expanded || true
echo
echo "== Targeted flattening gate and imports =="
rg -n -C 12 'use_b12x_sparse_indexer|use_flattening|B12X_MLA_SPARSE|VLLM_USE_B12X_SPARSE_INDEXER' \
vllm/v1/attention/backends/mla/indexer.py
echo
echo "== Predicate definition and call sites =="
rg -n -C 8 'def use_b12x_sparse_indexer\b|use_b12x_sparse_indexer\s*\(' vllm --glob '*.py'
echo
echo "== Current PR diff for the relevant module =="
git diff --unified=40 'HEAD^' -- vllm/v1/attention/backends/mla/indexer.py
echo
echo "== Reported commit diff (when present in local history) =="
git show --format=fuller --stat --oneline 6a1a9f9b3a -- vllm/v1/attention/backends/mla/indexer.py || true
git show --format= --unified=40 6a1a9f9b3a -- vllm/v1/attention/backends/mla/indexer.py || trueLength of output: 43841 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== B12X request-detection implementation =="
sed -n '600,665p' vllm/model_executor/layers/sparse_attn_indexer.py
echo
echo "== Backend configuration symbols and B12X selection references =="
rg -n -C 6 'B12X_MLA_SPARSE|_b12x_sparse_indexer_requested|attention_backend' \
vllm/model_executor/layers/sparse_attn_indexer.py \
vllm/v1/attention/backends/mla/indexer.py \
vllm --glob '*.py'
echo
echo "== Direct env-gated B12X branches in the metadata builder =="
sed -n '600,665p' vllm/v1/attention/backends/mla/indexer.py
sed -n '750,805p' vllm/v1/attention/backends/mla/indexer.py
sed -n '910,955p' vllm/v1/attention/backends/mla/indexer.py
sed -n '1080,1145p' vllm/v1/attention/backends/mla/indexer.py
echo
echo "== Existing focused tests for B12X flattening/backend selection =="
rg -n -C 5 'use_flattening|use_b12x_sparse_indexer|B12X_MLA_SPARSE|next_n' \
tests --glob '*.py' | head -n 300Length of output: 142171
Thus, with If you found this review helpful, would you consider giving us a shout-out on X? Thank you for using CodeRabbit! ✅ Action performedReview finished.
|
Validation — rank-sliced tr3 MTP layer now loads & servesThe two fixes in 2cb2e08 let the MTP draft head load a rank-sliced EXL3 ( Image: Runtime note: a tr3 MTP layer also needs LoadClean load, MTP-3 active. Before the fixes: Throughput (gpu-util 0.96, auto-profiled KV)
Accuracy (unchanged — MTP is lossless)
|
Adds --quantization exl3: any exllamav3 EXL3 checkpoint loads through a QuantizationConfig + LinearMethodBase pair that reproduces turboderp's dense no_reconstruct forward argument-for-argument at the exl3_gemm boundary (input suh sign-flip + Hadamard-128, direct trellis GEMM, output svh + Hadamard-128), with mcg/mul1 sentinel semantics, legacy su/sv expansion, reference pad/trim, and tp_import-discipline TP slicing (KV replication; Qwen3.5 [(0,1,2),3] tuple shards). Registry delta is exactly three lines (Literal member, lazy import, mapping entry). RoutedExperts.load_weights: qualify is_fused by the matched mapping entry, not tensor rank alone — per-expert EXL3 trellis tensors are rank-3 [K/16, N/16, 16*bpw] and were mangled by the fused transpose/chunk path (fail-loud downstream, but wrong). Fused entries are exactly those whose weight_name carries no per-expert index; behavior for genuinely fused checkpoints is unchanged. Backend enforces eager execution at build time: exl3_gemm autotunes with timing launches on first call per (m-bucket, k, n, K) hash, which is incompatible with CUDA-graph capture, and m-bucketing defeats warmup coverage. Quantized lm_head with added vocabulary raises (TP slicing would silently misalign). MoE expert loader is marked supports_moe_loading for llama4-style loader paths. GPU validation gates (owner-run): smoke-load Qwen3-0.6B-exl3, generate on Qwen3.6-27B-exl3_3.30bpw, top-logprob comparison vs exllamav3 pinned to no_reconstruct=True. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Modules carrying the MCG codebook marker are prepared once at weight-processing time via b12x prepare_trellis256_dense_weight (zero-copy, bits inferred from native storage) and applied through run_trellis256_dense — the fused CuTeDSL kernel whose output is byte-identical across batch size m. Legacy default-codebook and MUL1 shards, and any shard the fused prepare rejects, fall back per-shard to the bit-faithful exllamav3_ext parity op, which remains the oracle. VLLM_EXL3_FUSED=0 forces the parity path (A/B switch); VLLM_EXL3_LOG_PATH_SELECTION=1 logs the per-shard selection. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A multimodal wrapper offers the quantized head as language_model.lm_head while the checkpoint storage map keys it as top-level lm_head. The segment-collapse loop only considered interior segments, so the head silently fell back to UnquantizedLinearMethod and loading the checkpoint's lm_head.mcg then failed. Collapse leading model/language_model segments too; exact-prefix candidates still take precedence. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The fused dense entry performs its outer rotations through exllamav3_ext.had_r_128, imported by module name inside b12x. Load the extension through the parity path's VLLM_EXL3_ABI_SHIM/VLLM_EXL3_EXT_PATH contract first, so an all-fused model does not import an unshimmed site-packages copy (undefined at::cuda::getCurrentCUDABlasHandle on torch 2.12). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review follow-up (CodeRabbit): _moe_prefix_is_exl3 hardcoded gate_proj/up_proj/down_proj while _validate_codebooks keys off the layer's ckpt_*_proj_name fields, so a remapped-projection MoE checkpoint would pass validation logic but silently miss EXL3 detection. Thread the RoutedExperts layer through and read its names, defaulting to the standard trio for layer variants that do not carry them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Brandon Music <brandon.m.music@gmail.com>
…MTP crash Port two upstream vLLM fixes so tool calling works under MTP speculative decoding: - structured_output: advance the grammar from the authoritative new_token_ids step delta instead of the counter-derived window, so the reasoning-end (</think>) marker is detected under async scheduling + spec decode and the tool-call grammar engages instead of emitting unconstrained output. (vllm-project#48516, supersedes vllm-project#44993.) - deepseek_v32 DSA: derive has_indexer from index_k as well, so MTP draft steps 1+ (skip_topk under index_share_for_mtp_iteration) no longer crash fused_norm_rope's index_k assertion. (vllm-project#48528.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Builds on the rank-sliced EXL3 (ExLlamaV3 trellis) support added in local-inference-lab#139 by @brandonmusic, paired with the trellis3_t256 planned MoE from local-inference-lab/sparkinfer#49. No change to that work's weight loading, formats, or decode path; this commit only extends its batch dispatch. The vllm-project#139 backend serves every batch with m > 32 through the eager exllamav3_ext parity path: fp16-in/fp32-out staging (~25 GB extra HBM traffic per 4096-token step at GLM-5.2 geometry), python chunk loops, and one expert re-stream per 128-row chunk. That covers 100% of prefill, so prefill throughput is capped at ~2.0-2.3k tok/s while the same weights decode at full speed through the planned Trellis window. Build a second trellis_moe plan at max_tokens=max_num_batched_tokens with block_size_m=64 (env VLLM_EXL3_PREFILL_BLOCK_M, allowed {8,16,32,48,64}) beside the existing decode plan (32/8), and dispatch three ways in _apply_rank_sliced: * m in [min, max] -> decode plan (unchanged) * max < m <= capacity -> prefill plan (sparkinfer bind already accepts any tokens in [1, max_tokens]; zero kernel changes needed) * m < min -> parity path, whose persistent staging (xh/out32/token_sorted/weight_sorted) shrinks from capacity rows to one chunk while the prefill plan is live (~110 MiB/GPU returned) VLLM_EXL3_PREFILL_TRELLIS=0 restores the exact single-plan parity behavior (one-variable serving A/B lever). The parity branch now raises during CUDA graph capture instead of silently recording eager ext calls. The prefill arena (~1.05 GiB/GPU at capacity 3072) allocates during the profile pass, so vLLM's memory profiler sees it and the KV pool auto-shrinks instead of risking request-time OOM. Matched A/B serving brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw on the published verdictai/glm52-exl3-sparkinfer runtime image (4x RTX PRO 6000 Blackwell, TP4/DCP4/MTP3, identical boot geometry, only the kill-switch flipped, fresh JIT cache both arms; llm_decode_bench prefill-only, exact /tokenize, C1, 60 s/context): ctx parity trellis delta 8k 2,287 3,742 +63.6% 32k 2,218 3,557 +60.4% 64k 2,078 3,381 +62.7% 128k 1,920 3,032 +57.9% Decode C1 unchanged-to-better; 57k-token needle retrieval exact. (cherry picked from commit d57d775)
Optional companion to the prefill-plan commit; mocked plan/ext APIs, no GPU, sparkinfer, or exllamav3_ext required. Covers plan construction order, the three-way dispatch boundaries, chunk-capped parity staging, the VLLM_EXL3_PREFILL_TRELLIS=0 kill-switch, VLLM_EXL3_PREFILL_BLOCK_M override, elastic re-plan above scheduler capacity, and the parity-path capture guard. Follows the existing CPU-test pattern of tests/quantization/test_exl3.py from local-inference-lab#139. (cherry picked from commit f80a56d)
MTP next_n>2 was force-flattened onto the DeepGEMM next_n<=2 fallback, which corrupts B12X sparse-indexer code generation. The B12X/sparkinfer indexer supports native (B, next_n) on SM120, so keep it on the native path (FULL cudagraphs preserved). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The MTP draft head (deepseek_mtp) could not load a rank-sliced EXL3
(hybrid_tr3_tail) MoE layer -- e.g. a tr3-converted GLM-5.2 MTP layer 78 --
for two reasons:
1. get_draft_quant_config() built the draft Exl3Config via
get_quantization_config() but never called maybe_update_config(), which is
what hydrates rank_sliced_metadata from hybrid_tr3_tail (the target model
receives this call in config/vllm.py). Without it the draft config's
rank_sliced_metadata stayed None and the MTP experts fell back to the stock
FusedMoE path.
2. DeepSeekMTP.load_weights() did not normalize rank-sliced weight names, so a
checkpoint tensor `...experts.0.down_proj.rank0.mcg` mapped to
`...routed_experts.w2_rank0.mcg` while Exl3MoEMethod registers the fused
`w2_mcg` param -> KeyError. The target model (deepseek_v2.load_weights)
already strips `.rank{r}` via normalize_rank_sliced_weight_name; mirror it.
With both fixes a tr3 MTP layer loads and serves. (Running a tr3 MTP layer also
needs VLLM_EXL3_TRELLIS_MIN_M=1 so the MTP-N draft's small-m GEMMs stay inside
the Trellis cudagraph window -- a runtime env, not a code change.)
Validated on GLM-5.2-EXL3-TR3-3.0bpw with an EXL3-TR3 MTP layer-78 head (TP4,
DCP4, MTP-3): loads clean, Estonia 10/10 PASS, LAVD EXACT 5 / NEAR 5 / FAIL 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QsVTetPQQc5gP7LveJCWRE
2cb2e08 to
dc0286f
Compare
|
@coderabbitai review Correcting my earlier wording on this comment, which understated the change and could have led to a shallower review. This branch was rebased onto
So please review the rebased state on its merits rather than assuming a no-op rebase — particularly the indexer auto-merge and any interaction between the EXL3 Trellis path and the newer DCP/prefill work. |
|
✅ Action performedReview finished.
|
Rebased for the finalized Gilded Gnosis v20 baseForce-pushed Diff size is unchanged (16 files, +2452/-41) and all 14 commits replayed without conflicts — but this is not a no-op rebase: the branch now sits on 20 newer upstream commits. In particular Also validated against the finalized v20 common basePer the v20 release note, EXL3 is enabled by rebasing the EXL3 source layer onto the pinned stack (the base ships no EXL3/Trellis loader), not by pointing the base tag at an EXL3 checkpoint. So in addition to the branch above, the EXL3 layer was rebased onto the v20-final pins and built:
Of the 12 runtime overlay files, only Validated image: Runtime validation (4x RTX PRO 6000 Blackwell SM120a, TP4/DCP4, MTP-3, NVFP4 DS-MLA KV)
Two notes for the runtime side:
|
|
One CI note: This is the repo's label gate rather than anything in the diff — it was already failing on the previous head ( Also noting CodeRabbit reports |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== PR/base topology and changed files =="
git status --short
git branch --show-current
git merge-base HEAD origin/dev/gilded-gnosis || true
git diff --stat "$(git merge-base HEAD origin/dev/gilded-gnosis)" HEAD
git diff --name-status "$(git merge-base HEAD origin/dev/gilded-gnosis)" HEAD
echo
echo "== Target module structure =="
ast-grep outline vllm/v1/attention/backends/mla/indexer.py --items all --type class,function --view expanded
ast-grep outline vllm/model_executor/layers/quantization/exl3.py --items all --type class,function --view expanded
for f in mla_attention.py b12x_mla_sparse.py b12x_attn.py; do
fd -a -t f "^${f}$" vllm -x ast-grep outline {} --items all --type class,function --view expanded \;
done
echo
echo "== PR diff: indexer auto-merge target =="
base="$(git merge-base HEAD origin/dev/gilded-gnosis)"
git diff --unified=60 "$base" HEAD -- vllm/v1/attention/backends/mla/indexer.py
echo
echo "== Current flattening/alignment context and B12X predicate =="
rg -n -C 18 '_align_block_table_width|use_flattening|use_b12x_sparse_indexer|next_n' \
vllm/v1/attention/backends/mla/indexer.py \
vllm/model_executor/layers/sparse_attn_indexer.py
echo
echo "== EXL3 rank-sliced planning, capture, DCP/prefill, and workspace references =="
rg -n -C 10 'rank_sliced|_apply_rank_sliced|prefill|plan|capture|workspace|dcp|DCP|block_m|Trellis' \
vllm/model_executor/layers/quantization/exl3.py \
vllm/model_executor/models/deepseek_v2.py \
vllm/model_executor/models/glm4_moe.py \
vllm/v1/attention/backends/mla/indexer.py \
vllm/v1/attention/backends/mla \
--glob '*.py'
echo
echo "== Focused test coverage and current diff for relevant tests =="
rg -n -C 8 'align_block_table|use_flattening|use_b12x_sparse_indexer|rank_sliced|prefill.*trellis|Trellis|DCP' \
tests --glob '*.py'
git diff --unified=30 "$base" HEAD -- \
tests/quantization/test_exl3.py \
tests/quantization/test_exl3_prefill_plan.py \
vllm/model_executor/layers/quantization/exl3.pyLength of output: 2675 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Available repository refs and remotes =="
git remote -v
git branch -a
git show -s --format='HEAD %H%nparents %P%nsubject %s' HEAD
git for-each-ref --format='%(refname:short) %(objectname) %(subject)' \
refs/heads refs/remotes | head -n 200
echo
echo "== Pull request metadata =="
gh pr view 139 --repo local-inference-lab/vllm \
--json number,headRefName,headRefOid,baseRefName,baseRefOid,mergeStateStatus \
|| true
echo
echo "== Resolve a usable comparison base and inspect rebase-sensitive paths =="
base=""
for candidate in \
"origin/dev/gilded-gnosis" \
"dev/gilded-gnosis" \
"origin/89b4a98d" \
"89b4a98d"; do
if git rev-parse --verify --quiet "${candidate}^{commit}" >/dev/null; then
base="$(git merge-base HEAD "$candidate")"
echo "Resolved candidate: $candidate -> merge base: $base"
break
fi
done
if [ -z "$base" ]; then
echo "No local base ref was found; showing HEAD source context without a diff."
else
echo "== Changed files and relevant PR diffs =="
git diff --stat "$base" HEAD
git diff --name-status "$base" HEAD
git diff --unified=60 "$base" HEAD -- \
vllm/v1/attention/backends/mla/indexer.py \
vllm/model_executor/layers/quantization/exl3.py \
tests/quantization/test_exl3.py \
tests/quantization/test_exl3_prefill_plan.py
fi
echo
echo "== Module maps =="
ast-grep outline vllm/v1/attention/backends/mla/indexer.py --items all --type class,function --view expanded
ast-grep outline vllm/model_executor/layers/quantization/exl3.py --items all --type class,function --view expanded
echo
echo "== Current indexer alignment/flattening and B12X detection =="
rg -n -C 20 '_align_block_table_width|use_flattening|use_b12x_sparse_indexer|next_n' \
vllm/v1/attention/backends/mla/indexer.py \
vllm/model_executor/layers/sparse_attn_indexer.py
echo
echo "== EXL3 planned dispatch, capture safety, DCP and workspace interactions =="
rg -n -C 12 'def _apply_rank_sliced|def process_weights_after_loading|def apply\(|prefill|capture|workspace|DCP|dcp|plan|block_m|trellis' \
vllm/model_executor/layers/quantization/exl3.py \
vllm/v1/attention/backends/mla \
--glob '*.py'
echo
echo "== Relevant test definitions and assertions =="
rg -n -C 8 'def test_.*(flatten|align|b12x|rank_sliced|prefill|capture)|use_flattening|_align_block_table_width|_apply_rank_sliced' \
tests --glob '*.py'Length of output: 202941 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== GitHub PR base/head and changed-file list =="
gh api repos/local-inference-lab/vllm/pulls/139 \
--jq '{base: (.base.ref + "@" + .base.sha), head: (.head.ref + "@" + .head.sha), mergeable: .mergeable, changed_files: .changed_files}'
gh api --paginate repos/local-inference-lab/vllm/pulls/139/files \
--jq '.[] | [.filename, .status, .additions, .deletions] | `@tsv`'
echo
echo "== Local commits and all available parent diff summaries =="
git show -s --format='HEAD=%H%nparents=%P%nsubject=%s' HEAD
git log --oneline -n 20
git diff --stat HEAD^ HEAD
echo
echo "== Current indexer: imports, alignment helper, and flattening gate =="
rg -n '^(def _align_block_table_width|class DeepseekV32IndexerMetadataBuilder| def |from .*sparse_attn_indexer import|import .*sparse_attn_indexer)' \
vllm/v1/attention/backends/mla/indexer.py
sed -n '1,110p' vllm/v1/attention/backends/mla/indexer.py
sed -n '600,700p' vllm/v1/attention/backends/mla/indexer.py
sed -n '730,830p' vllm/v1/attention/backends/mla/indexer.py
sed -n '1080,1165p' vllm/v1/attention/backends/mla/indexer.py
echo
echo "== Current B12X request predicate =="
rg -n -C 15 'def use_b12x_sparse_indexer|def _b12x_sparse_indexer_requested|B12X_MLA_SPARSE' \
vllm/model_executor/layers/sparse_attn_indexer.py
echo
echo "== EXL3 rank-sliced method boundaries =="
rg -n '^\s*def (_apply_rank_sliced|_ensure_rank_sliced|_plan|_get|apply|process_weights_after_loading)|^\s*class Exl3MoEMethod' \
vllm/model_executor/layers/quantization/exl3.py
rg -n -C 14 'def _apply_rank_sliced|is_current_stream_capturing|capture|prefill_plan|decode_plan|workspace|bind\(' \
vllm/model_executor/layers/quantization/exl3.py
echo
echo "== Exact PR patches from GitHub for the relevant files =="
for path in \
vllm/v1/attention/backends/mla/indexer.py \
vllm/model_executor/layers/quantization/exl3.py \
tests/quantization/test_exl3_prefill_plan.py \
tests/quantization/test_exl3.py; do
echo "--- $path ---"
gh api "repos/local-inference-lab/vllm/pulls/139/files?per_page=100" \
--jq ".[] | select(.filename == \"$path\") | .patch" \
| sed -n '1,260p'
doneLength of output: 59085 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Indexer constructor: final auto-merged context =="
sed -n '337,450p' vllm/v1/attention/backends/mla/indexer.py
echo
echo "== Indexer alignment use sites and focused regression test =="
rg -n -C 10 '_align_block_table_width' \
vllm/v1/attention/backends/mla/indexer.py \
tests/model_executor/layers/test_sparse_attn_indexer_b12x.py
sed -n '1088,1165p' tests/model_executor/layers/test_sparse_attn_indexer_b12x.py
echo
echo "== EXL3 runtime cache key, plan allocation, and all binding paths =="
sed -n '1325,1575p' vllm/model_executor/layers/quantization/exl3.py
sed -n '1570,1708p' vllm/model_executor/layers/quantization/exl3.py
rg -n -C 6 '_RANK_SLICED_RUNTIMES|exl3_trellis_weights|exl3_pointer_tables|exl3_expert_map' \
vllm/model_executor/layers/quantization/exl3.py
echo
echo "== Final MTP-layer loading patch and supporting weight-name/model loading paths =="
git diff --unified=50 HEAD^ HEAD -- \
vllm/model_executor/models/deepseek_mtp.py \
vllm/model_executor/models/utils.py
rg -n -C 12 'normalize_rank_sliced_weight_name|mtp_block|load_weights' \
vllm/model_executor/models/deepseek_mtp.py \
vllm/model_executor/models/deepseek_v2.py \
vllm/model_executor/models/glm4_moe.py \
vllm/model_executor/models/utils.py
echo
echo "== Test collection references for target/draft resource separation and EXL3 MTP loading =="
rg -n -C 5 'target.*draft|draft.*target|rank.sliced|hybrid_tr3_tail|mtp_block|_RANK_SLICED_RUNTIMES' \
tests/quantization/test_exl3.py \
tests/quantization/test_exl3_prefill_plan.py \
tests/model_executor \
--glob '*.py' | head -n 600Length of output: 96012 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Line-numbered EXL3 runtime key and mutable buffers =="
nl -ba vllm/model_executor/layers/quantization/exl3.py | sed -n '1355,1665p'
echo
echo "== All cache lifecycle/ownership references =="
rg -n -C 10 '_RANK_SLICED_RUNTIMES|rank_sliced_runtime|trellis_scratch|prefill_scratch|token_sorted|expert_count' \
vllm tests --glob '*.py'
echo
echo "== Existing model/draft execution and graph resource isolation hooks =="
rg -n -C 8 'draft.*(workspace|scratch|graph|capture)|target.*(workspace|scratch|graph|capture)|(workspace|scratch|graph|capture).*draft' \
vllm tests --glob '*.py' | head -n 800Length of output: 1042 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Line-numbered EXL3 runtime key and mutable-buffer use =="
awk 'NR >= 1355 && NR <= 1665 { printf "%6d %s\n", NR, $0 }' \
vllm/model_executor/layers/quantization/exl3.py
echo
echo "== Cache lifecycle and ownership tests (bounded) =="
rg -n -C 8 '_RANK_SLICED_RUNTIMES|rank_sliced_runtime|trellis_scratch|prefill_scratch|token_sorted|expert_count' \
vllm/model_executor/layers/quantization/exl3.py \
tests/quantization \
tests/model_executor \
--glob '*.py' | head -n 500
echo
echo "== Target/draft graph-resource isolation mechanisms relevant to this runtime =="
rg -n -C 6 'draft.*(workspace|scratch|graph|capture)|target.*(workspace|scratch|graph|capture)|(workspace|scratch|graph|capture).*draft' \
vllm/v1 \
vllm/model_executor \
tests/model_executor \
tests/distributed \
--glob '*.py' | head -n 500Length of output: 82057
The I found one issue in the EXL3 runtime integration:
I did not run CUDA tests in this sandbox. |
The rank-sliced runtime cache was keyed only on device, dtype, shape, topk and planner settings. The cached value owns mutable Trellis/prefill scratch plus parity staging and sort buffers, so any two layers that hash equal share those allocations. A target MoE layer and a rank-sliced MTP draft layer collide on every component of that key: same hidden/intermediate size, same local expert count, same topk, identical planner env, and both resolve max_num_batched_tokens from the same scheduler config. The draft therefore reused the target's scratch, defeating the target/draft resource isolation that their independently captured CUDA graphs depend on and coupling graph lifetime across the two models. Key on the owning quant config instead. The draft is built with its own Exl3Config while every layer of one model shares a single config, so each model gets exactly one runtime. This is deliberately coarser than per-layer ownership: the prefill arena alone is ~1 GiB, so a per-layer cache would cost tens of GiB per rank on a 75+ layer model. Adds CPU regression tests asserting that two same-shape layers owned by different models produce distinct, stable scopes and cache keys that differ only by that scope.
|
@coderabbitai Confirmed and fixed in I verified the collision rather than assuming it: for GLM-5.2 the MTP draft layer is architecturally a copy of a decoder layer, so every component of the old key matches between a target MoE layer and the draft — same One correction to the suggested remedy. Making the runtime layer-owned would break the model. The cached value holds the prefill arena, which is ~1054 MiB on our production config (the runtime logs So the fix scopes the runtime to its owning model rather than to each layer:
Regression tests added in
Both pass locally. I have not been able to run the CUDA suite against this head yet — Also confirming your read on the |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Commit availability and affected paths =="
git show --no-ext-diff --format='commit %H%n%s' --name-status 26c4bfd -- \
vllm/model_executor/layers/quantization/exl3.py \
vllm/model_executor/models/utils.py \
tests/quantization/test_exl3.py
echo
echo "== EXL3 cache scope/key definitions and cache access =="
rg -n -C 14 \
'def _runtime_scope_id\b|_RANK_SLICED_RUNTIMES|runtime_key|rank_sliced.*key|Exl3Config' \
vllm/model_executor/layers/quantization/exl3.py
echo
echo "== Draft quantization-config construction and propagation =="
rg -n -C 14 \
'Exl3Config|quant_config|draft.*quant|draft_quant|maybe_update_config' \
vllm/model_executor/models/utils.py
echo
echo "== Claimed CPU-only regression tests =="
rg -n -C 18 \
'def test_rank_sliced_runtime_scope_is_per_owning_model\b|def test_rank_sliced_runtime_key_differs_across_models_with_same_shape\b|_runtime_scope_id|_rank_sliced.*key|_RANK_SLICED_RUNTIMES' \
tests/quantization/test_exl3.py
echo
echo "== Exact fix patch =="
git show --no-ext-diff --format= --unified=55 26c4bfd -- \
vllm/model_executor/layers/quantization/exl3.py \
vllm/model_executor/models/utils.py \
tests/quantization/test_exl3.pyLength of output: 34815
The new CPU tests correctly cover distinct/stable owner scopes and prove that two otherwise identical keys differ only by scope, while same-owner keys remain equal. I did not run CUDA tests in this sandbox. ✏️ Learnings added
|
Runtime bumped to v24 (target/draft scratch isolation)The validated runtime referenced in this PR is now: It supersedes the earlier v23 image, which predates the rank-sliced runtime scoping fix. The pins in the description are updated accordingly. Why: the EXL3 rank-sliced runtime cache was keyed only on device/dtype/shape/topk/planner settings, and the cached entry owns mutable Trellis/prefill scratch. The MTP-78 draft layer matches a target MoE layer on every one of those components, so it was reusing the target's scratch. The key is now scoped to the owning quant config, giving each model its own runtime. Measured effect (the fix is visible in memory, not in the log -- the planner line uses
The 119,552-token KV reduction corresponds to ~1056 MiB against a 1054.2 MiB arena, which is direct evidence that a second runtime is allocated. The reduced KV capacity is the intended cost of the isolation, and it is also why the fix is scoped per model rather than per layer -- per-layer ownership would have needed tens of GiB per rank across 75+ layers. Quality on v24: Estonia c2 x5 = 5/5 pass. LAVD c5 across two independent 5-run samples = 4 EXACT / 5 NEAR / 1 FAIL over 10 runs (sample 1: 2E/3N/1F, sample 2: 2E/3N/0F, matching the pre-fix image exactly). The single miss was a wrong ledger total at 7,114 completion tokens against a 24,576 cap, so answer quality rather than truncation, and within this profile's historical spread. Reporting the aggregate including the failure rather than only the clean sample. |
Runtime bumped to v26 — supersedes the v24 note aboveThe description pins have been updated to this image. v24 and the intermediate v25 are superseded — please disregard both. What changed since v24. v24 carried the per-model rank-sliced runtime scoping (target/draft scratch isolation). v25 additionally carried a change to the compile-cache architecture key that CodeRabbit correctly identified as a no-op: v26 carries the real fix:
This is not academic on the validation rig: its four boards report two different device names (Max-Q and non-Max-Q) while sharing compute capability 12.0. Validation on v26 (4x RTX PRO 6000 SM120a, TP4/DCP4, MTP-3, NVFP4 DS-MLA KV):
GPU KV cache 998,656 tokens (1.90x at 524,288). That is ~1 GiB below the pre-isolation 1,115,904 (2.13x), which is the intended cost of giving the draft model its own Trellis/prefill arena. Unit tests: CI status unchanged: |
2 participants
Summary
Adds native EXL3 (exllamav3 Trellis) loading and execution to Gilded Gnosis
vLLM. The backend serves native rank-sliced checkpoints without reconstructing
or requantizing routed-expert weights at load time.
The implementation provides:
--quantization exl3detection and tensor-storage parsingRelease validation also exposed target/draft lifetime hazards in DCP speculative
execution. This branch now includes:
The Sparkinfer kernel/API dependency is
local-inference-lab/b12x#49.
Rebase and release pins
dev/gilded-gnosisat89b4a98d1ffebb2dda1e1ac5e55238e3a9cfbd5826c4bfdd3ff2be0433e6fe07e0c3be535f5bb318d4438d490691f79022fdfc8149e1c5f161d15445The exact validated image is:
What this rebase actually changed (please review on its merits)
This branch was force-pushed after a rebase onto
dev/gilded-gnosis@89b4a98d(previous head
2cb2e085->dc0286f). The PR's own diff is the same size(16 files, +2452/-41) and all 14 commits replayed without conflicts, but this is
not a no-op rebase -- the branch now sits on 20 newer upstream commits, so
the code these edits apply to has moved. Concretely:
vllm/v1/attention/backends/mla/indexer.pywas modified by both sides.Upstream changed it +31/-2 in that span (including the new
_align_block_table_width), and this PR'suse_flatteningpredicate change(
and not use_b12x_sparse_indexer()) was auto-merged onto that newer file bygit rather than re-authored. Git reported no conflict because the two hunks are
~20 lines apart, but the semantics of the merged result were not human-reviewed --
this is the single highest-value thing to look at in this rebase.
mla_attention.py,b12x_mla_sparse.py,b12x_attn.pyandthe DCP workspace tests in the same span. Those are not in this PR's diff, but the
EXL3 Trellis MoE and indexer paths interact with them at runtime, so behavioural
review of that boundary is welcome.
unchanged.
For traceability: the EXL3 delta carried by this branch is byte-identical to the
tree the validated v23 image was built from (verified per file against each base --
10 changed lines in
deepseek_v2.py, 20 inindexer.py), so the validation resultsbelow correspond to exactly this contribution.
Finalized Gilded Gnosis v20 common base (2026-07-25)
This branch is rebased onto its own base branch for review and merge. It has also
been built and validated as an EXL3 source layer on the finalized v20 common
base. Per the v20 release note the base packages no EXL3/Trellis loader or
kernels, so EXL3 is enabled by rebasing the EXL3 layer onto the pinned stack --
not by pointing the
voipmonitor/vllmtag at an EXL3 checkpoint.voipmonitor/vllm:gilded-gnosis-v20-vllm5517197-sibe0edca-fi801d57a-cu132-20260725551719766029e78824a30d97ae6ac63917405b5fbe0edcaae6f5d284bb29a82325aba7a0ead6960fmodels/deepseek_v2.pyandv1/attention/backends/mla/indexer.pydiffer between the previous and thefinalized v20 tree. Both were re-derived from the v20-final versions with the
EXL3 edits replayed on top, so the new DCP/indexer/prefill work is preserved
rather than overwritten by the overlay.
Validated image:
Runtime validation, 4x RTX PRO 6000 Blackwell (SM120a), TP4/DCP4, MTP-3, NVFP4 DS-MLA KV:
v0.11.2.dev280+gilded.gnosis.v20.vllm5517197.sibe0edca.fi801d57a.cu132.20260725vLLM is using nccl==2.30.4Trellis m=1..32 block_m=8, prefill trellis block_m=64 arena=1054.2MiB capacity=3072 chunk=128 topk=8DCP_*=auto): query split 1,full CKV gather 1, top-k owner merge 1, indexer shards 0, CKV prefetch depth 1,
and
DCP_PREFILL_WORKSPACE=1(VLLM_DCP_PROJECT_BEFORE_MERGE=1+VLLM_B12X_MLA_DCP_GATHER_IN_WORKSPACE=1)Preallocated 30.8 MiB for 2 persistent CKV execution lane(s)andUsing native CKV layer prefetch with depth=1 and 2 workspace slotsthan the pre-fix image because the draft model now gets its own rank-sliced
runtime arena instead of sharing the target's -- the intended cost of the
target/draft scratch isolation.
Earlier images in this series measured LAVD at 2E/3N/0F (pre-fix) and, on one
5-run sample of the first scoped build, 2E/2N/1F; a second sample of that build
returned 2E/3N/0F. The single failure there was a wrong ledger total at 7,114
completion tokens against a 24,576 cap -- an answer-quality miss rather than
truncation, and within this profile's run-to-run spread. This build shows no
failures on either suite.
Note: the base's
Loading safetensors using InstantTensor loaderassertion doesnot appear on this path -- the EXL3 checkpoint is loaded by the EXL3 rank-sliced
loader, not the base InstantTensor path. All other boot assertions hold.
Production validation
Validated on a 332.19 GB GLM-5.2 checkpoint with 256 routed experts and four
RTX PRO 6000 Blackwell 96 GB GPUs.
Validated serving configuration:
Correctness and quality:
0.1124021, sample SD0.00259480.1036666, sample SD0.0018374Exact-image performance:
Release constraint
Asynchronous scheduling remains disabled in the published configuration. The
source-level target/draft isolation fixes are necessary, but hardware testing
still showed a quality regression when async retirement overlapped speculative
side-stream work and the module-global CKV gather state. The validated release
therefore uses synchronous scheduling, MTP1, DCP draft sharding enabled, global
top-k enabled, and
VLLM_B12X_MLA_SPEC_EXTEND_AS_DECODE=0.Notes and limits
unsupported shards fail closed to the parity implementation.
retain generic/parity behavior and are not claimed as performance targets.
does not expose a compatible cache builder.
g60c82d972string. Theimage's
ai.verdict.vllm.revisionOCI label records the installed PR head.CI status
CodeRabbit passes. The repository pre-run gate currently blocks the normal
pre-commit job because it requires a
verified,ready, orready-run-all-testslabel (or four prior merged PRs). None of those labelsexists in the repository, and the contributor token cannot create one, so a
maintainer label/configuration action is required to start that job.
AI assistance disclosure
Implementation, debugging, and documentation were developed with Codex and
Claude Code assistance. The author selected the design, reviewed the changes,
and ran the reported hardware validation.
Summary by CodeRabbit
New Features
Bug Fixes