[GG] fix(moe): safely overlap shared experts before resident grids

[voipmonitor]

Summary

• query the selected SparkInfer MoE plan before using the shared-expert auxiliary stream
• submit eligible shared-expert work before gate/router work
• join the auxiliary stream immediately before a resident routed-MoE launch
• reject overlap for the NVFP4/NF3 hybrid resident-grid path
• keep all CUDA graph, DCP channel, and workspace lifecycle changes out of this PR

Root cause and ordering contract

A resident-grid MoE kernel can deadlock if it is submitted while auxiliary work occupies an SM needed by its grid-wide barrier. Blanket serialization avoids the deadlock but creates a measurable DCP1 A16 decode regression.

The safe order is:

1. establish the input dependency and submit shared experts on the auxiliary stream
2. execute gate/router work on the consumer stream
3. join the auxiliary stream
4. submit the resident routed-MoE grid

This preserves gate/router overlap without allowing shared experts and the resident grid to execute concurrently.

Dependency

Requires local-inference-lab/sparkinfer#60.

This branch is based directly on dev/gilded-gnosis; it is not stacked. It contains the MoE portion split out of #149 plus the profiler-guided no-regression ordering correction.

Validation

• focused vLLM MoE/quantization suite: 29 passed, 2 skipped
• Ruff lint and format checks passed
• git diff --check passed
• TP8/DCP1/MTP0/A16 CC1:
  • v17 baseline mean: 87.745 tok/s
  • previous [GG] fix(speculation): isolate graph resources and profile MRV2 memory #149 ordering mean: 86.929 tok/s
  • corrected ordering median: 87.643 tok/s
  • remaining delta vs v17: -0.12%
• KV capacity unchanged at 578,368 tokens
• profiler: auxiliary idle gap reduced from 162 us to 79 us; v17 was 71 us

[coderabbitai]

Warning

Review limit reached

@voipmonitor, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 59 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 9835dd87-94b2-42e8-a829-aea6747f2c1a

📥 Commits

Reviewing files that changed from the base of the PR and between 73e4a8c and 6420fe5.

📒 Files selected for processing (8)

• tests/model_executor/layers/fused_moe/test_shared_experts_stream.py
• tests/model_executor/layers/test_b12x_moe_warmup.py
• tests/quantization/test_nvfp4_nf3_hybrid.py
• vllm/model_executor/layers/fused_moe/b12x_moe.py
• vllm/model_executor/layers/fused_moe/fused_moe_method_base.py
• vllm/model_executor/layers/fused_moe/runner/moe_runner.py
• vllm/model_executor/layers/fused_moe/runner/shared_experts.py
• vllm/model_executor/layers/quantization/nvfp4_nf3_hybrid.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/gg-moe-shared-expert-ordering-20260721

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[voipmonitor]

Closing because the SparkInfer capability contract in dependency #60 was based on an invalid causal attribution.

The underlying matrix now shows that M=1/4/7 are stable with and without auxiliary-stream overlap, while M=8 varies even under serial eager execution. No overlap-specific deadlock or corruption was reproduced, and the exact stress workload passes on the SparkInfer parent commit as well as the latest commit.

The vLLM scheduling change in this PR is therefore not justified as a correctness fix. SparkInfer 695c011 handles the actual all-CTA admission invariant locally with a cooperative kernel launch. The independent M=8 numerical behavior will be investigated separately.

[voipmonitor]

Final upstream diagnosis: the M=8 observation behind the proposed caller-side serialization was not an auxiliary-stream overlap failure. On the real GLM-5.2 TP8 NVFP4 shape, M=7 selects SparkInfer fused micro, while M=8 reaches 64 routed rows and selects the dynamic kernel. The dynamic default uses BF16 atomic scatter, so small serial run-to-run differences are expected; its deterministic-output mode produced eight bitwise-identical repeats.

Exact M=4 fused-micro stress with prequeued auxiliary GEMMs passed both before and after the cooperative-launch change, with no deadlock. SparkInfer now carries the all-CTA admission contract at the kernel launch itself, and SparkInfer PR #71 documents the GLM planner boundary. There is no evidence supporting a vLLM capability gate or shared-expert serialization change here, so keeping this PR closed is the correct outcome.