[https://nvbugs/6379316][fix] Reject MNNVL on split NVLink topology by karljang · Pull Request #16603 · NVIDIA/TensorRT-LLM

karljang · 2026-07-20T06:23:04Z

Summary

Fix the DeepSeek V3.2 H200 host-cache-offload and FP8 block-scale failures by addressing three distinct layers in the failure chain: invalid communication selection on split NVLink topologies, an empty-EP-rank hole in FP8 block-scale MoE, and cross-test GPU-memory retention in a reused MPI worker pool.

Bug scope

NVBug 6379316: fixes the split-topology MNNVL selection and resulting dispatch timeout.
NVBug 6384136: restores and passes test_dsa_host_cache_offload[host_cache_offload] and test_fp8_blockscale[baseline].

Fix 1: Reject unsupported MNNVL communication

Focus: topology-aware communication selection without disabling RDMA-based DeepEP.

Per-device active NVLink state does not prove that every visible GPU belongs to one fabric-attached MNNVL domain. PCIe NVL systems can contain multiple local NVLink islands connected only through PCIe/SYS, but the previous check treated them like a full NVSwitch fabric. This selected NVLinkOneSided across ranks that could not communicate through the assumed fabric, causing cross-island dispatch timeouts and sticky CUDA launch failures.

This change:

Detects split H100/H200 PCIe NVL systems by device name and uses NVML topology as a Hopper-specific fallback.
Rejects MNNVL communication on those systems before communicator setup.
Rejects DeepEP low-latency communication only on the affected split Hopper topology. DeepEP low latency otherwise uses RDMA and does not require MNNVL, so B200 remains supported.
Preserves the existing MNNVL path on full NVSwitch systems.

For the affected H200 configuration, auto-selection now falls back to AllGatherReduceScatter.

Fix 2: Handle an empty EP rank in FP8 block-scale MoE

Focus: production correctness after communication fallback.

In the all-gather/reduce-scatter path, a small or skewed batch can route no tokens to any expert owned by one EP rank. Individual empty experts are supported, but the DeepSeek FP8 block-scale grouped GEMM did not support the case where every local expert offset was zero.

This change:

Detects the all-empty local routing result on the GPU without a device-to-host synchronization.
Supplies one zero-valued dummy row through a compute-path-only copy of the expert offsets used by FC1, activation/prequantization, and FC2.
Keeps the real routing metadata unchanged, so finalization discards the dummy row and produces the correct zero contribution.
Adds a deterministic regression test that routes every token away from one EP rank and verifies an all-zero result while preserving nonempty-rank behavior.
Removes the waiver for test_fp8_blockscale[baseline], restoring its H200 coverage.

Fix 3: Isolate the memory-heavy H200 test

Focus: deterministic CI execution.

The host-cache-offload test runs close to H200 memory capacity. When it reused a long-lived MPI worker pool, allocations retained by earlier tests reduced the available memory and caused a later FlashMLA allocation to OOM or time out even though the test passed in a fresh standalone run.

This change marks the test with private_mpi_session, giving it a fresh worker pool while leaving normal session reuse unchanged for other tests. It also removes the waiver for test_dsa_host_cache_offload[host_cache_offload], restoring its H200 coverage.

Test coverage

Topology and communication selection:
- PCIe NVL device-name detection.
- Hopper split-island detection through NVML topology.
- Full-NVSwitch preservation.
- B200 preservation without using the Hopper topology fallback.
- NVML initialization.
- DeepEP low-latency selection.
FP8 block-scale MoE:
- Deterministic empty-EP-rank regression.
- Nonempty EP-rank parity with the non-EP reference.
- Repeated empty-rank execution.
Local MPI session-isolation unit tests: 27/27 passed.
Local standalone 8x H200 NVL test_dsa_host_cache_offload[host_cache_offload]: passed; MMLU 88.499 (threshold 86.383).
Local standalone 8x H200 NVL test_fp8_blockscale[baseline]: passed; MMLU 88.523 (threshold 86.383), GSM8K 95.603 (threshold 92.397).
Focused real CI on commit ff40d28d2c:
- PR_Github #60776: SUCCESS.
- L0_MergeRequest_PR #49056: SUCCESS.
- Fresh DGX_H200-8_GPUs-PyTorch-Post-Merge-1 #2148: SUCCESS.
- test_dsa_host_cache_offload[host_cache_offload]: PASSED.
- test_fp8_blockscale[baseline]: PASSED.
- An unrelated unit test passed its automatic retry; neither affected test required a retry.
Latest targeted CI on commit ff40d28d2c:
- PR_Github #61342 / L0_MergeRequest_PR #49573: SUCCESS (Partly Tested).
- The H200 stage was reused and reported Skip - Passed in the previous pipelines; this is not an additional fresh execution.
B200 compatibility correction at head commit c78efe9f34:
- tests/unittest/_torch/test_mnnvl_utils.py: 9/9 passed in a B200 development container.
- The real, unmocked DeepEPLowLatency.is_platform_supported() returned True on B200.
- Pre-commit passed for all three changed files.
- Targeted B200 CI is pending.

Focused CI completion report

Latest targeted CI completion report

karljang · 2026-07-20T06:23:34Z

/bot run

tensorrt-cicd · 2026-07-20T06:29:52Z

PR_Github #60293 [ run ] triggered by Bot. Commit: 8356d7e Link to invocation

tensorrt-cicd · 2026-07-20T06:59:50Z

PR_Github #60293 [ run ] completed with state SUCCESS. Commit: 8356d7e
/LLM/main/L0_MergeRequest_PR pipeline #48646 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

karljang · 2026-07-20T07:38:07Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-20T07:44:38Z

PR_Github #60317 [ run ] triggered by Bot. Commit: 8356d7e Link to invocation

tensorrt-cicd · 2026-07-20T10:52:01Z

PR_Github #60317 [ run ] completed with state FAILURE. Commit: 8356d7e
/LLM/main/L0_MergeRequest_PR pipeline #48662 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

karljang · 2026-07-20T15:55:52Z

/bot help

github-actions · 2026-07-20T15:56:01Z

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental) --high-priority]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Supports wildcard * for pattern matching (e.g., "*PerfSanity*" matches all stages containing PerfSanity). Examples: "A10-PyTorch-1, xxx", "PerfSanity". The broad patterns "*" and "*Post-Merge*", including equivalent escaped or repeated-star forms and their use in comma-separated lists, require the ci: post-merge approved PR label. Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline. Requires the ci: post-merge approved PR label applied by an active member of NVIDIA/trt-llm-ci-approvers. The approval label remains in place when new commits are pushed.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Supports wildcard * for pattern matching. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx", --extra-stage "Post-Merge". The broad patterns "*" and "*Post-Merge*", including equivalent escaped or repeated-star forms and their use in comma-separated lists, require the ci: post-merge approved PR label.

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

--high-priority (OPTIONAL) : Run the pipeline with high priority. This option is restricted to authorized users only and will route the job to a high-priority queue.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

karljang · 2026-07-20T16:34:48Z

/bot run --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1" --disable-fail-fast

karljang · 2026-07-20T17:43:39Z

/bot run --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1" --disable-fail-fast

karljang · 2026-07-20T17:45:40Z

/bot run --stage-list "DGX_H200-8_GPUs-PyTorch-Post-Merge-1" --disable-fail-fast

tensorrt-cicd · 2026-07-20T17:51:59Z

PR_Github #60422 [ run ] triggered by Bot. Commit: 3c816ec Link to invocation

tensorrt-cicd · 2026-07-20T17:54:24Z

PR_Github #60421 [ run ] triggered by Bot. Commit: 3c816ec Link to invocation

tensorrt-cicd · 2026-07-20T17:58:02Z

PR_Github #60422 [ run ] completed with state ABORTED. Commit: 3c816ec

Link to invocation

tensorrt-cicd · 2026-07-20T21:20:47Z

PR_Github #60421 [ run ] completed with state ABORTED. Commit: 3c816ec
/LLM/main/L0_MergeRequest_PR pipeline #48759 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

karljang · 2026-07-20T21:21:27Z

/bot run --stage-list "DGX_H200-8_GPUs-PyTorch-Post-Merge-1" --disable-fail-fast

tensorrt-cicd · 2026-07-20T21:34:52Z

PR_Github #60464 [ run ] triggered by Bot. Commit: 3c816ec Link to invocation

tensorrt-cicd · 2026-07-21T01:40:37Z

PR_Github #60464 [ run ] completed with state SUCCESS. Commit: 3c816ec
/LLM/main/L0_MergeRequest_PR pipeline #48797 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>

karljang · 2026-07-21T16:41:41Z

/bot run --stage-list "DGX_H200-8_GPUs-PyTorch-Post-Merge-1" --disable-fail-fast

tensorrt-cicd · 2026-07-21T16:47:58Z

PR_Github #60737 [ run ] triggered by Bot. Commit: f4a5393 Link to invocation

Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>

karljang · 2026-07-21T20:26:16Z

/bot run --stage-list "DGX_H200-8_GPUs-PyTorch-Post-Merge-1" --disable-fail-fast

tensorrt-cicd · 2026-07-21T20:29:13Z

PR_Github #60737 [ run ] completed with state FAILURE. Commit: f4a5393
/LLM/main/L0_MergeRequest_PR pipeline #49024 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

karljang · 2026-07-21T20:31:14Z

/bot run --stage-list "DGX_H200-8_GPUs-PyTorch-Post-Merge-1" --disable-fail-fast

tensorrt-cicd · 2026-07-21T20:38:51Z

PR_Github #60776 [ run ] triggered by Bot. Commit: ff40d28 Link to invocation

tensorrt-cicd · 2026-07-22T02:51:13Z

PR_Github #60776 [ run ] completed with state SUCCESS. Commit: ff40d28
/LLM/main/L0_MergeRequest_PR pipeline #49056 (Partly Tested) completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

karljang · 2026-07-22T05:29:29Z

/bot run --disable-reuse-test --disable-fail-fast --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1"

tensorrt-cicd · 2026-07-22T05:37:19Z

PR_Github #60895 [ run ] triggered by Bot. Commit: ff40d28 Link to invocation

tensorrt-cicd · 2026-07-22T11:43:49Z

PR_Github #60895 [ run ] completed with state FAILURE. Commit: ff40d28
/LLM/main/L0_MergeRequest_PR pipeline #49163 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

karljang · 2026-07-22T16:10:02Z

/bot run --disable-reuse-test --disable-fail-fast --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1"

tensorrt-cicd · 2026-07-22T16:15:57Z

PR_Github #61021 [ run ] triggered by Bot. Commit: ff40d28 Link to invocation

tensorrt-cicd · 2026-07-22T19:40:01Z

PR_Github #61021 [ run ] completed with state FAILURE. Commit: ff40d28
/LLM/main/L0_MergeRequest_PR pipeline #49278 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

karljang · 2026-07-22T20:44:28Z

/bot run --disable-reuse-test --disable-fail-fast --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1"

tensorrt-cicd · 2026-07-22T20:50:15Z

PR_Github #61086 [ run ] triggered by Bot. Commit: ff40d28 Link to invocation

tensorrt-cicd · 2026-07-23T09:33:41Z

PR_Github #61086 [ run ] completed with state FAILURE. Commit: ff40d28
/LLM/main/L0_MergeRequest_PR pipeline #49340 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

karljang · 2026-07-23T17:01:17Z

/bot run --stage-list "DGX_H200-8_GPUs-PyTorch-Post-Merge-1" --disable-fail-fast

tensorrt-cicd · 2026-07-23T17:12:56Z

PR_Github #61342 [ run ] triggered by Bot. Commit: ff40d28 Link to invocation

tensorrt-cicd · 2026-07-23T17:47:18Z

PR_Github #61342 [ run ] completed with state SUCCESS. Commit: ff40d28
/LLM/main/L0_MergeRequest_PR pipeline #49573 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

karljang · 2026-07-23T18:16:58Z

/bot run --disable-fail-fast --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1"

tensorrt-cicd · 2026-07-23T18:23:41Z

PR_Github #61361 [ run ] triggered by Bot. Commit: ff40d28 Link to invocation

tensorrt-cicd · 2026-07-23T22:11:02Z

PR_Github #61361 [ run ] completed with state FAILURE. Commit: ff40d28
/LLM/main/L0_MergeRequest_PR pipeline #49589 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>

karljang · 2026-07-24T00:16:24Z

/bot run --stage-list "DGX_B200-4_GPUs-PyTorch-1" --disable-reuse-test

karljang · 2026-07-24T00:20:57Z

/bot kill

karljang · 2026-07-24T00:23:03Z

/bot run --disable-fail-fast --extra-stage "DGX_H200-8_GPUs-PyTorch-Post-Merge-1"

tensorrt-cicd · 2026-07-24T00:27:36Z

PR_Github #61433 [ kill ] triggered by Bot. Commit: c78efe9 Link to invocation

tensorrt-cicd · 2026-07-24T00:27:40Z

PR_Github #61433 [ kill ] completed with state SUCCESS. Commit: c78efe9
Successfully killed previous jobs for commit c78efe9

Link to invocation

tensorrt-cicd · 2026-07-24T00:29:31Z

PR_Github #61434 [ run ] triggered by Bot. Commit: c78efe9 Link to invocation

tensorrt-cicd · 2026-07-24T04:20:55Z

PR_Github #61434 [ run ] completed with state SUCCESS. Commit: c78efe9
/LLM/main/L0_MergeRequest_PR pipeline #49658 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

github-actions Bot assigned karljang Jul 20, 2026

karljang changed the title ~~[6379316][fix] Reject MNNVL on split NVLink topology~~ [https://nvbugs/6379316][fix] Reject MNNVL on split NVLink topology Jul 20, 2026

karljang force-pushed the agent/reject-mnnvl-split-topology branch from c9bd803 to 3c816ec Compare July 20, 2026 17:27

karljang added 3 commits July 21, 2026 09:40

[6379316][fix] Reject MNNVL on split NVLink topology

2becc26

Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>

[6379316][test] Re-enable DeepSeek V3.2 H200 coverage

888cc5b

Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>

fix FP8 block-scale MoE on empty EP ranks

f4a5393

Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>

karljang force-pushed the agent/reject-mnnvl-split-topology branch from 8d7b539 to f4a5393 Compare July 21, 2026 16:41

[6379316][test] Isolate H200 host-cache test session

ff40d28

Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>

karljang added 2 commits July 23, 2026 22:55

[fix] Preserve DeepEP low latency on B200

d63915b

Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>

[fix] Scope split NVL detection to Hopper

c78efe9

Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>

Uh oh!

Conversation

karljang commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Bug scope

Fix 1: Reject unsupported MNNVL communication

Fix 2: Handle an empty EP rank in FP8 block-scale MoE

Fix 3: Isolate the memory-heavy H200 test

Test coverage

Uh oh!

karljang commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

karljang commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

karljang commented Jul 20, 2026

Uh oh!

github-actions Bot commented Jul 20, 2026

GitHub Bot Help

kill

skip

reuse-pipeline

Uh oh!

karljang commented Jul 20, 2026

Uh oh!

karljang commented Jul 20, 2026

Uh oh!

karljang commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

karljang commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

karljang commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

karljang commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

karljang commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

karljang commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

karljang commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

karljang commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

karljang commented Jul 20, 2026 •

edited

Loading