[None][test] Enable gen_only + ctx_only DeepSeek-V4-Pro perf-sanity on GB300

[chenfeiz0326]

Summary

Follow-up to #16540. That PR added the four DeepSeek-V4-Pro (FP4, 1.6T MoE) disaggregated perf-sanity shapes on GB300 (con8 / con180 / con666 / con4301) but registered only the e2e variant of each — the gen_only and ctx_only variants were left commented out in the test-db yamls:

• gen_only — deferred pending gen_only runtime characterization on the target CI cluster.
• ctx_only — deferred behind a shared-stage OOM-cascade investigation (the con4301 ctx_only case passed cleanly in isolation on OCI-AGA, so the leak was judged orthogonal to these configs).

This PR enables both remaining modes so all three test modes (aggr_upload-ctx_only-*, disagg_upload-gen_only-*, disagg_upload-e2e-*) run for every shape.

Why this needs no config/model changes

The four disagg config yamls, the deepseek_v4_pro_fp4 → DeepSeek-V4-Pro model registration, the role-specific ctx_worker_env_var / gen_worker_env_var handling, and the in-repo MoE load-balancer yamls all landed in #16540 and are unchanged here. parse_test_string() derives benchmark_mode from the test-ID prefix and resolves the same on-disk config for all three modes (only the launched worker fleet differs), so enabling gen_only/ctx_only is purely a test-list registration change.

Changes

File	Change
l0_gb300_multi_gpus_perf_sanity.yml	Uncomment the 4 aggr_upload-ctx_only-* DeepSeek-V4-Pro entries (8 → 12 active ctx_only tests)
l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen4_node2_gpu8.yml (con8)	Uncomment the gen_only entry (now 1 gen_only + 1 e2e)
l0_gb300_multi_nodes_perf_sanity_ctx6_node1_gpu4_gen1_node4_gpu16.yml (con666)	Uncomment the gen_only entry
l0_gb300_multi_nodes_perf_sanity_ctx3_node1_gpu4_gen1_node8_gpu32.yml (con180)	Uncomment the gen_only entry
l0_gb300_multi_nodes_perf_sanity_ctx12_node1_gpu4_gen1_node2_gpu8.yml (con4301)	Uncomment the gen_only entry
jenkins/L0_Test.groovy	GB300-4_GPUs PerfSanity-Post-Merge split 2 → 3 (keeps ~4 tests/shard for the heavier ctx_only set, giving the DSv4-Pro cases more per-node isolation given the #16540 shared-stage OOM note); the 4 GB300 DeepSeek-V4-Pro multi_nodes Disagg stages testCount 1 → 2 (gen_only + e2e)

Validation

• All 5 edited test-db yamls parse as valid YAML; each multi_nodes file holds exactly 1 gen_only + 1 e2e entry, matching testCount=2.
• All 12 registered DeepSeek-V4-Pro test IDs resolve to an existing config file under tests/scripts/perf-sanity/disaggregated/.
• multi_gpus ctx_only count (12) vs split (3) → ~4 tests/shard.

Test plan

• gen_only + ctx_only local functional runs on GB300 (aws-cmh)
• CI post-merge stages green (gen_only/ctx_only)

Note (carried from #16540): slurm_install.sh still hard-codes --cuda_architectures '100-real' (GB200); GB300 source builds need 103-real. Not addressed here.

Dev Engineer Review

• Enabled four DeepSeek-V4-Pro FP4 ctx_only tests in l0_gb300_multi_gpus_perf_sanity.yml (the previously commented entries are now active; TIMEOUT (90) preserved).
• Enabled one DeepSeek-V4-Pro FP4 disaggregated gen_only test per GB300 multi-node PerfSanity config (4 total) across:
  • ...ctx12_node1_gpu4_gen1_node2_gpu8.yml
  • ...ctx1_node1_gpu4_gen4_node2_gpu8.yml
  • ...ctx3_node1_gpu4_gen1_node8_gpu32.yml
  • ...ctx6_node1_gpu4_gen1_node4_gpu16.yml
  • (each TIMEOUT (120) preserved; gen_only now active alongside existing e2e).
• Updated jenkins/L0_Test.groovy for GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge by increasing the GB300 4-GPU split from 2 → 3 and adding the missing split variant ...-3 (with ...-1, ...-2, ...-3 now present).
• Updated DeepSeek-V4-Pro FP4 disaggregated multi-node PerfSanity post-merge stage generation so the relevant Disagg stage testCount values are increased from 1 → 2, matching the intent that each stage shape now runs two active tests (gen_only + e2e).
• Changes are scoped to test scheduling/coverage and are consistent with related configs; no model registration/worker environment/MoE load-balancer changes were made.
• Validation: edited YAML entries parse correctly and the enabled test IDs resolve to existing configs; multi-GPU distribution and sharding behavior matched expectations (~four tests per shard).

QA Engineer Review

Modified test-list files:

• tests/integration/test_lists/test-db/l0_gb300_multi_gpus_perf_sanity.yml
  • Enabled 4 aggr_upload-ctx_only-gb300_deepseek-v4-pro-fp4_8k1k perf/test_perf_sanity.py::test_e2e[...] entries (timeouts retained).
• tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_ctx12_node1_gpu4_gen1_node2_gpu8.yml
  • Enabled 1 disagg_upload-gen_only... perf/test_perf_sanity.py::test_e2e[...] entry (timeout retained).
• tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen4_node2_gpu8.yml
  • Enabled 1 disagg_upload-gen_only... perf/test_perf_sanity.py::test_e2e[...] entry (timeout retained).
• tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_ctx3_node1_gpu4_gen1_node8_gpu32.yml
  • Enabled 1 disagg_upload-gen_only... perf/test_perf_sanity.py::test_e2e[...] entry (timeout retained).
• tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_ctx6_node1_gpu4_gen1_node4_gpu16.yml
  • Enabled 1 disagg_upload-gen_only... perf/test_perf_sanity.py::test_e2e[...] entry (timeout retained).

Verdict: needs follow-up — CBTS coverage data isn’t provided, but local GB300 functional runs and CI post-merge stages were reported passing.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 49e72c1a-0597-4823-9e75-8c00b6dfec10

📥 Commits

Reviewing files that changed from the base of the PR and between 544d14c and 8a3f13d.

📒 Files selected for processing (1)

• jenkins/L0_Test.groovy

💤 Files with no reviewable changes (1)

• jenkins/L0_Test.groovy

---

Walkthrough

GB300 PerfSanity CI configuration adds a third PyTorch split, generates two active tests per DeepSeek-V4-Pro disaggregated stage, and enables five previously commented-out gen_only or ctx_only test cases.

Changes

GB300 PerfSanity coverage

Layer / File(s)	Summary
Expand PyTorch split coverage jenkins/L0_Test.groovy	Adds the third GB300 PyTorch PerfSanity split and updates the total split count from 2 to 3.
Enable disaggregated test coverage jenkins/L0_Test.groovy, tests/integration/test_lists/test-db/*.yml	Updates GB300 disaggregated stage generation from one to two tests per shape and activates corresponding gen_only and ctx_only YAML entries.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• NVIDIA/TensorRT-LLM#9138: Adds related multi-node disaggregated PerfSanity launch and routing support.
• NVIDIA/TensorRT-LLM#10598: Updates multi-node PerfSanity configuration from two splits to three.
• NVIDIA/TensorRT-LLM#10912: Adds related SBSA multi-node disaggregated PerfSanity stage generation.

Suggested reviewers: niukuo, mzweilz, joyang-nv, yuanjingx87

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title matches the main change and is concise, with the required ticket/type prefix format.
Description check	✅ Passed	The description clearly explains the change, rationale, validation, and test plan, though it omits the template's explicit PR Checklist section.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[longlee0622]

Approve for multi-gpu testing.

[chenfeiz0326]

/bot run --disable-fail-fast --stage-list "GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-1,GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-2,GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-3,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN4-NODE2-GPU8-Post-Merge-1,GB300-40_GPUs-10_Nodes-PyTorch-Disagg-PerfSanity-CTX6-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1,GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1,GB300-56_GPUs-14_Nodes-PyTorch-Disagg-PerfSanity-CTX12-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #61476 [ run ] triggered by Bot. Commit: 8a3f13d Link to invocation

[tensorrt-cicd]

PR_Github #61476 [ run ] completed with state FAILURE. Commit: 8a3f13d
/LLM/main/L0_MergeRequest_PR pipeline #49696 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chenfeiz0326]

/bot run --disable-fail-fast --stage-list "GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN4-NODE2-GPU8-Post-Merge-2,GB300-40_GPUs-10_Nodes-PyTorch-Disagg-PerfSanity-CTX6-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2,GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2,GB300-56_GPUs-14_Nodes-PyTorch-Disagg-PerfSanity-CTX12-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2"

[tensorrt-cicd]

PR_Github #61527 [ run ] triggered by Bot. Commit: 8a3f13d Link to invocation

[tensorrt-cicd]

PR_Github #61527 [ run ] completed with state FAILURE. Commit: 8a3f13d
/LLM/main/L0_MergeRequest_PR pipeline #49742 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation