[TRTLLM-13948][test] Add DeepSeek R1/V3.2/V3-Lite disaggregated accuracy tests

[asfiyab-nvidia]

This pull request adds new integration tests for the DeepSeekR1 and DeepSeekV3.2 models to validate the behavior of KV cache manager v1 using the NIXL Python transceiver in a disaggregated serving setup.

New test coverage:

Added test_kv_cache_v2_nixl_python in TestDeepSeekR1 and TestDeepSeekV32Exp to test DeepSeekR1 and DeepSeekV3.2 with NIXL Python transceiver and KV cache manager v1, including configuration for context and generation servers with tensor parallelism and cache settings.
Test suite and test database updates:

Registered the new parameterized tests for cache manager v1 in the QA test list (test_lists/qa/llm_function_core.txt).
Added the new tests to the l0_dgx_b300 test database for automated runs on the appropriate hardware.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Tests

• Added new integration test coverage for KV cache v2 functionality with disaggregated serving setup
• Extended test scenarios for NIXL transceiver in Python mode
• Updated test configurations to include cache manager v1 parameterization variants for enhanced compatibility testing

[asfiyab-nvidia]

/bot run --add-multi-gpu-test --disable-fail-fast

[tensorrt-cicd]

PR_Github #59041 [ run ] triggered by Bot. Commit: 648b5a5 Link to invocation

[tensorrt-cicd]

PR_Github #59041 [ run ] completed with state SUCCESS. Commit: 648b5a5
/LLM/main/L0_MergeRequest_PR pipeline #47570 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[asfiyab-nvidia]

/bot run --add-multi-gpu-test --disable-fail-fast

[tensorrt-cicd]

PR_Github #59251 [ run ] triggered by Bot. Commit: 6e02fbd Link to invocation

[tensorrt-cicd]

PR_Github #59251 [ run ] completed with state SUCCESS. Commit: 6e02fbd
/LLM/main/L0_MergeRequest_PR pipeline #47739 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[asfiyab-nvidia]

The failing tests listed below are unrelated to the transceiver changes made in this PR

unittest/auto_deploy/singlegpu/models
unittest/auto_deploy/singlegpu/models/test_glm4_moe_modeling.py::test_glm4_moe_moe_equivalence[dtype0-2-6]

[coderabbitai]

Walkthrough

Updated disaggregated serving integration tests for NIXL Python KV-cache manager v1, added DeepSeekR1 coverage, adjusted DeepSeekV32Exp configuration, and registered both tests in QA and DGX B300 test lists.

Changes

Disaggregated KV-cache coverage

Layer / File(s)	Summary
Update disaggregated server configuration tests/integration/defs/accuracy/test_disaggregated_serving.py	Worker discovery now uses context and generation server instance counts without explicit ports or URLs.
Configure DeepSeekV32Exp cache-manager v1 test tests/integration/defs/accuracy/test_disaggregated_serving.py	The test adds fixed token, MoE, tensor-parallel, expert-parallel, autotuner, and overlap-scheduling settings for the cache-manager v1 path.
Add DeepSeekR1 coverage and scheduling tests/integration/defs/accuracy/test_disaggregated_serving.py, tests/integration/test_lists/qa/llm_function_core.txt, tests/integration/test_lists/test-db/l0_dgx_b300.yml	Adds the DeepSeekR1 NIXL Python cache-manager v1 test and registers DeepSeekR1 and DeepSeekV32Exp cases in test lists.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant AccuracyTest
  participant launch_disaggregated_llm
  participant ContextServer
  participant GenerationServer
  participant NIXLPython
  AccuracyTest->>launch_disaggregated_llm: launch cache-manager v1 configuration
  launch_disaggregated_llm->>ContextServer: start context server
  launch_disaggregated_llm->>GenerationServer: start generation server
  ContextServer->>NIXLPython: write KV cache
  NIXLPython->>GenerationServer: provide KV cache
  GenerationServer-->>AccuracyTest: return generation result

Loading

Suggested reviewers: nv-xtf, bo-nv, lfr-0531

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is concise and matches the main change: adding disaggregated accuracy tests for DeepSeek R1 and V3.2/V3-Lite.
Description check	✅ Passed	The description covers the change and test coverage at a high level, though the template’s Description/Test Coverage sections are sparse.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

tests/integration/test_lists/qa/llm_function_core.txt (1)

1-1: 📐 Maintainability & Code Quality | 🔵 Trivial

Coverage currently limited to cache_mgr_v1 only for both new entries.

Both new QA entries (TestDeepSeekR1 and TestDeepSeekV32Exp test_kv_cache_v2_nixl_python[cache_mgr_v1]) only exercise the KV-cache-manager-v1 path, per the parametrize decorators added in test_disaggregated_serving.py. This matches the PR's intentional narrowing (per commit history) after CI issues with the v2 path, but flagging for follow-up: once the cache_mgr_v2 path stabilizes, this list (and l0_dgx_b300.yml) should be updated to add the [cache_mgr_v2] variant, consistent with how TestGemma3_1BInstruct/TestGPTOSS register both cache_mgr_v1 and cache_mgr_v2 ids here.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/test_lists/qa/llm_function_core.txt` at line 1, Update the
QA registrations for TestDeepSeekR1 and TestDeepSeekV32Exp to include the
test_kv_cache_v2_nixl_python[cache_mgr_v2] variant once the cache_mgr_v2 path is
stable, matching the existing cache_mgr_v1 entries and the dual registrations
used by TestGemma3_1BInstruct and TestGPTOSS; also add the corresponding variant
to l0_dgx_b300.yml.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integration/defs/accuracy/test_disaggregated_serving.py`:
- Around line 2809-2869: Set enable_autotuner to False in both ctx_server_config
and gen_server_config within TestDeepSeekR1.test_kv_cache_v2_nixl_python.
Preserve all other server settings and ensure both context and generation
servers skip autotuner warmup.

---

Nitpick comments:
In `@tests/integration/test_lists/qa/llm_function_core.txt`:
- Line 1: Update the QA registrations for TestDeepSeekR1 and TestDeepSeekV32Exp
to include the test_kv_cache_v2_nixl_python[cache_mgr_v2] variant once the
cache_mgr_v2 path is stable, matching the existing cache_mgr_v1 entries and the
dual registrations used by TestGemma3_1BInstruct and TestGPTOSS; also add the
corresponding variant to l0_dgx_b300.yml.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 47c62172-cd90-48d0-8774-b8a2019896ad

📥 Commits

Reviewing files that changed from the base of the PR and between 54d484f and 6e02fbd.

📒 Files selected for processing (3)

• tests/integration/defs/accuracy/test_disaggregated_serving.py
• tests/integration/test_lists/qa/llm_function_core.txt
• tests/integration/test_lists/test-db/l0_dgx_b300.yml

[asfiyab-nvidia]

/bot run --add-multi-gpu-test --disable-fail-fast

[tensorrt-cicd]

PR_Github #59320 [ run ] triggered by Bot. Commit: d555a53 Link to invocation

[tensorrt-cicd]

PR_Github #59320 [ run ] completed with state SUCCESS. Commit: d555a53
/LLM/main/L0_MergeRequest_PR pipeline #47805 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[asfiyab-nvidia]

/bot run

[tensorrt-cicd]

PR_Github #59491 [ run ] triggered by Bot. Commit: d555a53 Link to invocation

[tensorrt-cicd]

PR_Github #59491 [ run ] completed with state SUCCESS. Commit: d555a53
/LLM/main/L0_MergeRequest_PR pipeline #47950 completed with status: 'SUCCESS'

CI Report

Link to invocation

[nv-xtf]

/bot run --disable-fail-fast --disable-reuse-test --stage-list "DGX_B300-4_GPUs-PyTorch-Post-Merge-1, DGX_B300-4_GPUs-PyTorch-Post-Merge-2"

[tensorrt-cicd]

PR_Github #60913 [ run ] triggered by Bot. Commit: 56f434e Link to invocation

[tensorrt-cicd]

PR_Github #60913 [ run ] completed with state SUCCESS. Commit: 56f434e
/LLM/main/L0_MergeRequest_PR pipeline #49179 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[nv-xtf]

/bot run --disable-fail-fast --stage-list "DGX_B300-4_GPUs-PyTorch-Post-Merge-1, DGX_B300-4_GPUs-PyTorch-Post-Merge-2"

[tensorrt-cicd]

PR_Github #61153 [ run ] triggered by Bot. Commit: 56f434e Link to invocation

[Shixiaowei02]

Wondering if we should raise both memory guards to match the TP2 footprint, or otherwise restrict these tests to B300. The V3.2 and R1 FP4 checkpoints are approximately 389–400 GB, so TP2 requires roughly 195–200 GB per rank before KV cache and workspace.

skip_less_device_memory(80000) still permits 96 GB SM120 systems, where this configuration cannot fit and the TRTLLM MoE backend is unsupported. A B300-sized threshold such as 200000 or higher would keep the QA registrations from running on incompatible hardware.

cc: @Shixiaowei02 What do you think?

@chienchunhung @nv-xtf I noticed that this test is only triggered in l0_dgx_b300.yml, so there is no immediate issue. However, we have seen cases where QA ran it on other platforms and filed an NVBug as a result. To help prevent this from happening again, I wonder if we should consider increasing the skip_memory value to something like 200000.

[nv-xtf]

Wondering if we should raise both memory guards to match the TP2 footprint, or otherwise restrict these tests to B300. The V3.2 and R1 FP4 checkpoints are approximately 389–400 GB, so TP2 requires roughly 195–200 GB per rank before KV cache and workspace.
skip_less_device_memory(80000) still permits 96 GB SM120 systems, where this configuration cannot fit and the TRTLLM MoE backend is unsupported. A B300-sized threshold such as 200000 or higher would keep the QA registrations from running on incompatible hardware.
cc: @Shixiaowei02 What do you think?

@chienchunhung @nv-xtf I noticed that this test is only triggered in l0_dgx_b300.yml, so there is no immediate issue. However, we have seen cases where QA ran it on other platforms and filed an NVBug as a result. To help prevent this from happening again, I wonder if we should consider increasing the skip_memory value to something like 200000.

Good catch, I wasn't aware of this — +1 to 200000.

[tensorrt-cicd]

PR_Github #61153 [ run ] completed with state SUCCESS. Commit: 56f434e
/LLM/main/L0_MergeRequest_PR pipeline #49402 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[asfiyab-nvidia]

Wondering if we should raise both memory guards to match the TP2 footprint, or otherwise restrict these tests to B300. The V3.2 and R1 FP4 checkpoints are approximately 389–400 GB, so TP2 requires roughly 195–200 GB per rank before KV cache and workspace.
skip_less_device_memory(80000) still permits 96 GB SM120 systems, where this configuration cannot fit and the TRTLLM MoE backend is unsupported. A B300-sized threshold such as 200000 or higher would keep the QA registrations from running on incompatible hardware.
cc: @Shixiaowei02 What do you think?

@chienchunhung @nv-xtf I noticed that this test is only triggered in l0_dgx_b300.yml, so there is no immediate issue. However, we have seen cases where QA ran it on other platforms and filed an NVBug as a result. To help prevent this from happening again, I wonder if we should consider increasing the skip_memory value to something like 200000.

Good catch, I wasn't aware of this — +1 to 200000.

Thanks! Memory guard is updated to 200000

[asfiyab-nvidia]

/bot run --disable-fail-fast --stage-list "DGX_B300-4_GPUs-PyTorch-Post-Merge-1, DGX_B300-4_GPUs-PyTorch-Post-Merge-2"

[tensorrt-cicd]

PR_Github #61402 [ run ] triggered by Bot. Commit: 762de1a Link to invocation

[tensorrt-cicd]

PR_Github #61402 [ run ] completed with state SUCCESS. Commit: 762de1a
/LLM/main/L0_MergeRequest_PR pipeline #49629 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[nv-xtf]

/bot run --disable-fail-fast --stage-list "DGX_B300-4_GPUs-PyTorch-Post-Merge-1, DGX_B300-4_GPUs-PyTorch-Post-Merge-2"

[nv-xtf]

/bot run --disable-fail-fast --stage-list "DGX_B300-4_GPUs-PyTorch-Post-Merge-1, DGX_B300-4_GPUs-PyTorch-Post-Merge-2"

[tensorrt-cicd]

PR_Github #61460 [ run ] triggered by Bot. Commit: 762de1a Link to invocation

[tensorrt-cicd]

PR_Github #61460 [ run ] completed with state SUCCESS. Commit: 762de1a
/LLM/main/L0_MergeRequest_PR pipeline #49681 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[nv-xtf]

/bot skip --comment "Relevant tests covering this change have passed."

[tensorrt-cicd]

PR_Github #61510 [ skip ] triggered by Bot. Commit: 762de1a Link to invocation

[tensorrt-cicd]

PR_Github #61510 [ skip ] completed with state SUCCESS. Commit: 762de1a
Skipping testing for commit 762de1a

Link to invocation