[https://nvbugs/6448152][perf] TEST ONLY; DO NOT REVIEW: diagnose PP consensus traffic

[chienchunhung]

Caution

DIAGNOSTIC ONLY — DO NOT MERGE.

Purpose

Determine whether runtime/per-request C++ PP context-transfer consensus traffic is the dominant cause of NVBUG 6448152's throughput regression.

This branch is a range-equivalent replay of #16386 and diagnostic PR #16449 onto current main (4188dbe330) so Jenkins can construct a merge commit. Original #16449 head 91935fc corresponds to rebased commit 656873e; this PR replaces that early-local-completion arm with a stronger no-hot-traffic arm.

graph LR
    A["current main"] --> B["#16386 replay: worker-published consensus"]
    B --> C["#16449 replay: early local completion diagnostic"]
    C --> D["this PR: no runtime/per-request PP consensus traffic"]

Loading

Diagnostic behavior

When TRTLLM_DIAGNOSTIC_DISABLE_PP_CONTEXT_CONSENSUS_TRAFFIC=1 on the qualified C++ NIXL/UCX TP1/CP1 PP path:

• all PP ranks agree on the diagnostic mode at startup;
• no ContextTransferVoteMailbox is constructed;
• sender workers do not publish completion votes;
• scheduler polls cannot call PP collective reduction or mailbox polling;
• teardown sends no mailbox close markers;
• successful local completions commit and retire their retained request state immediately;
• a local failure, configured deadline violation, unsupported configuration, or mixed rank mode is fatal.

The exact GB300 perf-sanity YAML enables the flag. The PP4 TinyLlama integration test requires this runtime marker on every CTX PP worker:

DIAGNOSTIC OBSERVED: context transfer completed with no runtime PP consensus traffic

What remains

This does not remove all PP communication:

• the two one-time startup capability/mode allgathers remain;
• generation-side TP/DP status consensus remains;
• normal PP model-control/activation traffic and KV data transfer remain.

The experiment isolates the context-transfer runtime consensus mailbox, not the whole PP stack.

Interpretation

Historical comparison points:

• [https://nvbugs/6448152][perf] TEST ONLY; DO NOT REVIEW: diagnose local CTX completion #16449 diagnostic: 798.40 output tokens/s
• pre-regression baseline: 1562.83 output tokens/s

If this result remains near 800, runtime mailbox sends/polls plus consensus-state retention are not the dominant regression, and two-phase lifecycle or admission-control investigation should take priority over transport redesign.

If throughput materially recovers, the result justifies redesigning the C++ consensus transport. It would not distinguish packet/poll cost from retained-state retirement/cleanup, because this arm removes both.

Because this PR runs on newer main than the historical #16449 result, a surprising delta must be confirmed with a same-base #16449 control run before attribution.

Validation

• Full pre-commit suite on all changed files
• Python syntax validation
• Test-list AST validation
• Rebase range-diff checked; only conflict resolution preserved both current-main generation-log assertions and diagnostic CTX-log assertions
• Target next: GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #59681 [ run ] triggered by Bot. Commit: 1522ea7 Link to invocation

[tensorrt-cicd]

PR_Github #59681 [ run ] completed with state FAILURE. Commit: 1522ea7
/LLM/main/L0_MergeRequest_PR pipeline #48113 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #59735 [ run ] triggered by Bot. Commit: 1522ea7 Link to invocation

[tensorrt-cicd]

PR_Github #59735 [ run ] completed with state FAILURE. Commit: 1522ea7
/LLM/main/L0_MergeRequest_PR pipeline #48162 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation