[https://nvbugs/6312828][fix] Recover disaggregated requests after timeouts by chienchunhung · Pull Request #16402 · NVIDIA/TensorRT-LLM

chienchunhung · 2026-07-15T05:08:06Z

Summary

Fixes NVBug 6312828, where high-concurrency disaggregated Qwen3-32B requests could become terminal at the HTTP layer while backend KV-transfer work remained stranded, preventing the cluster from recovering.

This PR makes timeout and cancellation outcomes terminal and client-visible, keeps asynchronous CTX transfers progressing and reaped, and rejects stale or replayed C++ handshakes. In-flight transfer cancellation remains disabled by default, and kv_transfer_admission_window_multiplier remains 1.

Fix Scope

Apply one request deadline across routing, CTX/GEN dispatch, retries, and response streaming.
Return HTTP 504 before response headers; after SSE begins, emit an in-band timeout error followed by [DONE].
Do not retry expired deadlines or upstream HTTP 504 responses. Preserve only retries that are safe before GEN connection establishment.
Make coordinator accounting, reservations, sibling tasks, upstream streams, shutdown, and request cleanup cancellation-safe and bounded.
Terminalize queued requests before removing executor bookkeeping.
Keep an otherwise-idle CTX executor polling while any rank owns asynchronous KV-transfer work, and defer drain-required control operations until ownership is released on every rank.
Make default-off C++ cancellation phase-safe:
- cancellation wins before any peer publishes a transfer session;
- after the first peer arrives, the request drains naturally;
- GEN cancellation remains observe-only until C++ reports a rank-consistent terminal result.
Reject late or replayed handshakes with ready=false.
Bound pending pre-handshake cancellation state to 65,536 entries. Entries have at least 30 seconds of retention and are reclaimed under capacity pressure into separately bounded finalized-handshake history, preserving late-peer rejection without unbounded growth.
Preserve the full Qwen3-32B stress workload: 10,000 requests, concurrency 512, a 180-second request deadline, 5-second heartbeat, and 10-second inactivity timeout.

Scope boundaries:

C++ in-flight transfer cancellation remains disabled by default; permanently wedged active transport operations are not claimed resolved.
The Python V2 transceiver cancellation protocol is unchanged.
Admission-window tuning is separate work.
B200 DeepSeek under NVBug 6472256 and H100 Qwen3.5 under NVBug 6479324 remain excluded from this PR's resolution claim.

Review Guide

Suggested review order:

HTTP deadline and cleanup: openai_client.py, openai_disagg_server.py, openai_disagg_service.py, and disagg_coordinator.py.
Executor terminalization and transfer reaping: py_executor.py, request_utils.py, waiting_queue.py, and kv_cache_transceiver.py.
C++ cancellation and replay handling: cacheTransceiver.cpp, dataTransceiver.cpp, and the agent connection helpers.
Regression coverage: focused Python/C++ unit tests and the disaggregated stress harness.

The diff is test-heavy: approximately 3.8k added test lines versus 1.6k added production lines.

Validation

Current head: 3c7358cf3197

Both NVBug 6312828 Qwen3-32B hardware variants passed previously on production-code ancestor fca8e33571. Because this head adds the bounded cancellation-state fix and removes the remaining B200 waiver, fresh-head B200 and H100 runs are in progress before final verification is claimed.

Workload	Scope	Validation	Waiver after this PR
B200 `test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]` — 10,000 requests	Original NVBug 6312828 scenario; later re-waived under NVBug 6472256	Historical pass: helper #60877 / QA #311. Fresh-head run requested.	Unwaived
H100 `test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]` — 10,000 requests	NVBug 6312828 hardware counterpart	Historical pass: helper #60878 / QA #312. Fresh-head mirror run requested.	Unwaived
B200 `test_disaggregated_stress_test[input8k-output1k-conc512-deepseek_r1_v2_fp4_stress]` — 35,000 requests	Separate NVBug 6472256 workload	Not verified or claimed resolved here.	Remains waived
H100 `test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_5_4b_fp8_stress]` — 3,000 requests	Separate NVBug 6479324 workload	Blocked by a hybrid-cache datatype defect. PR #16505 contains the corresponding generic dtype fix, but this exact workload has not been verified or unwaived here.	Remains waived

Static validation on 3c7358cf3197 passes, including clang-format, Ruff, codespell, DCO, waiver/test-list validation, and the full pre-commit diff checks. Two independent self-review passes found no remaining actionable P0/P1/P2 issue. The new MPI/CUDA C++ race tests require the hardware CI environment and could not be executed locally.

Fresh-head targeted QA and full pre-merge CI results will be added here when complete.

chienchunhung · 2026-07-15T05:37:38Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-15T05:43:30Z

PR_Github #59370 [ run ] triggered by Bot. Commit: fb8b982 Link to invocation

…16402) This update ensures that commented-out lines in the Groovy configuration are ignored during stage parsing, preventing disabled stage configurations from being processed. Signed-off-by: yufeiwu-nv <230315618+yufeiwu-nv@users.noreply.github.com>

tensorrt-cicd · 2026-07-15T12:32:17Z

PR_Github #59370 [ run ] completed with state FAILURE. Commit: fb8b982
LLM_FUNCTION_AUTO_V2C #262 completed with status: 'UNSTABLE'
QA verify test: disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress] (NVBug 6312828, branch codex/nvbug6312828-proxy-deadline, fork chienchunhung, dry_run_close=true)

Link to invocation

chienchunhung · 2026-07-15T18:42:24Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-15T19:06:55Z

PR_Github #59512 [ kill ] triggered by Bot. Commit: 0ad94a2 Link to invocation

tensorrt-cicd · 2026-07-15T19:07:08Z

PR_Github #59512 [ kill ] completed with state SUCCESS. Commit: 0ad94a2
Successfully killed previous jobs for commit 0ad94a2

Link to invocation

chienchunhung · 2026-07-15T19:07:39Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-15T19:14:14Z

PR_Github #59514 [ run ] triggered by Bot. Commit: 0ad94a2 Link to invocation

chienchunhung · 2026-07-15T22:46:01Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-15T22:51:30Z

PR_Github #59545 [ run ] triggered by Bot. Commit: 6fa5884 Link to invocation

tensorrt-cicd · 2026-07-16T04:03:11Z

PR_Github #59545 [ run ] completed with state FAILURE. Commit: 6fa5884
LLM_FUNCTION_AUTO_V2C #270 completed with status: 'UNSTABLE'
QA verify test: disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress] (NVBug 6312828, branch codex/nvbug6312828-proxy-deadline, fork chienchunhung, dry_run_close=true)

Link to invocation

chienchunhung · 2026-07-16T15:19:18Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-16T15:26:01Z

PR_Github #59737 [ run ] triggered by Bot. Commit: 92bfff6 Link to invocation

tensorrt-cicd · 2026-07-16T16:22:35Z

PR_Github #59737 [ run ] completed with state FAILURE. Commit: 92bfff6
LLM_FUNCTION_AUTO_V2C #271 completed with status: 'UNSTABLE'
QA verify test: disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress] (NVBug 6312828, branch codex/nvbug6312828-proxy-deadline, fork chienchunhung, dry_run_close=true)

Link to invocation

chienchunhung · 2026-07-17T03:15:52Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-17T03:21:35Z

PR_Github #59856 [ run ] triggered by Bot. Commit: f58de71 Link to invocation

tensorrt-cicd · 2026-07-17T04:35:11Z

PR_Github #59856 [ run ] completed with state FAILURE. Commit: f58de71
LLM_FUNCTION_AUTO_V2C #272 completed with status: 'UNSTABLE'
QA verify test: disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress] (NVBug 6312828, branch codex/nvbug6312828-proxy-deadline, fork chienchunhung, dry_run_close=true)

Link to invocation

chienchunhung · 2026-07-17T05:24:41Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-17T05:30:21Z

PR_Github #59886 [ run ] triggered by Bot. Commit: e877ea8 Link to invocation

tensorrt-cicd · 2026-07-17T07:24:59Z

PR_Github #59886 [ run ] completed with state FAILURE. Commit: e877ea8
LLM_FUNCTION_AUTO_V2C #273 completed with status: 'UNSTABLE'
QA verify test: disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress] (NVBug 6312828, branch codex/nvbug6312828-proxy-deadline, fork chienchunhung, dry_run_close=true)

Link to invocation

chienchunhung · 2026-07-23T01:08:26Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-23T01:13:58Z

PR_Github #61139 [ run ] triggered by Bot. Commit: 3c7358c Link to invocation

tensorrt-cicd · 2026-07-23T01:15:30Z

PR_Github #61116 [ run ] completed with state ABORTED. Commit: 447c60b

Link to invocation

tensorrt-cicd · 2026-07-23T01:24:30Z

PR_Github #61139 [ run ] completed with state FAILURE. Commit: 3c7358c
LLM_FUNCTION_AUTO_V2C #313 completed with status: 'UNSTABLE'
QA verify test: disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress] (NVBug 6312828, branch codex/nvbug6312828-proxy-deadline, fork chienchunhung, dry_run_close=true)

Link to invocation

chienchunhung · 2026-07-23T03:07:22Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-23T03:12:47Z

PR_Github #61175 [ run ] triggered by Bot. Commit: 3c7358c Link to invocation

tensorrt-cicd · 2026-07-23T10:31:45Z

PR_Github #61175 [ run ] completed with state FAILURE. Commit: 3c7358c
/LLM/main/L0_MergeRequest_PR pipeline #49423 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-07-23T15:09:56Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-23T15:24:21Z

PR_Github #61316 [ run ] triggered by Bot. Commit: 3c7358c Link to invocation

chienchunhung · 2026-07-23T15:29:20Z

/bot run --only-qa-verify test disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress]

tensorrt-cicd · 2026-07-23T15:36:52Z

PR_Github #61320 [ run ] triggered by Bot. Commit: 3c7358c Link to invocation

tensorrt-cicd · 2026-07-23T15:40:30Z

PR_Github #61316 [ run ] completed with state ABORTED. Commit: 3c7358c

Link to invocation

chienchunhung · 2026-07-23T17:07:51Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-23T17:14:25Z

PR_Github #61347 [ run ] triggered by Bot. Commit: 3c7358c Link to invocation

tensorrt-cicd · 2026-07-23T17:14:30Z

PR_Github #61320 [ run ] completed with state ABORTED. Commit: 3c7358c

Link to invocation

tensorrt-cicd · 2026-07-23T23:28:37Z

PR_Github #61347 [ run ] completed with state SUCCESS. Commit: 3c7358c
/LLM/main/L0_MergeRequest_PR pipeline #49574 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung · 2026-07-24T23:18:59Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-24T23:25:09Z

PR_Github #61658 [ run ] triggered by Bot. Commit: 0c4771a Link to invocation

tensorrt-cicd · 2026-07-25T11:35:14Z

PR_Github #61658 [ run ] completed with state FAILURE. Commit: 0c4771a
/LLM/main/L0_MergeRequest_PR pipeline #49863 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

github-actions Bot assigned chienchunhung Jul 15, 2026

chienchunhung added the api-compatible Accepted LLM API contract change that is backwards-compatible label Jul 15, 2026

chienchunhung mentioned this pull request Jul 15, 2026

[https://nvbugs/6448152][perf] TEST ONLY; DO NOT REVIEW: diagnose local CTX completion #16449

Closed

chienchunhung force-pushed the codex/nvbug6312828-proxy-deadline branch from 92bfff6 to f58de71 Compare July 17, 2026 02:48

NVIDIA deleted a comment from tensorrt-cicd Jul 17, 2026

chienchunhung force-pushed the codex/nvbug6312828-proxy-deadline branch from 447c60b to 3c7358c Compare July 23, 2026 01:06

chienchunhung added 9 commits July 24, 2026 16:03

[NVBUG 6312828][fix] Terminalize disaggregated request timeouts

32ed512

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][test] Bound post-stress recovery concurrency

83959e0

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][test] Gate post-stress recovery readiness

2d24457

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][fix] Reap idle context transfers safely

1ee9af2

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][fix] Coordinate default-off transfer cleanup

813fd8c

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][fix] Reject replayed KV handshakes

fd245e2

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][test] Correct transfer cleanup assertions

104f527

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][fix] Repair focused CI regressions

e3adc98

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[NVBUG 6312828][fix] Bound pre-handshake cancellation state

0c4771a

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung force-pushed the codex/nvbug6312828-proxy-deadline branch from 3c7358c to 0c4771a Compare July 24, 2026 23:18

Uh oh!

Conversation

chienchunhung commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Fix Scope

Review Guide

Validation

Uh oh!

chienchunhung commented Jul 15, 2026

Uh oh!

tensorrt-cicd commented Jul 15, 2026

Uh oh!

tensorrt-cicd commented Jul 15, 2026

Uh oh!

chienchunhung commented Jul 15, 2026

Uh oh!

tensorrt-cicd commented Jul 15, 2026

Uh oh!

tensorrt-cicd commented Jul 15, 2026

Uh oh!

chienchunhung commented Jul 15, 2026

Uh oh!

tensorrt-cicd commented Jul 15, 2026

Uh oh!

chienchunhung commented Jul 15, 2026

Uh oh!

tensorrt-cicd commented Jul 15, 2026

Uh oh!

tensorrt-cicd commented Jul 16, 2026

Uh oh!

chienchunhung commented Jul 16, 2026

Uh oh!

tensorrt-cicd commented Jul 16, 2026

Uh oh!

tensorrt-cicd commented Jul 16, 2026

Uh oh!

chienchunhung commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

chienchunhung commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

tensorrt-cicd commented Jul 17, 2026

Uh oh!

chienchunhung commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

chienchunhung commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

chienchunhung commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

chienchunhung commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

chienchunhung commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

chienchunhung commented Jul 15, 2026 •

edited

Loading