[https://nvbugs/6487040][test] Robustify gen_only per-iter device-step-time metric parsing by chenfeiz0326 · Pull Request #16717 · NVIDIA/TensorRT-LLM

chenfeiz0326 · 2026-07-22T08:52:53Z

Dev Engineer Review

Updated parse_gen_worker_device_step_time parsing/aggregation to avoid false None when num_generation_tokens cannot be parsed despite valid prev_device_step_time samples:
- _scan_gen_worker_device_step_time now returns per-file (by_ngen, all_iter_count, all_iter_mean) where all_iter_mean is a Welford mean over all numeric prev_device_step_time rows with iter >= 5, independent of whether num_generation_tokens parsed.
- _mean_at_mode_ngen now:
  - prefers the existing steady-state num_generation_tokens mode bucket (by_ngen) when non-empty
  - falls back to the per-file un-bucketed all_iter_mean when by_ngen is empty but all_iter_count indicates usable rows
  - returns None only when no usable numeric prev_device_step_time rows exist across any scanned file
  - emits print_info when fallback is used
- parse_gen_worker_device_step_time polling/settling loop now uses the scanner’s monotonic total_count (raw iter>=5 usable row count) to decide when logs have flushed before computing the mean.
Added tests/integration/defs/perf/test_gen_worker_device_step_time_parser.py:
- CPU-only pytest module using synthetic gen_server_*.log files
- isolates the shipped parser logic by slicing the parser block from tests/integration/defs/perf/test_perf_sanity.py and exec()-ing it into a minimal namespace (stubbing print_info) to avoid heavy integration dependency import chains
- test coverage includes mode-bucket selection, skipping iter < 5, ignoring non-numeric prev_device_step_time (e.g., N/A), fallback when num_generation_tokens is missing/unparsable, mixed-worker averaging, None when no usable data exists, None when logs are missing, and scan counting correctness.
Kept speculative tree behavior unchanged aside from a method rename:
- MTPEagleDynamicTreeWorker.forward(...) → MTPEagleDynamicTreeWorker._forward_impl(...) in tensorrt_llm/_torch/speculative/mtp_dynamic_tree.py with the NVTX decorator preserved.

QA Engineer Review

Test code changes:
- Added (new): tests/integration/defs/perf/test_gen_worker_device_step_time_parser.py
  - test_healthy_bare_int_uses_mode_ngen_bucket
  - test_iters_below_five_are_skipped
  - test_na_device_step_time_is_ignored
  - test_missing_num_generation_tokens_falls_back_to_all_iter_mean
  - test_unparsable_num_generation_tokens_falls_back (parametrized)
  - test_mixed_workers_bucket_and_fallback_average
  - test_no_usable_lines_returns_none
  - test_missing_log_file_returns_none
  - test_scan_counts_all_usable_rows_regardless_of_ngen
- Modified (parser logic, no new test functions): tests/integration/defs/perf/test_perf_sanity.py (updated prev_device_step_time gen-worker scan/aggregation logic used by the metric parser)
Test-list changes:
- Modified: tests/integration/test_lists/waives.txt
  - Updated perf/test_perf_sanity.py::test_e2e waiver entries for disagg_upload-gen_only on gb200 DeepSeek by removing a subset of SKUs associated with nvbugs/6487040 (as part of aligning CI expectations with the parser fix).
Coverage in tests/integration/test_lists/ (test-db/ for CI, qa/ for manual QA): not indicated for the new pure-CPU unit tests; the waiver removals are intended to re-enable/validate the previously skipped gen_only disagg perf-sanity cases.
Verdict: needs follow-up.

Description

The disagg gen_only perf-sanity regression metric
mean_gen_worker_per_iter_device_step_time is parsed from each
gen_server_{i}.log by parse_gen_worker_device_step_time()
(tests/integration/defs/perf/test_perf_sanity.py). Since #16298 restricted the
mean to steady-state iterations by bucketing rows on num_generation_tokens, a
gen worker whose prev_device_step_time lines are all present and numeric can
still yield a None metric when num_generation_tokens fails to match
_NUM_GEN_TOKENS_RE on every row (the key is absent from the states dict, or
rendered in a non-bare-int form). A None metric then trips
PerfSanityTestConfig.check_test_failure() with a misleading
RuntimeError: ... is missing 'prev_device_step_time' even though the value was
logged and the workload passed. This is the false-failure class reported in
nvbugs 6487036 (disagg_gen_missing_prev_device_step_time) and 6487040
(disagg_gen_prev_device_step_time_parse_mismatch), children of the L0_PostMerge
#2846 triage bug 6484688.

Re-running the 6487040 case (disagg_upload-gen_only-gb200_deepseek-v32-fp4_8k1k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL)
on GB200 passed (metric 12.91 ms), so the failure is intermittent — the
exact per-line trigger from #2846 was not captured. This change hardens the
parser so the intermittent extraction miss can no longer turn a present metric
into a spurious CI failure, rather than chasing a specific unreproduced render.

Fix: _scan_gen_worker_device_step_time now also accumulates a per-file
un-bucketed all-iter Welford mean over every iter >= 5 numeric
prev_device_step_time row. _mean_at_mode_ngen still prefers the steady-state
mode-ngen bucket (behavior unchanged for healthy logs) but falls back to that
all-iter mean for any worker whose buckets are empty, so a present metric is
never dropped to None. The fallback path emits a print_info. The parser
returns None only when no usable row exists in any worker at all — restoring the
pre-#16298 guarantee while keeping the steady-state refinement.

Test Coverage

New tests/integration/defs/perf/test_gen_worker_device_step_time_parser.py
(10 CPU-only cases): healthy bare-int mode-bucket; iter < 5 and N/A skips;
missing / unparsable num_generation_tokens -> all-iter fallback (not None);
mixed bucketed + fallback worker averaging; and the None cases (no usable
lines, missing file). The test importlib-loads the parser block with a stubbed
print_info so it runs standalone without the module's GPU/DB import chain.
Verified behavior-preserving on a real passing gen_server_0.log captured from
the v32 gen_only mtp3 case on lyris GB200: 12.91 ms before and after the
change (the mode-ngen path, unchanged).

PR Checklist

Please check this after reviewing the above items as appropriate for this PR.

coderabbitai · 2026-07-22T08:57:43Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

Walkthrough

The gen-worker step-time parser now retains usable samples without parseable num_generation_tokens, falls back to an all-iteration mean when needed, and uses the updated aggregation during polling. New isolated CPU-only tests cover these paths. The MTP worker entry method is renamed to _forward_impl, and selected performance waivers are removed.

Changes

Gen-worker device step-time parsing

Layer / File(s)	Summary
Per-file scan fallback `tests/integration/defs/perf/test_perf_sanity.py`	Per-file scans retain generation-token buckets alongside all-iteration counts and means for usable `prev_device_step_time` rows.
Aggregation and parser integration `tests/integration/defs/perf/test_perf_sanity.py`	Aggregation selects the most-sampled generation-token bucket or falls back to the all-iteration mean, while polling consumes the updated scan structure.
Isolated parser regression tests `tests/integration/defs/perf/test_gen_worker_device_step_time_parser.py`	Synthetic-log tests validate filtering, bucket selection, fallback formats, multi-worker averaging, empty results, and scan counts.

MTP worker entrypoint

Layer / File(s)	Summary
Worker method rename `tensorrt_llm/_torch/speculative/mtp_dynamic_tree.py`	`MTPEagleDynamicTreeWorker.forward(...)` is renamed to `_forward_impl(...)` with the same signature and body flow.

Performance waiver cleanup

Layer / File(s)	Summary
Performance waiver removals `tests/integration/test_lists/waives.txt`	The `disagg_upload-gen_only` waiver block is narrowed to three DeepSeek v32 32k4k entries, removing several other SKU-specific entries.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Parser
  participant GenWorkerLogs
  participant Scanner
  participant Aggregator
  Parser->>GenWorkerLogs: read gen_server_*.log files
  Parser->>Scanner: scan usable rows
  Scanner-->>Parser: return per-file scans and total count
  Parser->>Aggregator: select settled or timeout result
  Aggregator-->>Parser: return bucket mean or all-iteration fallback

Suggested reviewers: emmaqiaoch, brnguyen2, larryxfly, zhanruisunch, junyixu-nv

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the parser hardening change for gen_only device-step-time metrics.
Description check	✅ Passed	The description follows the template with Description, Test Coverage, and PR Checklist sections and is sufficiently detailed.
Docstring Coverage	✅ Passed	Docstring coverage is 93.75% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integration/defs/perf/test_gen_worker_device_step_time_parser.py`:
- Around line 30-34: Update the isolated parser loader used by test_perf_sanity
to accurately document that it reads and executes a source slice rather than
using importlib. At the exec call, add a narrowly scoped Ruff S102 suppression
with justification that _TPS_PATH is a fixed, repository-local trusted path;
avoid broad lint disables or unrelated changes.
- Around line 214-229: Strengthen
test_scan_counts_all_usable_rows_regardless_of_ngen by assigning distinct
step-time values to the bucketed and fallback-only row groups, while preserving
their separate num_generation_tokens states. Assert _mean_at_mode_ngen() returns
the mean for the selected ngen bucket rather than all_mean, and update expected
aggregate values accordingly; complete the required coverage review for this
test change.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2b12569b-ab7d-4b3f-8135-ad7d00ad5c39

📥 Commits

Reviewing files that changed from the base of the PR and between 9095cc1 and 5fb50f1.

📒 Files selected for processing (2)

tests/integration/defs/perf/test_gen_worker_device_step_time_parser.py
tests/integration/defs/perf/test_perf_sanity.py

chenfeiz0326 · 2026-07-22T09:41:19Z

This is a [test]-only change to the perf-sanity harness parser
(parse_gen_worker_device_step_time) that runs inside every gen_only
disaggregated PerfSanity stage. To validate the fix against real CI, I picked 10
gen_only disagg cases spanning B200 / GB200 / GB300 and
deepseek-r1 / deepseek-v32 / gpt-oss-120b / qwen3-235b / glm-5 (including the
exact nvbugs/6487040 diagnostic case, gb200_deepseek-v32-fp4_8k1k_con1_...).

The 10 cases map to 5 post-merge Disagg-PerfSanity stages (these stages shard
1-test-per-shard by runtime duration, so all shards of each stage are listed to
guarantee the picked cases run):

DGX_B200-16_GPUs-2_Nodes-...-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge

disagg_upload-gen_only-b200_deepseek-r1-fp4_8k1k_con256_ctx1_dep4_gen1_dep8_eplb0_mtp1_ccb-NIXL
disagg_upload-gen_only-b200_deepseek-r1-fp4_8k1k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL

GB200-8_GPUs-2_Nodes-...-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge
3. disagg_upload-gen_only-gb200_gpt-oss-120b-fp4_8k1k_con512_ctx1_tp1_gen1_dep2_eplb0_mtp0_ccb-NIXL

GB200-8_GPUs-2_Nodes-...-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge
4. disagg_upload-gen_only-gb200_gpt-oss-120b-fp4_8k1k_con128_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL
5. disagg_upload-gen_only-gb200_qwen3-235b-fp4_8k1k_con64_ctx1_tp1_gen1_tep4_eplb0_mtp0_ccb-NIXL

GB200-12_GPUs-3_Nodes-...-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge (bug's own family)
6. disagg_upload-gen_only-gb200_deepseek-v32-fp4_8k1k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL ← nvbugs/6487040
7. disagg_upload-gen_only-gb200_deepseek-v32-fp4_32k4k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL
8. disagg_upload-gen_only-gb200_deepseek-r1-fp4_8k1k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL

GB300-12_GPUs-3_Nodes-...-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge
9. disagg_upload-gen_only-gb300_glm-5-fp4_8k1k_con1_ctx1_dep2_gen1_tep8_eplb0_mtp3_ccb-NIXL
10. disagg_upload-gen_only-gb300_glm-5-fp4_8k1k_con1024_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL

/bot run --stage-list "DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-1,DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-3,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-4,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-5,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge-1,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge-2"

chenfeiz0326 · 2026-07-22T09:51:00Z

/bot run --stage-list "DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-1,DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-3,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-4,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-5,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge-1,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge-2"

tensorrt-cicd · 2026-07-22T09:57:30Z

PR_Github #60958 [ run ] triggered by Bot. Commit: abe8454 Link to invocation

tensorrt-cicd · 2026-07-22T10:26:56Z

PR_Github #60958 [ run ] completed with state FAILURE. Commit: abe8454
/LLM/main/L0_MergeRequest_PR pipeline #49221 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chenfeiz0326 · 2026-07-22T14:11:24Z

/bot run --disable-fail-fast --stage-list "DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-1,DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU2-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-3,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-4,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-5,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge-1,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge-2"

tensorrt-cicd · 2026-07-22T14:17:56Z

PR_Github #60997 [ run ] triggered by Bot. Commit: 7a43ab8 Link to invocation

tensorrt-cicd · 2026-07-22T15:33:33Z

PR_Github #60997 [ run ] completed with state SUCCESS. Commit: 7a43ab8
/LLM/main/L0_MergeRequest_PR pipeline #49256 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

fredricz-20070104

Approve.

chenfeiz0326 · 2026-07-23T09:41:31Z

/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-3,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-4,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-5,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-6,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-7,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-8"

tensorrt-cicd · 2026-07-23T09:47:00Z

PR_Github #61272 [ run ] triggered by Bot. Commit: a30c829 Link to invocation

tensorrt-cicd · 2026-07-23T11:22:04Z

PR_Github #61272 [ run ] completed with state FAILURE. Commit: a30c829
/LLM/main/L0_MergeRequest_PR pipeline #49507 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chenfeiz0326 · 2026-07-23T11:49:49Z

/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-4,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-8,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-5,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3"

tensorrt-cicd · 2026-07-23T11:55:53Z

PR_Github #61284 [ run ] triggered by Bot. Commit: 91be512 Link to invocation

tensorrt-cicd · 2026-07-23T12:31:56Z

PR_Github #61284 [ run ] completed with state FAILURE. Commit: 91be512
/LLM/main/L0_MergeRequest_PR pipeline #49518 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

The disagg gen_only perf-sanity regression metric mean_gen_worker_per_iter_device_step_time is parsed from each gen_server_{i}.log by parse_gen_worker_device_step_time(). Since NVIDIA#16298 introduced steady-state ngen bucketing, a gen worker whose prev_device_step_time lines are all present and numeric can still yield a None metric if num_generation_tokens fails to match _NUM_GEN_TOKENS_RE on every row (absent from the states dict, or a non-bare-int render). A None metric then trips PerfSanityTestConfig.check_test_failure() with a misleading "missing 'prev_device_step_time'" RuntimeError even though the data was logged and the workload passed (nvbugs 6487036 / 6487040). Fix: _scan_gen_worker_device_step_time now also accumulates a per-file un-bucketed all-iter Welford mean over every iter>=5 numeric prev_device_step_time row. _mean_at_mode_ngen prefers the steady-state mode-ngen bucket (unchanged for healthy logs) but falls back to that all-iter mean for any worker whose buckets are empty, so a present metric is never dropped to None. The fallback path logs a print_info. Returns None only when no usable row exists in any worker at all. Verified behavior-preserving on a real passing gen_server_0.log (v32 gen_only mtp3 on lyris GB200: 12.91 ms before and after). Adds test_gen_worker_device_step_time_parser.py (10 CPU-only cases covering the mode-bucket path, iter<5 / N/A skips, the missing / unparsable num_generation_tokens fallback, mixed-worker averaging, and the None cases). Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

… test - Fix loader docstring: it reads a fixed source slice and exec()s it; it does not use importlib. Document the trusted, repository-local path and add a narrowly-scoped `# noqa: S102` for the intentional exec. - Strengthen test_scan_counts_all_usable_rows_regardless_of_ngen: use distinct step times (10.0 bucketed vs 99.0 fallback-only) and assert _mean_at_mode_ngen() returns the bucket mean, so the test can no longer pass if aggregation wrongly selects the all-iter mean when a bucket exists. Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

…f-sanity cases Remove the SKIP waivers for the two disagg gen_only NIXL perf-sanity cases now that the device-step-time parser fix is in place, so CI validates them: - gb200 deepseek-v32-fp4 8k1k con1 ...tep8 eplb0 mtp3 (nvbugs/6487040) - gb200 deepseek-v32-fp4 8k1k con4096...dep32 eplb256 mtp0 (nvbugs/6487036) Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

… bot rerun Un-waive the 3 still-waived affected cases from nvbug 6487036 so the /bot run can validate the prev_device_step_time parser fix: - disagg_upload-gen_only-gb200_deepseek-r1-fp4_8k1k_con1024_ctx1_dep4_gen1_dep32_eplb0_mtp3_ccb-NIXL - disagg_upload-gen_only-gb200_deepseek-r1-fp4_8k1k_con1_ctx1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL - disagg_upload-gen_only-gb200_gpt-oss-120b-fp4_8k1k_con128_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

chenfeiz0326 · 2026-07-24T04:38:34Z

/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-1,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-5,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1"

tensorrt-cicd · 2026-07-24T04:44:29Z

PR_Github #61501 [ run ] triggered by Bot. Commit: daa4def Link to invocation

chenfeiz0326 requested review from a team as code owners July 22, 2026 08:52

chenfeiz0326 requested review from EmmaQiaoCh, LarryXFly, ZhanruiSunCh, brnguyen2 and schetlur-nv July 22, 2026 08:52

github-actions Bot assigned chenfeiz0326 Jul 22, 2026

coderabbitai Bot reviewed Jul 22, 2026

View reviewed changes

Comment thread tests/integration/defs/perf/test_gen_worker_device_step_time_parser.py Outdated

Comment thread tests/integration/defs/perf/test_gen_worker_device_step_time_parser.py Outdated

chenfeiz0326 requested a review from a team as a code owner July 22, 2026 14:05

chenfeiz0326 requested a review from JunyiXu-nv July 22, 2026 14:05

chenfeiz0326 requested a review from fredricz-20070104 July 23, 2026 02:51

fredricz-20070104 approved these changes Jul 23, 2026

View reviewed changes

chenfeiz0326 added 2 commits July 24, 2026 12:13

chenfeiz0326 force-pushed the fix/gen-only-device-step-time-parse-none-6487040 branch from 91be512 to daa4def Compare July 24, 2026 04:13

Uh oh!

Conversation

chenfeiz0326 commented Jul 22, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

Uh oh!

coderabbitai Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

chenfeiz0326 commented Jul 22, 2026

Uh oh!

chenfeiz0326 commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

chenfeiz0326 commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

fredricz-20070104 left a comment

Choose a reason for hiding this comment

Uh oh!

chenfeiz0326 commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

chenfeiz0326 commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

chenfeiz0326 commented Jul 24, 2026

Uh oh!

tensorrt-cicd commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chenfeiz0326 commented Jul 22, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 22, 2026 •

edited

Loading