[https://nvbugs/6450338][fix] Default to FP32 accumulation in allreduce fusion kernels by dongfengy · Pull Request #16753 · NVIDIA/TensorRT-LLM

dongfengy · 2026-07-22T20:06:45Z

Description

Fixes the intermittent GPT-OSS 120B accuracy failures on DGX-B200 tracked in https://nvbugs/6450338 (TestGPTOSS::test_w4_4gpus[v1_kv_cache-ep4-trtllm-fp8], GPQA 55.051 < 55.734).

The custom oneshot/twoshot allreduce fusion kernels accumulate in the native 16-bit dtype by default (use_fp32_acc() is off unless ALL_REDUCE_FUSION_KERNEL_ACC_FP32 is set). The autotuner (allreduce_strategy=AUTO) picks tactics on speed alone, so machine state during profiling decides whether the eval runs on the bf16-accumulating ONESHOT/TWOSHOT kernels or on fp32-accumulating NCCL variants. On models with large residual-stream outliers (GPT-OSS 120B, residual values up to ~75k where bf16 ulp is 512) this is a ~5-point GPQA swing, and the CI failure is the tail of that lottery.

This PR flips the default to FP32 accumulation, matching NCCL and the newer MNNVL kernels (which accumulate in fp32 unconditionally). ALL_REDUCE_FUSION_KERNEL_ACC_FP32=0 opts out and is bit-identical to the legacy behavior.

Measurements (all on B200 tp4, autotuner cache pinned so only the accumulation dtype changes):

At the failing commit 696904a6d387: GPQA 62.121 → 65.657 with fp32 accumulation on the ONESHOT assignment (bit-stable across repeats); GSM8K 89.992 → 90.675.
On current main: GPQA identical to the bf16 default in 6/6 pinned A/B pairs (2 machines × 3 autotuner-cache states); GSM8K 90.296 → 89.992 (−0.304, i.e. 4/1319 questions, constant across all pairs).
Perf: mpirun -n 4 python tests/microbenchmarks/all_reduce.py --benchmark A/B (fp32 vs ALL_REDUCE_FUSION_KERNEL_ACC_FP32=0, same machine, back-to-back): across all 36 rows that hit the modified kernels (ONESHOT/TWOSHOT × 4 fusion patterns × 256B–256MB), fp32 ranges from 4% faster to 2% slower (average ~1% faster), inside the ±3% run-to-run spread of the untouched strategies (NCCL/NCCL_SYMMETRIC/MNNVL/UB). No perf regression.

Test Coverage

accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v1_kv_cache-ep4-trtllm-fp8] (the failing test from the bug), run pinned on two B200 machines × three autotuner-cache states.
tests/microbenchmarks/all_reduce.py --benchmark for the perf A/B above.

PR Checklist

PR title and description follow the repo conventions
Test coverage listed above
Opt-out env var preserves the legacy behavior bit-exactly

🤖 Generated with Claude Code

Dev Engineer Review

Updated use_fp32_acc() in cpp/tensorrt_llm/kernels/communicationKernels/allReduceFusionKernels.cu so FP32 accumulation is now the default.
FP32 accumulation is used when ALL_REDUCE_FUSION_KERNEL_ACC_FP32 is unset or its first character is not '0'; setting ALL_REDUCE_FUSION_KERNEL_ACC_FP32=0 restores the legacy behavior (native-dtype accumulation).
Change is scoped to FP32-accumulation dispatch for allreduce fusion kernels; no public APIs/exported declarations were modified.
Expected impact: improves accuracy for GPT-OSS-like accuracy failures tied to autotuner selection between BF16-accumulating fusion kernels and FP32-accumulating NCCL variants, with no reported performance regression for the tested modified kernels/microbenchmarks.

QA Engineer Review

No test changes.

coderabbitai · 2026-07-22T20:08:57Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a9058aea-cbd5-4c5a-bf4b-d587b5667b91

📥 Commits

Reviewing files that changed from the base of the PR and between 861456c and b4be9b9.

📒 Files selected for processing (1)

cpp/tensorrt_llm/kernels/communicationKernels/allReduceFusionKernels.cu

🚧 Files skipped from review as they are similar to previous changes (1)

cpp/tensorrt_llm/kernels/communicationKernels/allReduceFusionKernels.cu

Walkthrough

FP32 accumulation in the all-reduce fusion kernel is now enabled by default. Setting ALL_REDUCE_FUSION_KERNEL_ACC_FP32 to a value beginning with 0 disables it.

Changes

FP32 accumulation control

Layer / File(s)	Summary
Update FP32 accumulation selection `cpp/tensorrt_llm/kernels/communicationKernels/allReduceFusionKernels.cu`	`use_fp32_acc()` enables FP32 accumulation when the environment variable is unset or does not begin with `0`; a leading `0` disables it.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Suggested reviewers: zongfeijing

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title follows the repository’s required ticket-and-type format and accurately summarizes the FP32 accumulation change.
Description check	✅ Passed	The description includes the issue, solution, test coverage, and a relevant checklist, so it is mostly complete.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

🧹 Nitpick comments (1)

cpp/tensorrt_llm/kernels/communicationKernels/allReduceFusionKernels.cu (1)
720-727: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Const-qualify the cached environment pointer.

fp32_acc is never reassigned and its contents are only inspected. Use char const* const to enforce const-correctness and east-const style.

As per coding guidelines: “Prefer const or constexpr over macros, declare unmodified variables const, and use east-const style.”
Proposed fix
-    static char* fp32_acc = std::getenv("ALL_REDUCE_FUSION_KERNEL_ACC_FP32");
+    static char const* const fp32_acc = std::getenv("ALL_REDUCE_FUSION_KERNEL_ACC_FP32");
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cpp/tensorrt_llm/kernels/communicationKernels/allReduceFusionKernels.cu`
around lines 720 - 727, Update the cached environment pointer in use_fp32_acc()
to char const* const, preserving the existing getenv call and return logic.
Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@cpp/tensorrt_llm/kernels/communicationKernels/allReduceFusionKernels.cu`:
- Around line 720-727: Update the cached environment pointer in use_fp32_acc()
to char const* const, preserving the existing getenv call and return logic.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1335b1b2-c5ef-42ee-b41e-5fadeb00b6aa

📥 Commits

Reviewing files that changed from the base of the PR and between b294868 and 4db94a3.

📒 Files selected for processing (1)

cpp/tensorrt_llm/kernels/communicationKernels/allReduceFusionKernels.cu

dongfengy · 2026-07-22T20:10:06Z

/bot run

tensorrt-cicd · 2026-07-22T20:16:01Z

PR_Github #61078 [ run ] triggered by Bot. Commit: 4db94a3 Link to invocation

tensorrt-cicd · 2026-07-22T20:53:50Z

PR_Github #61078 [ run ] completed with state FAILURE. Commit: 4db94a3
/LLM/main/L0_MergeRequest_PR pipeline #49332 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

dongfengy · 2026-07-22T20:56:36Z

/bot run

dongfengy · 2026-07-22T23:15:49Z

/bot run

tensorrt-cicd · 2026-07-22T23:21:28Z

PR_Github #61112 [ run ] triggered by Bot. Commit: 4db94a3 Link to invocation

tensorrt-cicd · 2026-07-22T23:40:37Z

PR_Github #61112 [ run ] completed with state SUCCESS. Commit: 4db94a3
/LLM/main/L0_MergeRequest_PR pipeline #49366 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

dongfengy · 2026-07-22T23:43:00Z

/bot run

tensorrt-cicd · 2026-07-22T23:49:32Z

PR_Github #61118 [ run ] triggered by Bot. Commit: 4db94a3 Link to invocation

tensorrt-cicd · 2026-07-23T00:18:19Z

PR_Github #61118 [ run ] completed with state SUCCESS. Commit: 4db94a3
/LLM/main/L0_MergeRequest_PR pipeline #49372 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

dongfengy · 2026-07-23T00:24:29Z

/bot run

tensorrt-cicd · 2026-07-23T00:33:28Z

PR_Github #61127 [ run ] triggered by Bot. Commit: 4db94a3 Link to invocation

dongfengy · 2026-07-23T00:34:46Z

/bot kill

dongfengy · 2026-07-23T00:35:07Z

/bot run

tensorrt-cicd · 2026-07-23T00:40:36Z

PR_Github #61132 [ kill ] triggered by Bot. Commit: 861456c Link to invocation

tensorrt-cicd · 2026-07-23T00:41:52Z

PR_Github #61134 [ run ] triggered by Bot. Commit: 861456c Link to invocation

tensorrt-cicd · 2026-07-23T00:41:55Z

PR_Github #61132 [ kill ] completed with state ABORTED. Commit: 861456c

Link to invocation

tensorrt-cicd · 2026-07-23T00:42:19Z

PR_Github #61127 [ run ] completed with state ABORTED. Commit: 4db94a3

Link to invocation

tensorrt-cicd · 2026-07-23T00:57:41Z

PR_Github #61134 [ run ] completed with state FAILURE. Commit: 861456c
/LLM/main/L0_MergeRequest_PR pipeline #49386 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

dongfengy · 2026-07-23T03:39:54Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-23T03:46:06Z

PR_Github #61186 [ run ] triggered by Bot. Commit: 861456c Link to invocation

tensorrt-cicd · 2026-07-23T13:58:16Z

PR_Github #61186 [ run ] completed with state FAILURE. Commit: 861456c
/LLM/main/L0_MergeRequest_PR pipeline #49435 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…ce fusion kernels The custom oneshot/twoshot allreduce fusion kernels accumulate in the native 16-bit dtype by default (use_fp32_acc: off unless ALL_REDUCE_FUSION_KERNEL_ACC_FP32 is set). On models with large residual-stream outliers (GPT-OSS 120B) this costs measurable accuracy whenever the autotuner selects the ONESHOT/TWOSHOT tactics over NCCL variants: with identical pinned autotuner tactics, enabling FP32 accumulation recovers +3.5 GPQA points end-to-end (62.121 -> 65.657). Flip the default to FP32 accumulation; opt out with ALL_REDUCE_FUSION_KERNEL_ACC_FP32=0 (legacy behavior, bit-preserved). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>

dongfengy · 2026-07-24T02:38:09Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-24T02:46:31Z

PR_Github #61473 [ run ] triggered by Bot. Commit: b4be9b9 Link to invocation

tensorrt-cicd · 2026-07-24T06:11:03Z

PR_Github #61473 [ run ] completed with state SUCCESS. Commit: b4be9b9
/LLM/main/L0_MergeRequest_PR pipeline #49694 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

dongfengy · 2026-07-24T06:59:46Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-24T07:06:01Z

PR_Github #61528 [ run ] triggered by Bot. Commit: b4be9b9 Link to invocation

tensorrt-cicd · 2026-07-24T11:29:26Z

PR_Github #61528 [ run ] completed with state SUCCESS. Commit: b4be9b9
/LLM/main/L0_MergeRequest_PR pipeline #49743 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

dongfengy requested a review from a team as a code owner July 22, 2026 20:06

dongfengy requested a review from zongfeijing July 22, 2026 20:06

github-actions Bot assigned dongfengy Jul 22, 2026

coderabbitai Bot reviewed Jul 22, 2026

View reviewed changes

dongfengy force-pushed the fix/allreduce-fp32-default-6450338 branch from 4db94a3 to 861456c Compare July 23, 2026 00:34

dongfengy force-pushed the fix/allreduce-fp32-default-6450338 branch from 861456c to b4be9b9 Compare July 24, 2026 02:35

Uh oh!

Conversation

dongfengy commented Jul 22, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

Uh oh!

coderabbitai Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

dongfengy commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

dongfengy commented Jul 22, 2026

Uh oh!

dongfengy commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

dongfengy commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

dongfengy commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

dongfengy commented Jul 23, 2026

Uh oh!

dongfengy commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

dongfengy commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

dongfengy commented Jul 24, 2026

Uh oh!

tensorrt-cicd commented Jul 24, 2026

Uh oh!

tensorrt-cicd commented Jul 24, 2026

Uh oh!

dongfengy commented Jul 24, 2026

Uh oh!

tensorrt-cicd commented Jul 24, 2026

Uh oh!

tensorrt-cicd commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dongfengy commented Jul 22, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 22, 2026 •

edited

Loading