Fix ring all-reduce in-place corruption at gpu_num ≥ 3#253
Merged
Conversation
added 2 commits
June 9, 2026 01:31
…pying input into a private buffer before the loop.
…ct buffer aliasing and copy input into a private tensor before the ring loop.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #253 +/- ##
==========================================
- Coverage 86.88% 86.78% -0.10%
==========================================
Files 125 127 +2
Lines 6113 5993 -120
==========================================
- Hits 5311 5201 -110
+ Misses 802 792 -10 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
added 2 commits
June 9, 2026 04:41
Resolve conflict in ark/ops/ops_test_common.cpp: keep both ARK_UT_CORRECTNESS env-var gating (#251) and GPU-count skip guard.
…lace-fix) against main; conflict is in ark/ops/ops_test_common.cpp — preserve both P2's UT-gating logic and P4's GPU-count skip logic.
- ops_test_common.cpp:63: join LOG args onto one line (fits in 80 cols) - unittest_utils.h:248-253: re-align backslashes in UNITTEST_SKIP macro
…e-inplace-fix): ops_test_common.cpp:63 and unittest_utils.h:248-253.
…n branch pr-c-allreduce-inplace-fix.
…n branch pr-c-allreduce-inplace-fix.
…esolving any conflicts; cause: BEHIND (behind_by=1).
…esolving any conflicts; cause: BEHIND (behind_by=1).
Alphabetize #include directives to satisfy Google-style clang-format. Fixes the linters CI failure at line 4:1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix ring all-reduce in-place corruption at gpu_num ≥ 3
In-place ring all-reduce mutates
inputbefore the next iteration'ssend when
outputaliasesinput. At TP=2 the single iteration hidesthe bug; at gpu_num ≥ 3 every subsequent send carries stale data.
Fix: detect buffer aliasing (
output.ref()->buffer()->id() == input.ref()->buffer()->id()) and copyinputinto a private tensorbefore the ring loop.
Also sorted includes in
ops_all_reduce_test.cppto satisfyclang-format.