merge initial public release into main - #12
Merged
Merged
Conversation
… a backward_utils.py file
…ient sync. llama experiment with PP=1 and DP=2 shows that comm is entirely overlapped with backward
…_insert_comm_ops that inserts a2a comm ops detected by user annotations with 'collective'='all_to_all_single'. Also changed stage boundary API to be another graph annotation (possible under new assumption that there are no graph breaks). Now, entire model is distributed in one invocation of the piper backend, which transforms the graph using _inser_comm_ops and _split_gm_by_stages. Need to implement priority scheduling to avoid interference between synchronous a2a EP ops and asynchronous allreduce DP ops.
…ug: a2a and p2p comm in test_mixtral.py is slower than expected, also running multiple iterations triggers a CUDA device side assert
This reverts commit 6b27bd3.
This reverts commit cc6dda2.
This reverts commit c506131.
Signed-off-by: Andy Ruan <andyubryh@gmail.com>
…ce sync bug: wait on an event after completing bwd for all microbatches to make sure allreduce isn't launched until bwd compute completes
…nified commuincation/computation task DAG given the following inputs: pipeline-parallel schedule represented as a 2D array, pytorch model with annotations for EP dispatch/combine and optional annotations for pipeline stage boundaries, desired PP and DP/EP degrees, bucketing flag. The controller sends to each actor a DAG of tasks to execute and the model weights / computation graph for each compute task that the actor runs. The actor executes the DAG tasks in topological order and is agnostic to any scheduling decisions
…0 forward/backward compute times
…n 2 per pp rank pair. also limit model compilation to a single dp rank in the case DP > 1. other changes reflect deepseek overlapping implementation: currently assumes comp-comm pattern matches for overlapping task chains
…ical time steps instead. refactor PiperActor so that all intermediate storage is managed by run_dag, and the actual compute/communication methods operate on data passes as inputs
…am buffer back to single buffer per trainable param to avoid cuda memory bug. implement optional a2a_ar_no_overlap to avoid EP/DP interfrence: orders ar operations after all a2a complete, scheduled on same stream to avoid interference. add memory debug logs.
… rather than per-stage. fix slow startup time by pipelinine piper backend work across DP ranks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.