Problem
Piper does not currently support context or tensor parallelism.
Expected Behavior
Piper's scheduling language, scheduling backend, and runtime should be extended to express CP/TP schedules, insert the appropriate collectives into the TrainingDAG, and manage the necessary sharding and collectives at runtime.
Problem
Piper does not currently support context or tensor parallelism.
Expected Behavior
Piper's scheduling language, scheduling backend, and runtime should be extended to express CP/TP schedules, insert the appropriate collectives into the
TrainingDAG, and manage the necessary sharding and collectives at runtime.