Problem
Piper currently uses TorchInductor under the hood to compile model code into optimized Triton kernels, but there is room to get better performance by doing better kernel generation or leveraging existing highly-tuned kernels like Megatron's hand-tuned transformer and MoE kernels.
Expected behavior
An interface or automatic solution for generating or substituting in highly-tuned kernels in for chunks of the model.
Problem
Piper currently uses TorchInductor under the hood to compile model code into optimized Triton kernels, but there is room to get better performance by doing better kernel generation or leveraging existing highly-tuned kernels like Megatron's hand-tuned transformer and MoE kernels.
Expected behavior
An interface or automatic solution for generating or substituting in highly-tuned kernels in for chunks of the model.