The kernel fusion process currently relies on a set number of operations that can easily be fused (eg, elementwise ops) or kernels that have compatible workgroup shapes. We should almost always be able to merge operations with compatible workgroup shapes through coercion but it may not be efficient if one has a much larger dispatch size. The layer norm benchmark is a good example of this behavior. It should be merged into 1-2 kernels but is currently several more
Long term, it would be better to structure the fusion process in terms of graph rewrites with something like egg
The kernel fusion process currently relies on a set number of operations that can easily be fused (eg, elementwise ops) or kernels that have compatible workgroup shapes. We should almost always be able to merge operations with compatible workgroup shapes through coercion but it may not be efficient if one has a much larger dispatch size. The layer norm benchmark is a good example of this behavior. It should be merged into 1-2 kernels but is currently several more
Long term, it would be better to structure the fusion process in terms of graph rewrites with something like egg