You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
You might also be interested in this PR which updates MosaicBERT to FlashAttention 2 and removes all the issues with the custom Triton FlashAttention + ALiBi implementation.
Hi @Zhihan1996,
You might also be interested in this PR which updates MosaicBERT to FlashAttention 2 and removes all the issues with the custom Triton FlashAttention + ALiBi implementation.