feat(moe): support sonic-moe for small model training#12

hscos wants to merge 3 commits into

sirl-stablescitix/Megatron-LM:sirl-stablefrom

feature/support-sonic-moescitix/Megatron-LM:feature/support-sonic-moeCopy head branch name to clipboard