[CPU] Merge MoE ±bias tests via bias=[False, True] - #94
Draft
mingfeima wants to merge 5 commits into
Draft
Conversation
make_routing draws a fresh score, so ref (from the original score) and the kernel no longer selected the same experts. Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
Casting scaled_weight() to bf16 truncates relative to the FP8 kernel's float accumulation and breaks test_fp8_moe_bias at larger shapes (e.g. m=32, n=k=128). Match main's test_fp8_moe and keep float32 dequant weights in the reference path. Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
- Add run_fused_experts / make_routing helpers to drop None arg walls - Merge bf16/fp8/mxfp4 non-bias and bias tests into one parametrized case - bias=True also covers clamped SwiGLU (alpha/limit); bias=False keeps silu Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
test_*_moe/test_*_moe_biasinto single parametrized cases withbias=[False, True].run_fused_experts,make_routing) to cut boilerplate in CPU MoE tests.Bug fixes
test_int4_moe: must keep the same routing for ref and kernel. Replacing softmax/topk of the sharedscorewithmake_routing(...)drew a fresh score and broke the comparison.test_fp8_moe/test_fp8_moe_bias: do not castscaled_weight()to bf16 before the reference. That truncates vs the FP8 kernel’s float accumulate and fails at larger shapes such asm=32, n=k=128(test_fp8_moe_bias[False-2-4-128-128-32]). Keep float32 dequant weights for the ref (same as main’stest_fp8_moe).Test plan
test/registered/cpu/test_moe.py(int4 + fp8 ± bias)CI States
Latest PR Test (Base): ❌ Run #29483941434
Latest PR Test (Extra): ❌ Run #29483941266