SM90 (Hopper) FP4 MegaMoE fused kernel with swapAB small-batch path - #53

Open

qiushixiaoyu wants to merge 22 commits into

sgl-project:devfrom

qiushixiaoyu:fp4_swapAB

Commits on Jul 19, 2026

Add SM90 FP4 MegaMoE implementation
yinding
committed
7db2517
Avoid L1 N32 swapAB bucket for FP4 MegaMoE
yinding
committed
4bbfb32
FP4 swapAB L1: align routing-weight fold order with non-swap path

yinding
and
claude
committed
61a5e24
Fix SM90 FP4 MegaMoE build against post-#364 signatures
yinding
committed
33f04a8
Simplify SM90 FP4 MegaMoE heuristics to FP8 parity

yinding
and
claude
committed
128b6f0
Raise SM90 FP4 MegaMoE decode->prefill boundary to e=80

yinding
and
claude
committed
250f479
Use continuous FP32 activation scale in SM90 FP4 MegaMoE

yinding
and
claude
committed
7a39245

Commits on Jul 28, 2026

Tune SM90 MegaMoE decode heuristics for GLM5.2
yinding
committed
d38fc27

Commits on Jul 29, 2026

Add SM90 FP4 MegaMoE SiTU support for Kimi K3
yinding
committed
c4bdb56
Use fast tanh identities for SM90 FP4 MegaMoE SiTU fast-math

yinding
and
claude
committed
9f6136b
Load FP4 MegaMoE activation scales with a single 2D TMA

yinding
and
claude
committed
79d3b53
Default SiTU to 128/64 activation scales with e=512 prefill boundary

yinding
and
claude
committed
8727cc2
Enable swapAB for SiTU with the 128/64 recipe

yinding
and
claude
committed
8bb2eef

Commits on Jul 30, 2026

Add SiTU 128/64 prefill mid band for e in (80, 112]

yinding
and
claude
committed
5e81045
Calibrate SiTU prefill mid-band lower bound to e=76

yinding
and
claude
committed
c4d22c8
Raise FP4 swapAB default bound to e=20

yinding
and
claude
committed
532422a
Select SiTU 128/64 config via an analytic cost model

yinding
and
claude
committed
adb9b51
Add experimental FP4 tuning env overrides from small-batch probing

yinding
and
claude
committed
a933155
Hoist tile-invariant dispatch out of the math K-stage loop

yinding
and
claude
committed
28e3ad9
Add experts-per-wave probe env from 512+ tuning sweep

yinding
and
claude
committed
11facff
Add normal-path stage breakdown and wide-load-decode probe env

yinding
and
claude
committed
9fcbd52

Commits on Jul 31, 2026

Remove dead experiment probe envs

yinding
and
claude
committed
aee79a7