Why it matters
Dropless MoE preserves data. Understanding shapes modern MoE.
Advertisement
The architecture
Standard MoE drops overflow.
Dropless: use variable-batch expert kernels.
Sparse GEMM kernels.
Advertisement
How it works end to end
MegaBlocks: sparse matmul.
Grouped GEMM for variable batches.
No capacity waste.