Why it matters

MoE kernels shape efficiency. Understanding shapes serving.

Advertisement

The architecture

Fused kernels: router + gather + expert + scatter.

Grouped GEMM for experts.

MoE fused kernelsFused router + gatherone kernelGrouped GEMMvariable batchesScatter resultscombinecuBLAS grouped GEMM + FlashAttention-MoE variants + Triton custom
MoE kernels.
Advertisement

How it works end to end

Grouped GEMM: cuBLAS + cuTENSOR.

Fused router: Triton.

Scatter-combine kernels.