Why it matters

MoE serving requires different infra. Understanding shapes modern LLM.

Advertisement

The architecture

Params massive (100B+).

Sparse activation (5-15% per token).

Routing + expert compute + combine.

MoE serving needsMassive params100B-1TSparse activationtop-K expertsRouting + all-to-allexpert parallelismvLLM + TRT-LLM + SGLang all growing MoE support in 2025
MoE serving.
Advertisement

How it works end to end

Memory: hold all experts.

Compute: only top-K active.

Communication: all-to-all for routing.