MoE Load Balancing

Why it matters

Load balance shapes MoE quality + efficiency. Understanding shapes training.

The architecture

Auxiliary loss: expert usage variance.

Capacity factor: cap tokens per expert.

How it works end to end

Load-balance loss (Switch Transformer).

Capacity factor 1.0-1.25 typical.

DeepSeek: bias-based auxiliary-free.