Why it matters

Expert choice avoids load balance loss. Understanding shapes MoE options.

Advertisement

The architecture

Compute affinity matrix.

Each expert selects top-K tokens.

Perfect load balance built-in.

Expert choice routingAffinity matrixtokens x expertsExpert pickstop-K tokensPerfect balanceno aux lossZhou et al. 2022; auto load balance; some tokens may not be picked
Expert choice.
Advertisement

How it works end to end

Zhou et al. 2022.

No load balance loss needed.

Some tokens under-served (padded).