Why it matters

All-to-all shapes MoE latency. Understanding shapes fabric requirements.

Advertisement

The architecture

Send phase: token -> expert-owning GPU.

Compute at expert.

Return phase.

MoE all-to-allSend tokensto expert GPUsComputeexpert forwardReturn outputsback to originNVLink intra-node + IB across; fastest fabric essential
All-to-all.
Advertisement

How it works end to end

NVLink intra-node + IB across.

Fused kernels reduce overhead.

DeepEP: DeepSeek's all-to-all lib.