Core concept
Bandwidth-optimal: total transfer = 2(N-1)/N × size. Bandwidth-bound for large messages.
Advertisement
How it works
Latency scales with N (2N-1 steps). Poor for small messages + many workers.
Advertisement
Trade-offs + gotchas
Data parallel training. Large gradient transfers. Standard NCCL default.