Why it matters

All-reduce is data-parallel workhorse. NCCL picks by size + topology.

Advertisement

The architecture

Ring: bandwidth-optimal.

Tree: latency-optimal.

NCCL picks.

All-reduce algorithmsRingbandwidth optimalTreelatency optimalNCCLpicks by sizeNCCL 2.x auto-selects; Reduce-Scatter + All-Gather composition equivalent
All-reduce algorithms.
Advertisement

How it works end to end

NCCL auto-select.

Ring + tree + DBT.

RS + AG composition.