Why it matters

Collectives shape distributed training. Understanding shapes design.

Advertisement

The architecture

All-reduce: sum across all.

All-gather: concatenate from all.

Reduce-scatter: reduce + distribute.

Collective opsAll-reduceDDP gradsAll-gatherFSDP paramsReduce-scatterFSDP gradsAll-reduce = reduce-scatter + all-gather; FSDP splits phases for efficiency
Collectives.
Advertisement

How it works end to end

Ring: bandwidth-optimal.

Tree: latency-optimal.

All-reduce = reduce-scatter + all-gather.

Bandwidth: 2*(N-1)/N * data size.