Once a model no longer fits — or trains too slowly — on one device, you split it across many, and the moment you do, the devices have to talk. That conversation is not incidental overhead; for large training runs the time spent moving tensors between GPUs can rival the time spent computing on them. The vocabulary of that conversation is a small set of collective communication primitives — all-reduce, all-gather, reduce-scatter, broadcast, all-to-all — and the economics are captured by a two-parameter cost model and one beautiful algorithm, the ring all-reduce, whose cost is provably bandwidth-optimal. This article is the shared foundation that the parallelism articles (data, tensor, pipeline, and expert parallelism) all lean on: what the primitives do, what they cost, and which strategy summons which collective.
Why collectives, not point-to-point
A collective is a communication operation that all participating devices enter together and leave together, each contributing and receiving data according to a fixed pattern. You could, in principle, wire up distributed training with raw point-to-point send/recv calls, but you would be reinventing — badly — operations that libraries like NCCL, RCCL, and MPI already implement with topology-aware, bandwidth-optimal algorithms.
The reason collectives matter for transformer training is that gradients, activations, and parameters are replicated or sharded across devices, and keeping them consistent requires structured all-to-many exchanges, not one-off messages. When 512 GPUs each finish a backward pass, every one holds a different gradient for the same weight; they must be summed and the result handed back to all 512. That is a single named operation — all-reduce — and the whole art of scaling is choosing algorithms whose cost grows slowly, ideally not at all, as the device count N climbs. Understanding the primitives and their costs is what separates a run that scales to hundreds of devices from one that stalls at eight.
The five primitives you need
Five collectives cover essentially all of distributed training. Broadcast copies one device’s buffer to every device (used to distribute initial weights). Reduce combines all devices’ buffers with an operator (usually sum) and leaves the result on one device. All-reduce does the same reduction but leaves the result on every device — the workhorse of data parallelism.
All-gather concatenates each device’s shard so every device ends up holding all shards (if device i holds chunk x_i, afterwards all hold [x_0, x_1, …, x_{N-1}]). Reduce-scatter is its mirror image: it sums the corresponding pieces across devices but scatters the result so device i keeps only the i-th reduced chunk. All-to-all is a full transpose — every device sends a distinct piece to every other device, so the j-th block on device i lands as the i-th block on device j. Keep these shapes in mind: the rest of the article is really just accounting for how many bytes each one moves.
The alpha-beta cost model
To predict communication time we use the classic alpha-beta model. The time to send a message is split into a fixed latency term and a per-byte bandwidth term:
T(message) = α + β · M
α = latency per message (seconds, fixed startup)
β = 1 / bandwidth (seconds per byte)
M = message size (bytes)
For an algorithm of H sequential steps:
T = α · H + β · (bytes moved on the critical path)The two terms compete. Small messages are latency-bound — dominated by α · H, so you want few hops. Large messages are bandwidth-bound — dominated by β · bytes, so you want to minimize total volume, even at the cost of more steps. Gradient buffers in training are large (megabytes to gigabytes), so training collectives live firmly in the bandwidth-bound regime, and the design goal becomes: move the fewest bytes possible per device, independent of how many hops that takes. This single observation is what makes the ring algorithm the right answer.
Ring all-reduce: the algorithm
Arrange the N devices in a logical ring, each with a left and right neighbor. Split every device’s buffer into N equal chunks. The all-reduce runs in two phases of N-1 steps each.
Phase 1 — reduce-scatter. On each step, every device sends one chunk to its right neighbor and receives one from its left, adding the incoming chunk into its own. Chunks flow around the ring, accumulating sums as they go. After N-1 steps, each device holds exactly one chunk that is the fully reduced sum for that slice — but a different slice on each device. Phase 2 — all-gather. Now the finished chunks circulate the same way, but this time devices overwrite rather than add, so after another N-1 steps every device holds every finished chunk. The genius is that at each of the 2(N-1) steps a device sends only 1/N of the buffer, and all N links in the ring are busy simultaneously — no device and no wire ever sits idle waiting on a central root.
Deriving the 2(N-1)/N cost
Now count the bytes. Let M be the full buffer size in bytes, so each chunk is M/N. In the reduce-scatter phase a device sends one chunk on each of N-1 steps; the all-gather phase is identical. So the total data sent by any single device is:
bytes_sent = (N-1) · (M/N) [reduce-scatter]
+ (N-1) · (M/N) [all-gather]
= 2(N-1)/N · M
Time (alpha-beta, 2(N-1) steps):
T = 2(N-1) · α + 2(N-1)/N · M · βLook at what happens as N grows: the factor 2(N-1)/N → 2. The bandwidth cost per device approaches 2M and then stops — it is essentially independent of the number of devices. That is why ring all-reduce is called bandwidth-optimal: information theory says each device must send out its data and receive the result, giving a lower bound of 2(N-1)/N · M, and the ring hits it exactly. The only price that grows with N is the latency term 2(N-1)·α, which is tiny for the large buffers of real training.
A worked example
Take N = 8 GPUs synchronizing the gradients of a 250M-parameter model in fp16, so M = 250M × 2 bytes = 500 MB. Assume interconnect bandwidth B = 50 GB/s (so β = 1/B = 2 × 10^-11 s/byte) and per-message latency α = 5 µs.
bytes_sent = 2(8-1)/8 · 500 MB = (14/8) · 500 = 875 MB
bandwidth term = 875 MB / 50 GB/s = 0.0175 s = 17.5 ms
latency term = 2(8-1) · 5 µs = 70 µs = 0.07 ms
T_all-reduce ≈ 17.6 ms (99.6% bandwidth, 0.4% latency)Two lessons fall out. First, the run is overwhelmingly bandwidth-bound, confirming that the byte count — not the hop count — is what to optimize. Second, contrast the ring with a naive ‘send everything to a root and broadcast back’ scheme, whose root must move roughly 2(N-1)·M = 7 GB — eight times more traffic through one bottleneck link. The ring spreads the same reduction across all links so no single device moves more than 875 MB. That is the whole game: same result, a fraction of the cost, and it barely changes as you add GPUs.
Why all-reduce = reduce-scatter + all-gather
The two-phase structure of the ring is not a coincidence — it reflects an exact identity: all-reduce = reduce-scatter followed by all-gather. Reduce-scatter produces the fully summed result but leaves each device holding only its 1/N slice; all-gather then distributes those finished slices so everyone has the whole thing. Compose them and every device ends with the complete reduced buffer — the definition of all-reduce.
This decomposition is more than a curiosity; it is a lever. Because each half costs only (N-1)/N · M ≈ M in bandwidth, systems that can keep data sharded pay for just one half. This is exactly what fully-sharded data parallelism (FSDP / ZeRO) exploits: parameters live sharded, an all-gather reconstructs each layer just in time for its forward pass, and a reduce-scatter distributes gradients without ever materializing the full buffer on one device. You trade the convenience of a replicated tensor for half the peak memory and a communication bill split into two independently schedulable pieces — each of which can be overlapped with computation.
Mapping parallelism to collectives
Each parallelism strategy is essentially defined by which collective it fires and when. Data parallelism (DP) replicates the model and shards the batch; after backward, it needs one gradient all-reduce per step (or a reduce-scatter + all-gather pair under FSDP). Tensor parallelism (TP) shards individual matrix multiplies across devices, so a single layer’s forward pass injects an all-reduce (or all-gather / reduce-scatter) mid-layer — frequent, latency-sensitive traffic that demands a fast intra-node link.
Pipeline parallelism (PP) splits the model into stages along depth and passes activations forward and gradients backward between adjacent stages — these are point-to-point send/recv exchanges, not collectives, which is why PP is bandwidth-cheap but exposes pipeline bubbles. Expert parallelism (EP) for mixture-of-experts routes each token to its chosen expert on another device, an operation that is naturally an all-to-all (dispatch) and a second all-to-all (combine). Read a distributed training stack and you can recite its collectives: DP→all-reduce, TP→all-reduce, PP→send/recv, EP→all-to-all.
Practical implications and pitfalls
The theory has sharp practical edges. Because ring all-reduce is bandwidth-optimal, the way to hide it is to overlap it with computation: launch the gradient all-reduce for early layers while the backward pass is still churning through later ones, so communication and compute run concurrently. Frameworks bucket gradients precisely to enable this. A common pitfall is the opposite — many tiny all-reduces, each paying the α latency and never saturating bandwidth; fusing them into large buffers is almost always a win.
Topology also matters: a ring assumes uniform links, but real clusters are hierarchical (fast NVLink within a node, slower network between nodes), so libraries use hierarchical or tree algorithms that all-reduce within a node first, then across nodes. Finally, watch the interaction with precision — reducing in fp16 can lose small gradient contributions, so many stacks reduce in fp32 or use stochastic rounding. And remember the model itself: at very large N the residual 2(N-1)·α latency term eventually reasserts itself, which is one reason ultra-large runs blend ring with tree-based collectives to keep hop counts logarithmic.