Inside a server, GPUs talk over NVLink and the problem is a hardware one. The moment a job outgrows one chassis, every gradient has to cross a network — and a network has queues, routes, congestion, and a CPU that would very much like to touch each packet. InfiniBand plus RDMA exists to make that crossing look as much like a memory copy as physics allows: the adapter reads GPU memory directly, the kernel never sees the data, and switches can do the arithmetic on the way past. This is the scale-out fabric — how it is built, where it congests, and how to tell whether it is really what is slowing your run down.
Scale-up ends at the node boundary
A modern training server is already a small parallel machine: eight or so GPUs wired together by NVLink and NVSwitch into an all-to-all fabric with an order of magnitude more bandwidth than anything leaving the box. That is scale-up, a separate topic. Everything below is scale-out: the fabric between chassis, where bandwidth per GPU drops sharply and latency stops being a hardware constant and becomes a queuing outcome.
The step change matters because data-parallel training does not care where a peer lives. An all-reduce over 512 ranks touches every node in the job, so the slowest link in the path sets the pace for all of them: one rank whose packets queued behind someone else’s elephant flow makes 511 GPUs wait. That coupling — not raw bandwidth — is why fabric design gets the attention it does at cluster scale.
RDMA — moving bytes with the CPU out of the path
RDMA (remote direct memory access) is the primitive the whole fabric is built on. A conventional socket send copies your buffer into kernel memory, the kernel builds packets, the NIC sends them, and the receiver reverses all of it — copies and context switches per message, with CPU cycles burned in proportion to bandwidth. At 400 Gb/s that model does not close.
RDMA moves the data path into hardware. You register a memory region once, pinning it and handing the adapter its translation, open a queue pair to a peer, and post work requests that the HCA executes on its own. An RDMA WRITE is one-sided: the initiator names a remote address and a key, and the bytes land there with no code running on the far side at all. The kernel still handles setup, but not the steady state. That is what kernel bypass means: control through the kernel, data around it.
GPUDirect RDMA — the adapter writes straight into HBM
Kernel bypass alone still leaves an awkward hop. Without help, gradient data must be copied out of GPU memory into a host staging buffer before the adapter can send it, and copied back on the far side — two extra PCIe trips plus host memory bandwidth per message.
GPUDirect RDMA deletes the bounce. The GPU exposes part of its memory through a PCIe aperture, the driver hands the adapter a mapping for it, and the HCA then DMAs peer-to-peer directly to and from HBM. The data never enters system memory. The catch is topological: peer-to-peer works best when the GPU and its adapter sit under the same PCIe switch, and degrades — sometimes severely — when traffic must cross a root complex or an inter-socket link. Hence an adapter beside each GPU, and hence rank-to-device binding being more than cosmetic. The sibling feature this is not: GPUDirect Storage, which does the same trick for NVMe.
The fabric stack, from cable to collective
The top row is physical: GPUs running the workload, NVLink/NVSwitch inside the chassis, an InfiniBand HCA (host channel adapter) per GPU or per pair, and a switch fabric wiring the HCAs together. Everything above is software on the same wires. GPUDirect RDMA provides the zero-copy transport; NCCL turns a job’s ranks into rings and trees and issues the transfers, with SHARP offloading reductions into the switches where supported; adaptive routing steers flows around hot spots. The bottom layer is operational and easy to underrate: a scheduler binding ranks to GPUs and NUMA domains, telemetry on per-link errors, and the cable and firmware hygiene that keeps one bad optic from taxing a whole run.
Anatomy of an InfiniBand subnet
An InfiniBand network is a subnet, centrally managed rather than self-organising. A subnet manager — a process on a switch or a host — sweeps the fabric, discovers every port, assigns each a local identifier, computes forwarding tables, and pushes them into the switches. Endpoints run no routing protocol and negotiate nothing; they are told.
That buys deterministic, low-latency forwarding: a switch looks up a destination LID and forwards, cut-through rather than store-and-forward, so per-hop latency is hundreds of nanoseconds. It also makes the subnet manager a real dependency — when it re-sweeps after a link event, routes change under a running job. Ports are typically four lanes wide, current generations running hundreds of gigabits per second per port; treat any specific figure as generation-dependent and check the parts you actually have.
Fat trees, oversubscription and rail-optimized layouts
The dominant topology is a fat tree: servers attach to leaf switches, leaves attach to spines, and if each leaf devotes as much uplink capacity as downlink, the network is non-blocking — any permutation of node pairs can communicate at full rate. Cutting uplinks to save switches and optics gives you oversubscription, and a 2:1 leaf halves the bandwidth available to traffic leaving it. For web serving that is fine, because flows are small and uncorrelated. For an all-reduce, where every rank transmits at once, oversubscription shows up directly in step time.
Rail-optimized wiring exploits the node’s internal fabric. Each GPU’s adapter is a rail, and all the rail-k adapters across the cluster land on the same leaf switch. Same-index ranks then reach each other in one hop, and traffic that must change rails hops over NVLink inside the server rather than climbing to a spine.
Lossless links, congestion and adaptive routing
InfiniBand links are lossless by construction. A sender may only transmit when the receiver has advertised buffer credits, so packets are not dropped for lack of space and there is no TCP-style loss signal. That removes retransmission from the common path but changes the failure mode: a congested link pushes back instead of dropping, and the backpressure propagates upstream into switches carrying unrelated traffic.
The trigger is incast — many senders converging on one receiver, exactly what the last phase of a reduction looks like. Congestion control handles it end-to-end: switches mark packets experiencing congestion, the receiver notifies the sender, the sender paces its injection rate. Adaptive routing attacks the other half, spreading flows across equal-cost uplinks by live queue occupancy rather than a static hash — which matters because a handful of enormous, long-lived collective flows is the worst possible input to hash-based balancing.
SHARP — doing the reduction inside the switch
In an ordinary all-reduce, gradients travel to where the arithmetic happens and the result travels back. SHARP (scalable hierarchical aggregation and reduction protocol) moves the arithmetic into the network instead. Switch ASICs are organised into an aggregation tree by a manager service; children send contributions upward, each switch sums what it receives and forwards a single result, and the final value is multicast back down.
Two things improve at once. Spine traffic drops, because N contributions collapse into one at each level instead of being relayed. And latency for small collectives falls, because the operation completes in tree depth rather than a chain of host-to-host steps — the thing small, frequent synchronisations are bound by. It is not free: aggregation trees are a finite switch resource, the feature must be enabled end to end, and supported data types and operators are limited.
When does the network actually become the bottleneck
Do the arithmetic before blaming the fabric. In a data-parallel step every rank must exchange the whole gradient buffer, and a ring all-reduce puts about 2×S bytes on each rank’s wire, where S is the gradient size — a figure that barely changes as you add nodes.
Take an illustrative case: a model with 8 billion parameters, gradients in 16-bit, so S ≈ 16 GB. Each rank moves roughly 32 GB. A 400 Gb/s rail is about 50 GB/s of payload at best, so the collective needs on the order of 0.6 s of wire time per step. If the backward pass takes 2 s and you overlap well, it disappears. If you scale out until each rank’s compute is 0.5 s, it cannot — per-rank communication stayed flat while compute per rank shrank. That is the scaling wall in one line, and it is why sharded schemes and larger per-device batches are network decisions as much as memory ones.
Overlap, placement and what to measure
The first lever is overlap. Gradients become available progressively during the backward pass, so frameworks bucket them and launch each reduction as its bucket fills, hiding communication behind the compute still to come. Bucket too finely and you pay latency per message; too coarsely and there is no compute left to hide behind.
The second is placement. A job packed onto adjacent leaves, ranks bound to the GPU and adapter sharing a PCIe switch, has a different fabric profile from the same job scattered across a pod — same code, same hardware, different step time.
The third is measurement. Establish real point-to-point bandwidth with a microbenchmark, run a collective benchmark at the job’s actual message sizes, and compare against the arithmetic above. Watch port counters for symbol errors and for wait time indicating backpressure: a marginal cable that retrains under load never appears as an outage, only as a run that is quietly slower than it should be.