Why architecture matters here
Distributed training loss curves that plateau earlier than expected almost always trace to network. A single flapping cable stalls collectives; a topology-blind assignor puts ranks far apart; a firmware bug on the HCA causes retries.
The architecture matters because collectives are joint operations — one slow rank slows all ranks. NCCL discovers topology; SHARP does inline reductions in switches; adaptive routing avoids congestion. Each layer solves specific failures.
Get them cooperating and the fabric disappears from your bottleneck analysis; get them wrong and you cannot even see where the time went.
The architecture: every piece explained
The top strip is the physical fabric. GPUs run training. NVLink / NVSwitch is the intra-node fabric — hundreds of GB/s between GPUs on the same server. IB HCA is the InfiniBand host channel adapter per server (or per pair of GPUs). IB switch fabric connects nodes in a fat tree or dragonfly topology at 200-400 Gb/s per link.
The middle row is the software stack. GPUDirect RDMA lets HCAs DMA directly to GPU memory — no host bounce. NCCL / SHARP implements all-reduce, all-gather, reduce-scatter; SHARP performs the reduction inside the switch when supported. Topology-aware algorithms choose ring, tree, or double-binary tree based on topology to minimize hops. Adaptive routing reroutes around congested links.
The lower rows are ops. Job scheduler binds MPI ranks to specific GPUs and NUMA domains for locality. Telemetry watches per-link errors, retransmits, and bandwidth. Ops runs cable tests, firmware updates, PXE boots, and NCCL validation sweeps before every large training job.
End-to-end flow
End-to-end: a training job launches 1024 ranks across 128 nodes. Scheduler binds each rank to a specific GPU and matches CPU affinity to the IB HCA. NCCL bootstraps, discovers topology, and picks a tree-based all-reduce with SHARP inline reduction on supported switches. GPUDirect RDMA avoids host bounces; latency per hop is microseconds. Training progresses at 54% MFU. A cable degrades mid-run; telemetry flags rising errors; the fabric reroutes; the run continues with a 1% slowdown until the cable is swapped in the next maintenance window.