Why architecture matters here
The reason NVLink deserves architectural attention is that it changes what parallelism strategies are affordable. Distributed training is a hierarchy of communication intensities: data parallelism syncs gradients once per step (tolerant, works over Ethernet), pipeline parallelism passes activations at stage boundaries (moderate), and tensor parallelism performs collective operations inside every single layer — two allreduces per transformer block, thousands per step. That last tier demands single-digit-microsecond latency and hundreds of GB/s, which only a memory-semantic scale-up fabric provides. The universal rule of thumb — tensor parallel within the NVLink domain, data/pipeline parallel across nodes — is not a style preference; it is the fabric's bandwidth cliff expressed as a sharding policy.
Inference feels the same cliff. Serving a 70B model in FP8 across four GPUs with tensor parallelism means every token's forward pass performs cross-GPU reductions; over NVLink these add a few percent to latency, over PCIe they dominate it. Prefill-decode disaggregation, KV-cache transfer between workers, MoE expert routing — each is designed around the assumption that intra-node GPU traffic is nearly free. When the fabric degrades (a flapping link silently halving bandwidth), those designs degrade with it, usually diagnosed weeks later as an unexplained tail-latency regression.
There is also a capacity dimension: unified addressing over NVLink lets a model that doesn't fit in one GPU's 80 GB borrow a neighbor's HBM at 900 GB/s — slower than local (3.3 TB/s) but 14x faster than host memory over PCIe. Understanding the fabric is understanding which of these borrowing strategies are real options.
The architecture: every piece explained
Top row of the diagram: the endpoints. Each H100 exposes 18 NVLink 4 ports; each port is a bundle of differential lanes running at 100 Gbit/s of signaling, yielding 25 GB/s per direction per link — 450 GB/s out, 450 GB/s in, 900 GB/s aggregate when all 18 are lit. Links carry memory-semantic packets: reads, writes, and atomics against remote HBM, tagged with the flat unified address space, not network frames. The GPU's memory subsystem treats a remote access like a long-latency local one — the SM issues a load, the MMU resolves it to a peer aperture, and the request rides the link, no driver or DMA descriptor in the path.
Middle tier: NVSwitch. An HGX H100 board carries four NVSwitch ASICs; every GPU splits its 18 links across all four switches, so any GPU-to-GPU path is one switch hop and the bisection is non-blocking — all eight GPUs can simultaneously stream at full bandwidth without contention. Third-generation NVSwitch also embeds SHARP engines: allreduce arithmetic executed inside the switch, so a reduction touches the fabric once instead of ring-hopping through every GPU, roughly doubling effective allreduce bandwidth for large messages. The Fabric Manager daemon programs switch routing tables at boot and quarantines failed links.
Bottom row: the software contracts. CUDA's peer access APIs (cudaDeviceEnablePeerAccess) stitch GPUs into one address space; NCCL probes the topology at init and builds rings and trees that traverse each physical link exactly once per direction; NVLink-C2C extends the same protocol to Grace CPUs (coherent CPU-GPU memory), and NVL72 extends the switch domain to 72 GPUs across a rack — one scale-up domain the size of yesterday's cluster.
The generational cadence matters for capacity planning. NVLink 1 on P100 offered 20 GB/s per link over four links; V100 moved to six links at 25 GB/s; A100 doubled to twelve links (600 GB/s aggregate) and introduced the third-generation switch; H100's eighteen links reach 900 GB/s; Blackwell's NVLink 5 doubles again to 1.8 TB/s per GPU. The through-line: per-GPU fabric bandwidth roughly doubles every generation while PCIe doubles every three to four years, so the ratio between scale-up and host connectivity keeps widening — and architectures that assumed the host path was merely slow find it proportionally slower every refresh. Mixed fleets therefore need per-generation performance models, not one number; an allreduce sized for H100 domains will underfill a Blackwell rack and drown an A100 box.
End-to-end flow
Follow one tensor-parallel allreduce through the fabric on an 8-GPU HGX node. A transformer layer's column-parallel matmul finishes; each GPU holds a partial sum of the activation tensor, say 256 MB. The framework calls ncclAllReduce. NCCL long ago probed the topology — eight GPUs, four switches, all-to-all — and chose its NVLS algorithm, which offloads the reduction to NVSwitch SHARP. Each GPU DMA-engines its partial tensor into the switch tier as multicast writes; the SHARP ALUs inside each NVSwitch sum corresponding chunks in flight and multicast results back down to all eight GPUs. Total fabric crossings per element: up once, down once — versus 2(N-1)/N crossings for a classic ring.
The transfer itself is chunked and pipelined. NCCL slices the 256 MB into staged buffers so that DMA-in, in-switch reduction, and DMA-out overlap; SM involvement is minimal — a few cooperative thread arrays orchestrating, not copying. At 900 GB/s effective, the reduction completes in roughly 300 microseconds and the next layer's matmul launches, often overlapped via a separate CUDA stream so communication hides behind compute.
Contrast the same call on a PCIe-only box: NCCL falls back to a ring through host memory or P2P over the PCIe root complex, bandwidth drops to ~50 GB/s shared, latency jumps an order of magnitude, and the allreduce that took 300 µs now takes 5 ms — per layer, per step. Multiply by 80 layers and thousands of steps, and the fabric is the training speed. The same arithmetic explains inference: a decode step's two reductions per layer ride the identical path, just with kilobyte-scale messages where latency, not bandwidth, dominates — which is why NCCL switches to latency-optimized tree/LL protocols for small sizes.
At the node boundary the two fabrics hand off. A cross-node allreduce in a hierarchical NCCL plan runs in phases: reduce-scatter inside each NVLink domain at 900 GB/s, one inter-node exchange over InfiniBand at 400 Gbit/s per rail (GPUDirect RDMA moving HBM-to-HBM without touching host memory), then an allgather back inside each domain — the expensive hop is crossed exactly once per element, by design. NVL72 shifts the boundary itself: with 72 GPUs in one switch domain, yesterday’s inter-node phase becomes intra-domain, and the parallelism plan redraws around the new cliff. The invariant worth memorizing: communication plans are shaped by where the bandwidth cliff sits, and every fabric generation moves it.