A rack of eight GPUs is only one machine to the extent that any GPU can reach any other GPU’s memory quickly. That is the entire job of NVLink and NVSwitch: the scale-up fabric that lives inside a node or a rack, underneath everything a distributed framework does. What makes it different from a network is not just that it is fast — it is that it is memory-semantic. A load instruction issued by a streaming multiprocessor can land in a neighbour GPU’s HBM with no descriptor, no driver call, and no packetization step you wrote. This piece walks the link anatomy, the crossbar, peer access and unified addressing, how topology shows up in collective bandwidth, and why the gap between this fabric and PCIe keeps widening.
A memory fabric, not a network
The most useful mental model for NVLink is that it is an extension of the GPU’s memory subsystem, not a NIC. Traffic on the link consists of reads, writes and atomics carrying physical addresses, not frames carrying payloads. When an SM issues a load whose address resolves to a peer aperture, the memory management unit routes the request out over a link, the remote GPU’s memory controller services it, and the data comes back into the requesting warp’s registers. The warp stalls exactly the way it stalls on a local HBM miss — longer, but structurally identical.
That single property is what separates scale-up from scale-out. Over a network, moving data between GPUs is an explicit transfer you must stage, post and complete. Over NVLink it can be an ordinary dereference. Everything else in this article — the bandwidth, the crossbar, the algorithms that become viable — is downstream of the fact that remote memory is addressable rather than sendable.
Link anatomy — where the aggregate number comes from
A single NVLink ‘link’ is a small bundle of differential lanes operating as a full-duplex pair, and a GPU exposes many of them. The headline figure quoted for a part is almost always the aggregate bidirectional number: links times per-link per-direction bandwidth times two. An H100 with eighteen fourth-generation links at 25 GB/s per direction gives 450 GB/s out, 450 GB/s in, and the familiar 900 GB/s aggregate.
Reading that number correctly matters, because a unidirectional workload can only ever use half of it. A one-way broadcast from a single source is capped near 450 GB/s on such a part, and an engineer who budgeted 900 will conclude the fabric is broken. The generational cadence is the other half of the story: per-link rate and link count have both climbed, from four links on P100 through six on V100, twelve on A100, eighteen on H100, and roughly a doubling again with Blackwell’s NVLink 5. Treat these as per-generation performance models, not one constant.
NVSwitch — the crossbar that flattens all-to-all
Direct GPU-to-GPU links alone give you a mesh, and a mesh has a problem: with eight endpoints and a fixed link budget, each pair gets only a fraction of a GPU’s ports, and any pair that is not directly wired must route through a neighbour, stealing that neighbour’s bandwidth. NVSwitch removes both effects. Each GPU fans all of its links into a tier of switch ASICs — four of them on a typical eight-GPU HGX board — so every GPU-to-GPU path is one switch hop wide open at full per-GPU bandwidth.
The consequence is uniformity. Every ordered pair of GPUs in the domain sees the same bandwidth and the same latency, and all pairs can transfer simultaneously without contending, because the switch tier is provisioned non-blocking. Recent switch generations also embed reduction arithmetic (NVLink SHARP) so that a sum can be computed in the switch rather than shuttled through every endpoint. Uniformity is worth more than raw speed to a scheduler: it means GPU placement inside the domain stops being a tuning parameter.
Peer access and unified addressing
The hardware capability is exposed through a small, blunt API surface. CUDA gives every allocation in a process a slot in one unified virtual address space, so a pointer is globally unambiguous; cudaDeviceCanAccessPeer asks whether a path exists, and cudaDeviceEnablePeerAccess maps the remote device’s memory into the local context. After that call, a device pointer belonging to GPU 3 can be dereferenced directly inside a kernel running on GPU 0.
Two flavours of use follow. Bulk: cudaMemcpyPeer (or a plain cudaMemcpyAsync between two device pointers) hands a large transfer to the copy engines, which saturate the links without occupying SMs. Fine-grained: a kernel simply reads or writes remote addresses, including atomics, which is what makes producer/consumer patterns and remote signalling possible at all. The cost model to keep in your head is that peer memory behaves like a slower, higher-latency tier of the same hierarchy — still far above host memory, still well below local HBM.
The bandwidth gap — NVLink versus the host path
The scale-up fabric only earns its complexity because the alternative path between two GPUs is so much narrower. PCIe is a host interconnect — shared and tree-structured — and the shape of that path is a subject of its own. What matters here is only the magnitude it sits at, one tier below the fabric.
| Path | Shape | Order of magnitude |
|---|---|---|
| Local HBM | on-package | several TB/s |
| NVLink peer | point-to-point / switched | hundreds of GB/s to TB/s |
| PCIe peer | shared tree, host root complex | tens of GB/s |
Those magnitudes are illustrative, but the ratio is the durable fact: roughly an order of magnitude per step. And the gap widens, because per-GPU fabric bandwidth has doubled roughly every generation while PCIe doubles every few years. A design that treats the host path as ‘merely slower’ gets proportionally more wrong at each refresh.
What peer access changes about which algorithms are viable
Parallelism strategies sort themselves by how often they communicate. Data parallelism synchronizes once per step and tolerates almost any medium. Pipeline parallelism passes activations at stage boundaries and tolerates a fair amount. Tensor parallelism reduces inside every layer, twice per transformer block, thousands of times per step — and only a memory-semantic fabric makes that affordable. The familiar rule that tensor parallelism stays inside the NVLink domain while data and pipeline parallelism span nodes is not a convention; it is this bandwidth cliff written as a sharding policy.
Peer access also unlocks patterns that are not collectives at all. A model too large for one device can borrow a neighbour’s HBM as an addressable tier rather than spilling to host memory. Expert-parallel MoE layers route tokens to remote experts as an all-to-all that would be prohibitive elsewhere. Inference workers hand KV-cache blocks between prefill and decode stages directly. Each of these silently assumes intra-domain traffic is nearly free.
Topology and how it lands on collective bandwidth
Not every multi-GPU box has the flat switched fabric described above. PCIe form-factor cards may carry only a bridge between adjacent pairs, or no NVLink at all. Older switchless designs wired a hybrid cube-mesh, where some pairs were directly connected and others were two hops apart. On such machines the communication library must build paths that respect the wiring, and the achievable bandwidth of a collective is set by the weakest link any path is forced to traverse.
NCCL handles this by probing the machine at initialization — enumerating devices, links, switches and PCIe ancestry — and constructing rings and trees that use each physical link at most once per direction, then selecting an algorithm and protocol per message size. The mechanics of those algorithms belong to the collectives article; the fabric-side point is narrower: the topology is an input to that search. Change the wiring, or lose a link, and the plan the library picks changes with it.
When the domain is not what you think it is
Fabric faults are rarely loud. A link that trains at reduced width, or flaps and is quarantined by the Fabric Manager, does not raise an error — the remaining links absorb the traffic and everything keeps working at lower bandwidth. The symptom surfaces weeks later as an unexplained step-time or tail-latency regression on one host in a fleet. Per-link counters (nvidia-smi nvlink, DCGM link and error metrics) and a periodic bandwidth probe are the only way to catch it early; treat achieved peer bandwidth as a monitored SLO, not an assumption.
Software can shrink the domain too. Containers or schedulers that hand a job a GPU subset spanning switch or partition boundaries, virtualization layers that block peer mappings, and processes that never call the enable-peer-access path all produce a job silently running on a slower topology than the hardware offers. Beyond the node the picture changes entirely — that is scale-out territory.
Sizing the scale-up domain
The number that most shapes a cluster design is the one people quote least: how many GPUs sit inside a single memory-semantic domain. For years that was eight, and an entire generation of parallelism plans internalized it — tensor parallel degree of eight or fewer, everything wider pushed to a slower tier. Rack-scale switch domains such as NVL72 move the boundary to dozens of GPUs, and coherent CPU-GPU links (NVLink-C2C) extend the same protocol to host memory, so a plan that was tightly constrained on one generation becomes loose on the next.
The practical discipline is to stop treating domain size as a constant. Ask, for a given machine: how many GPUs are one hop apart, at what per-GPU bandwidth, and where exactly does the cliff sit? A sharding plan is a hypothesis about that answer — re-measure it per generation and after every fabric incident, and the tuning becomes arithmetic rather than folklore.