Ada Lovelace / Blackwell architecture — SM, FP formats, memory, NVLink

Why architecture matters here

Generation choices matter because software written for Hopper misses Blackwell wins (FP4). Architecture matters because SM + precisions + interconnect all shift.

The architecture: every piece explained

The top strip is generations. Hopper H100. H200. Blackwell. GB200.

The middle row is changes. SM redesign. FP formats. Memory bandwidth. NVLink.

The lower rows are ops. Training vs inference. Power + thermals. Ops — capacity + tuning + supply.

End-to-end flow

End-to-end: LLM inference workload targets Blackwell B200. FP4 kernels used for weight matmul; FP8 for activations. HBM3e bandwidth utilization 80%. NVLink 5.0 fabric for tensor parallelism. Power planned per rack.

Reading a generation as a set of ratios

A new GPU generation is almost never a uniformly faster version of the previous one. It is a set of ratios that moved by different amounts, and essentially every practical consequence — which kernels speed up, which do not, how many cards you need, what you have to rewrite — falls out of which ratio moved and by how much.

The ratios worth tracking are few. Low-precision matrix throughput per chip. Memory bandwidth per chip. Memory capacity per chip. On-die cache per chip. Bandwidth to the next GPU over. Power draw for all of the above. Read a generation as a vector of those six and the marketing headline stops being the interesting number, because the headline is nearly always the first one, and the first one is nearly always the one that moved most.

This article is about the differentials specifically: what the Ada Lovelace, Hopper and Blackwell generations changed relative to what came before, and what those changes oblige you to do differently. It deliberately does not re-explain the machine itself. For the standing architecture — SMs, warps, the memory hierarchy — see GPU architecture overview, the memory hierarchy, and the H100 SM. This one covers only the deltas.

One caution before the details. Part numbers, availability and headline specifications move faster than any written reference, and several of these lines have shipped refreshes since. Treat the mechanisms below as durable and every specific figure as something to re-check against a current datasheet before you spend money on it.

Two lineages, and what actually separates them

The most common analytical mistake with these generations is to line them up as one sequence. They are not one sequence. Ada Lovelace is the graphics-derived lineage — the consumer GeForce cards, the workstation boards, and the L40 / L40S class of server card built from the same silicon. Hopper is the datacenter lineage. Blackwell is a generation name that spans both: there are Blackwell consumer parts and Blackwell datacenter parts, and they are different chips with different memory systems that happen to share an architecture name and a numeric feature set.

What separates the lineages is not primarily the tensor core feature list. Ada's tensor cores are perfectly capable of low-precision inference work, and that capability is most of the pitch for the L40S-class server card. The divisions that actually hold are three, and all three are about getting data to the compute rather than the compute itself.

Memory technology. The graphics lineage uses GDDR on a conventional board; the datacenter lineage uses stacked HBM on a package. That single choice sets both the capacity ceiling and the bandwidth ceiling, and it is why an Ada-class card can post a respectable low-precision matmul number and still lose badly on anything capacity- or bandwidth-bound. See HBM for why the stacking changes the curve.

Peer interconnect. Covered below; it is the single largest determinant of whether a card can participate in a multi-GPU model.

Form factor and serviceability. Actively-cooled double-width consumer boards with display outputs are not the same deployment problem as passively-cooled datacenter modules that expect chassis-driven airflow or a cold plate. This decides how many fit in a rack long before performance does.

What changed inside the SM, and why your launch config ages

The gross shape of the SM has been stable for several generations: it is still partitioned, each partition still has its own scheduler and register file slice, and the same physical SRAM array still backs both L1 and shared memory. What moves generation to generation is the budget — how many registers, how much shared memory, how many resident warp and block slots one SM will actually give you, and what fraction of the SM's area went to the matrix pipe rather than the general one.

The datacenter line has additionally added machinery that changes what those budgets are spent on. A dedicated copy engine that moves multidimensional tiles from global to shared memory without burning a warp's worth of address arithmetic, and a scheduling tier above the thread block that lets several cooperating blocks be co-resident and address each other's shared memory, both belong to the Hopper-line datacenter parts rather than to Ada. Their internals are covered by the tensor memory accelerator and the H100 SM article; what matters here is only that they exist on one lineage and not the other, so a kernel written around them is not portable across the split.

The generational consequence that catches people is quieter than any of that. Occupancy is a minimum over several resource ceilings, and the standing arithmetic is worked through in the occupancy article. The generational point is that those ceilings are not constants. A kernel compiled with an explicit launch-bounds or maximum-registers directive — chosen years ago, on a specific part, to land exactly on a particular warps-per-SM target — is carrying a hard-coded answer to an arithmetic problem whose inputs have since changed. On the new part it does not fail. It quietly spills, or quietly runs at a warp count the newer SM did not need, and the regression looks like "the new card underperformed" rather than like a stale build flag.

The rule that follows is simple and widely ignored: any explicit occupancy or register directive is generation-scoped, and should be re-derived, not inherited, when you retarget. Autotuned tile sizes carry the same expiry date.

Narrow formats as a runtime policy, not a datatype

The visible headline across these generations is the addition of narrower floating-point formats to the matrix pipe — eight-bit on the Hopper and Ada generations, and a four-bit path on Blackwell. The formats themselves are worth understanding, and the numerics of the four-bit case in particular are treated properly in FP4 training on Blackwell and the tensor core dtype ladder. What is architecturally interesting for this article is a different thing: the generations also shipped the machinery for changing precision at runtime, per tensor, per layer, while the model runs.

The idea usually travels under the name "transformer engine," which undersells it by making it sound like a fixed-function block. It is better understood as a policy loop wrapped around the matrix pipe. Narrow formats have almost no dynamic range, so every narrow tensor needs a scale factor that maps its actual distribution onto the representable one. That scale has to come from somewhere, and it cannot be a constant because activation magnitudes drift as the network runs. So the runtime keeps a rolling record of the observed absolute maximum for each tensor it is managing, derives a scale from that history, and applies it on the next pass — with the fallback that if a tensor's distribution is too spread to survive the narrowing, that operator stays in a wider format while its neighbours do not.

Three practical consequences follow, and they are what actually shows up in a deployment.

First, precision becomes heterogeneous by layer. The interesting unit of decision is no longer "the model is FP8," it is "these matmuls are narrow, those are not, and the accumulator stayed wide throughout." Attention logits, normalisation, and the output projection commonly stay wider than the bulk feed-forward matmuls, because those are where a lost bit shows up as a visible quality change.

Second, there is a warm-up. A scale derived from history is wrong until there is history. Benchmarks taken over the first handful of iterations after a cold start will read low, and the fix is to discard them, not to tune anything.

Third, and most expensively: the silicon does not deliver the format, the software stack does. A new narrow path is a set of instructions, and your model reaches them only if the framework, the kernel library and the serving runtime all know how to emit them for that architecture. A workload that was carefully plumbed for eight-bit does not automatically pick up a four-bit path on new hardware — the quantisation recipe, the calibration, the kernel selection and often the checkpoint format all have to be redone. This is routinely the gap between the generational speedup on the slide and the generational speedup in your service.

The large L2, and where the break-even batch sits

Ada's most consequential quiet change was the size of its last-level on-die cache, which grew by roughly an order of magnitude relative to the preceding consumer generation — the exact figure varies considerably by die and by SKU within the generation, so treat it as a ratio and not a number. What L2 is and why you cannot address it directly belongs to the memory hierarchy article; the generational claim here is narrower, and it is about economics rather than mechanism.

GDDR-based cards have far less DRAM bandwidth than HBM-based ones. A very large on-die cache partially compensates, because it raises the size of the working set that can be re-read without touching DRAM at all. For autoregressive decode — which re-reads weights every single step and is bandwidth-bound almost by definition — that changes the arithmetic of where the break-even batch size sits. If the weights that a step must sweep fit largely in cache, a bandwidth-poor card behaves far better than its DRAM number predicts. Once the working set exceeds the cache, the effect disappears abruptly and you are back to the DRAM number.

The operational reading: on this class of card, the performance cliff is a function of model footprint plus live KV cache, and it is a cliff rather than a slope. Small models with modest concurrency can look surprisingly strong. The same card with a long-context workload, where the KV cache grows per request and evicts everything else, falls off the edge — and it falls off at a batch size you can estimate in advance rather than discovering in production. This is also why benchmark numbers for these parts are unusually sensitive to sequence length: the benchmark either fit in cache or it did not.

Shader execution reordering, and the general principle behind it

Ada introduced a hardware mechanism worth understanding even if you never write a graphics shader, because the problem it attacks is the oldest problem in the execution model. Threads execute in lockstep groups. When the threads in a group take different paths, the group pays for all of the paths. That is divergence, and the usual advice about it is entirely static: arrange your data so that neighbouring threads do the same thing.

Shader execution reordering makes that rearrangement dynamic. At a designated point, work items that have become divergent are re-binned by the hardware into fresh, coherent groups before the next stage runs, so the expensive stage executes over threads that agree about what they are doing. The motivating case is ray tracing, where a group of rays starts coherent and becomes hopelessly divergent as soon as they hit different materials — precisely the case where no amount of static data layout helps, because the divergence is generated at runtime by the data.

Be honest about the scope: this is exposed through the ray tracing pipeline, it is a feature of the graphics lineage, and it does not apply to the dense matmul and attention kernels that dominate machine-learning work, which are coherent by construction and were never divergence-limited. The reason to know about it is the direction it points. Sparse and conditional computation — mixture-of-experts routing, early exit, per-token branching — has exactly the runtime-generated divergence that static layout cannot fix, and hardware-assisted re-sorting is the shape of the answer if that class of workload ever justifies the silicon on the compute line.

What actually decides deployment - links, ECC, licensing

For a workload that fits on one card, the consumer and workstation parts are genuinely competitive on throughput per dollar. What removes them from consideration is almost never the compute. It is the following list, and it is worth going through before a purchase rather than after.

Peer-to-peer interconnect. This is the decisive one. The scale-up fabric — dedicated high-bandwidth links between GPUs, with a switch in larger systems — is a datacenter-lineage feature; the anatomy is in the NVLink article. Its absence is what matters here. Without peer links, GPUs talk over PCIe, frequently by way of host memory, at a small fraction of the bandwidth. That does not make multi-card impossible, and it is important to be precise about which form of parallelism it breaks. Tensor parallelism exchanges activations at every layer boundary and becomes impractical without peer links — the communication dominates. Pipeline parallelism, which splits the model by layer and passes an activation once per stage boundary, still works perfectly well over PCIe, and it is how large models are routinely run across consumer boards today. You pay for it in utilisation rather than in capability: stages serialise, bubbles appear, and per-card efficiency drops sharply compared with a peer-linked box running the same model tensor-parallel. See pipeline parallelism for what that costs.

Memory capacity per card. Downstream of the GDDR/HBM split, and it interacts with the previous point in the worst possible way: the cards with the least memory are also the ones that cannot pool it.

Error correction. Availability varies by SKU within a lineage, not just between lineages — some professional and server boards in the graphics lineage carry it, consumer boards generally do not. For a long-running training job an uncorrected flip is a silently wrong checkpoint; see ECC on GPUs.

Licensing. Two separate things, both easy to miss. The consumer driver license has historically carried a restriction on datacenter deployment; its exact scope has been revised more than once, and it must be read against the terms currently in force rather than remembered. Separately, virtualising a GPU across tenants is its own licensed product with its own per-seat cost — see vGPU — which is a real line item in any multi-tenant plan. Neither of these shows up in a benchmark, and both have killed otherwise sound hardware proposals late.

The trend line - why the ridge point keeps moving right

Put several generations side by side and the striking thing is how unevenly the ratios grew. Peak low-precision matrix throughput has grown fastest by a wide margin, helped along by each new narrower format roughly doubling the nominal figure again. Memory bandwidth has grown, substantially, but not nearly as fast. Memory capacity per package has grown more slowly still. Clock frequency has been approximately flat for years; the gains came from area, parallelism and narrower arithmetic, not from speed. Power went up throughout.

The consequence is mechanical. The roofline's ridge point — the arithmetic intensity at which a kernel stops being limited by memory and starts being limited by compute — sits at the ratio of peak compute to peak bandwidth. When the numerator grows faster than the denominator, generation after generation, that point moves right. Every new part demands more arithmetic per byte fetched than the last one did merely to keep the machine busy.

So a kernel that was comfortably compute-bound two generations ago can be memory-bound today without a single line of it changing. It got faster in absolute terms and worse in utilisation terms simultaneously, and the second fact is the one that determines whether the upgrade paid for itself. This is the memory wall, and each generation widens it in exactly the way that is hardest to see from a benchmark headline.

Autoregressive decode is the workload where this is most visible, because its arithmetic intensity is essentially fixed by the algorithm: one pass over the weights per step, a handful of operations per weight. Decode does not move right with the ridge point, so decode captures a shrinking share of each generation's headline gain. This is precisely why so much recent inference engineering — larger and continuous batching, speculative decoding, disaggregating prefill from decode — is in substance an effort to manufacture arithmetic intensity that the algorithm did not supply. The hardware trend line is what makes that work necessary, and it is why the software effort per generation has been rising rather than falling.

Choosing a generation, and actually collecting the speedup

The selection question resolves cleanly once you know which ratio binds your workload.

Capacity-bound — the model plus its KV cache does not fit — is a hard constraint before it is an economic one. It selects the HBM-bearing datacenter lineage, or it selects multiple cards, which by the previous section means it selects peer links too.

Bandwidth-bound, which is most single-stream decode, is decided on delivered bytes per second per unit cost, not on peak matmul. Generational compute gains largely miss this workload. The cache effect above is a genuine exception worth measuring on the graphics-lineage server parts if your footprint is small.

Compute-bound — training, prefill, large-batch offline scoring — is the case where the newest narrow format is worth real money, because this is the workload the headline number was measured on.

Multi-GPU by necessity collapses to interconnect, and the question is which split you can afford. If the deployment needs tensor-parallel latency, peer links are not optional. If a pipeline split and its utilisation penalty are acceptable, PCIe-only cards stay on the table — which is a real and frequently-taken option, not a consolation prize.

Then there is the step that gets skipped, which is verifying that your software will emit code for the part you bought. A binary shipped without architecture-specific machine code for the new target will still run: the driver just compiles the intermediate representation at load time. You pay a startup stall for that, and more importantly the just-in-time path generally will not conjure instructions that did not exist when the intermediate representation was produced. The new pipes sit idle while the card reports perfectly healthy utilisation. Build for the target architecture explicitly, pin the toolchain and kernel-library versions that know about it, and confirm on a profiler that the new instructions are actually being issued — Nsight will tell you in minutes what a throughput graph will not tell you in a week.

The blunt version: a generation upgrade is a software project with a hardware purchase attached. Budgeting for the card and not for the port is the most reliable way to buy a new architecture and measure the old one.