GPU HBM architecture — bandwidth, banks, coalescing, roofline

Stacking next to the die buys width, not clock speed

An HBM device is a stack: several DRAM dies mounted one on top of another over a base logic die, with signals running vertically through the silicon on through-silicon vias rather than out to package pins. The stack does not sit on the motherboard. It sits on a silicon interposer inside the GPU package, a few millimetres from the compute die, and the interposer carries the data lines between them.

That geometry is the whole product. A signal crossing micrometres of interposer is cheap to drive and easy to route, so you can afford an enormous number of parallel wires — on the order of a thousand data signals per stack across the HBM2 and HBM3 generations, with the interface widening again in later ones. A conventional memory device attached to a printed circuit board gets a few dozen. Bandwidth is width multiplied by per-pin rate, and HBM takes the extreme end of the width axis while running each pin comparatively slowly. Short, low-capacitance links also cost less energy per bit, which stops mattering as an abstraction and starts mattering as watts once the interface is moving terabytes per second.

The bill for that arrangement is paid in manufacturing. Interposers consume package area, the through-silicon via process is exotic, stacking multiplies the yield risk of every die involved, and testing dies before they are buried in a stack is its own discipline. Only a handful of vendors ship the parts at all, which is why accelerator supply and HBM supply tend to be the same conversation. And heat is structural: a die in the middle of a stack has no direct path to the heatsink, so it runs hot, and hot DRAM leaks faster — a fact that comes back below as a bandwidth tax.

Inside a stack - channels, pseudo-channels, bank groups, banks

A stack is not one memory. It is a pile of largely independent ones, and the independence is what produces the bandwidth number on the datasheet.

At the top of the ladder are channels. Each has its own command bus and its own timing state, so a request queued on one channel neither waits for nor blocks a request on another. In the HBM2 generation and later, each channel is further split into two pseudo-channels that share the command wires but own separate data paths and separate memory arrays — a cheap way to double the number of independent request streams without doubling the command pins. Pseudo-channel mode dates from the HBM2 generation, and later generations have gone on rebalancing the split: more, narrower channels rather than fewer, wider ones, because independence is what the controller can actually exploit. Below that sit bank groups, and inside each group a small number of banks.

The bank is the atom. It is an array of cells with its own row of sense amplifiers, and it is the smallest thing that can be activated independently. A single DRAM array is slow in absolute terms; the design compensates by having many of them, so the controller can be opening a row in one bank while streaming column data out of another. Bank groups exist for a subtler timing reason: consecutive column commands aimed at the same group must be spaced further apart than commands aimed at different groups, so alternating across groups roughly doubles the sustainable command rate.

None of this is exposed to CUDA. No API lets you name a bank, and the mapping from a physical address to a channel and bank is a vendor-chosen hash, generally designed so that a linear sweep spreads evenly across every channel in the device. You cannot control the mapping; you can only feed it address patterns it handles well or badly.

Row buffers and the activate/precharge cycle

Reading a DRAM bank takes three distinct operations, and knowing them is the difference between guessing at memory performance and reasoning about it.

Activate copies one addressed row of cells into the bank's sense amplifiers. The read is destructive — the cells are drained in the process — so the sense amps now hold the only live copy. Column read selects some bytes out of that latched row and drives them onto the data bus; this is the cheap step, and it can be repeated. Precharge writes the row back to the cells and resets the bitlines so a different row can be activated. The sense amplifier array is the row buffer, and it holds on the order of a kilobyte.

Every request therefore lands in one of three cases. A row hit finds its row already open and pays only a column read. A row miss on an idle bank pays an activate plus a column read. A row conflict — a different row is open in the target bank — pays precharge, then activate, then read: the full row cycle, tens of nanoseconds, during which that bank serves nobody.

Access order decides which case you get, and nothing else does. Walking contiguously through a region yields long runs of column reads per activation, which is the regime the whole memory system is designed for. Hopping between distant addresses that happen to land in the same bank yields a conflict per access and collapses that bank's throughput. There is a second ceiling too: activation draws a current spike, so the standard timing rules cap how many activates may be issued inside a rolling window. A pattern that is nothing but row opens hits that limit before it ever saturates the data pins.

DRAM bank conflicts are not shared-memory bank conflicts

The word "bank" names two different things one level apart, and conflating them sends people to the wrong fix. It is worth separating them explicitly.

A shared-memory bank conflict happens on-chip, inside an SM. The scratchpad SRAM is divided into banks; when lanes of a single warp address different words that map to the same bank, the accesses serialize by a few cycles per way. It is a warp-level phenomenon, computable on paper from the index arithmetic, and fixed by padding a tile or swizzling the index. That is the subject of the shared memory architecture article, and everything about it stays inside the SM.

A DRAM bank conflict happens off-chip, inside the HBM stack. It occurs when in-flight requests want different rows of the same bank, and the cost is not cycles but a full precharge-activate cycle measured in tens of nanoseconds. It is not a warp-level property at all: it depends on which requests from which warps happen to be outstanding at the same moment, and on the controller's address hash, which is undocumented. So you cannot derive it — you observe it, in DRAM-side profiler counters, and you attack it by changing strides and traversal order rather than by padding a scratchpad tile.

The two do share one reflex: an awkward power-of-two stride is usually the culprit, and nudging a leading dimension is usually the cure. The mechanism is different, the magnitude is different by orders, and the counters live in different sections of the profiler.

Bursts, sectors, and coalescing seen from the DRAM side

A column read does not return a byte. It returns a burst: a fixed number of beats clocked out on the pseudo-channel's data pins. Multiply that narrow data width by the burst length and you get a minimum transfer granularity of a few tens of bytes. This is the physical origin of the GPU's 32-byte memory sector — it is not a cache-design choice so much as the smallest parcel DRAM is willing to hand over. Request four bytes and thirty-two move.

Coalescing, then, is the warp-side name for a DRAM-side fact. A warp whose lanes cover one contiguous run resolves into a handful of bursts drawn from one or two already-open rows. A warp whose lanes scatter resolves into as many bursts as there are lanes, most of the moved bytes discarded, spread over rows the controller must now open and close. The rules for arranging addresses so the first thing happens belong to the coalescing article; the reason those rules exist is the burst and the row.

There is a second-order effect worth knowing. Because the controller derives the channel and bank from address bits, a stride that is a large power of two can alias — every request from a warp landing on the same channel or the same bank after the hash, concentrating the entire kernel's traffic onto a fraction of the memory system while the rest idles. This is the classic partition-camping failure, and it is why matrix leading dimensions in fast libraries are so often a suspiciously non-round number.

conceptual address decomposition (the real mapping is a vendor hash)

  [ ....... row ....... | bank grp | bank | channel/pseudo-ch | column | byte ]
                                             ^ deliberately low-ish bits, so a
                                               linear sweep spreads over every
                                               channel in the device

walking a row-major matrix down a column, floats, leading dimension ld:

  addr(lane) = base + lane*ld*4          # stride between adjacent lanes = ld*4

  ld = 1024  -> stride 4096 B  -> every lane a different row, and after the
                                  hash only a few distinct banks: conflicts
  ld = 1025  -> stride 4100 B  -> row and bank indices both walk: the same
                                  accesses now spread across the device

Refresh - the bandwidth you never get to use

DRAM cells are capacitors and capacitors leak, so every row has to be read out and written back within a retention window measured in tens of milliseconds. The controller spreads that work out, issuing refresh commands at a steady average interval, and while a bank is refreshing it cannot answer a read. Modern parts support per-bank refresh so the whole channel does not stall for each one, which keeps the aggregate cost to a small single-digit percentage of peak at nominal temperature. Annoying, not decisive.

Two things make it worse. Density is one: more rows on a die means more refresh work to fit in the same window, which is why the tax has crept upward across generations rather than shrinking. Temperature is the other, and it is the one that bites operationally. Cell retention degrades sharply with heat, so parts switch to a doubled refresh rate above a temperature threshold — roughly doubling the overhead precisely when the GPU is hottest and you most want throughput.

The practical consequence is that a thermally marginal node loses memory bandwidth as well as clocks, and the loss is nearly invisible. There is no counter labelled "refresh stole your bandwidth." It shows up as a job that benchmarks fine in the first ten minutes and runs several percent slower in hour three, on the racks with the worst airflow. If you are chasing an unexplained throughput drift across otherwise identical nodes, memory temperature belongs on the list alongside the usual clock throttling — deployment and cooling are bandwidth concerns, not just power concerns.

ECC and what error protection costs

Datacenter HBM is error-protected in more than one layer, and the layers do different jobs. On-die ECC lives inside the DRAM, corrects single-cell failures, and is invisible to software; it exists because at current cell densities a background rate of weak cells is a manufacturing certainty rather than an anomaly. Link protection guards the wires with a check code and a retry on failure — a retry costs latency, and a climbing retry rate is an early symptom of a marginal stack. Above both sits the ECC the driver reports on, covering stored data, correcting single-bit errors and detecting multi-bit ones.

None of it is free, because check bits are bits. They occupy storage that is therefore not holding your tensors, and they ride the bus alongside the data they protect, consuming cycles that are therefore not moving your tensors. Whether you see that cost depends entirely on where the check bits live. Parts built with dedicated ECC storage inside the stack absorb it into the numbers already quoted on the datasheet, so enabling protection changes nothing visible. Products that implement ECC by carving check bits out of the ordinary memory array show it plainly: usable capacity drops by a few percent and bandwidth by a similar order when protection is switched on. Check which kind you have before you write a capacity plan against the marketing figure.

Operationally, watch the counters. Correctable errors are logged and rows get retired over time; a rising retirement trend is a hardware ticket, not a curiosity. An uncorrectable error is a different event entirely — it kills the process, frequently requires a device reset, and on a long training run it costs you whatever sits between the failure and your last checkpoint.

Why achieved bandwidth is always below peak

Peak bandwidth is pin count multiplied by per-pin rate. Nothing reaches it, and the gap is the sum of everything above plus two more effects.

Bus turnaround. The data path of a channel is shared between reads and writes, and reversing its direction costs idle beats. A kernel that reads and writes in roughly equal measure — which is to say most elementwise work — pays that penalty constantly and will sustain noticeably less than a read-only stream over the same footprint.

Not enough requests in flight. This is the one people miss. Bandwidth is throughput, DRAM latency is long, and by Little's law you must keep bandwidth multiplied by latency worth of bytes outstanding at every instant just to keep the bus busy. Halve the number of resident warps on a perfectly coalesced kernel and you can roughly halve its achieved bandwidth without touching a single address. This is why occupancy, loop unrolling, wide vector loads and asynchronous copy all raise measured bandwidth: they raise the count of concurrent outstanding requests.

Add refresh, row conflicts, activate limits and any imbalance across channels, and a large purely streaming read typically lands somewhere in the eighty-to-ninety percent range of peak, with anything less friendly landing lower. The consequence for analysis is direct: build your performance model on a measured ceiling, never the datasheet one. A roofline drawn against marketing bandwidth systematically reports that your kernel is worse than it is, and sends people optimizing a kernel that is already done. The model itself is covered in the roofline article; what belongs here is the number you feed it.

establish YOUR ceiling before you draw any roof

  1. pure streaming read, buffer far larger than L2
       acc += a[i]                       -> BW_read     (best case)

  2. copy: equal read and write traffic, worst turnaround case
       b[i] = a[i]                       -> BW_copy     (expect below BW_read)

  3. the kernel under test
       bytes_it_must_move / kernel_time  -> BW_kernel

  report BW_kernel / BW_copy for a read-write kernel,
         BW_kernel / BW_read for a read-dominated one.
  never   BW_kernel / BW_datasheet.

Capacity and bandwidth are sold together

You do not buy bandwidth. You buy stacks, and every stack arrives carrying both its capacity and its slice of the interface. That coupling drives more architecture decisions than it gets credit for.

Within a generation, the only ways to give one GPU more bandwidth are more stacks or a faster interface, and both are frozen at tape-out. The single field-level knob is more GPUs. Meanwhile mid-generation refreshes usually add capacity by stacking taller or denser dies, which raises capacity faster than it raises bandwidth — so the bytes-per-second available per byte stored quietly falls, and a model that newly fits on one device may not decode any faster on it.

That reframes model-parallel choices. Tensor parallelism shards weights across devices, so each GPU reads only its fraction of the parameters per token and the aggregate bandwidth applied to a single token multiplies by the degree of sharding. Tensor parallelism is therefore a latency optimization as much as a capacity one, which is why serving stacks use it on models that would comfortably fit on fewer cards — at the price of a collective on the critical path of every layer. Pipeline parallelism is the opposite trade: it adds capacity, but a token still traverses every stage in sequence, so per-token latency does not improve and the win arrives as throughput through microbatching. And offloading to host memory swaps an HBM read for a link read an order of magnitude slower, which only pays for state touched rarely — cold cache blocks, inactive experts. See sharding and pipeline parallelism for those mechanics.

HBM versus GDDR as a design point

Both are the same DRAM cells. They differ in how the bits escape the die, and every other difference follows from that one.

GDDR pushes serial speed down a narrow bus that an ordinary circuit board can route, which keeps cards affordable and repairable while capping both aggregate width and total capacity. HBM pays interposer and packaging cost to buy width instead. For a workload that reads its entire weight set to produce one token, width wins by a margin large enough that price stops being a real comparison. For a workload dominated by arithmetic over a small working set — much of graphics — GDDR is the better trade, which is why the split has persisted rather than resolving.

The practical corollary catches people out regularly: a consumer card with a respectable FLOP number will still generate tokens slowly, because token generation is a bandwidth product, not an arithmetic one.

The generational trend and the memory wall

Each HBM generation adds stacks, taller stacks, denser dies and a faster interface, and periodically the per-stack bus itself widens. All of it helps. None of it keeps up.

Arithmetic throughput has grown faster, generation over generation, than memory bandwidth, and the reason is geometric rather than strategic: adding tensor cores inside a reticle is a far easier engineering problem than adding pins, stacks and interposer area around one. Capacity has grown slower still relative to the models people want to run. So machine balance — peak operations per second divided by peak bytes per second — has been climbing for years, and every increase pushes the roofline's ridge point further right, dropping more kernels into the bandwidth-bound region on the same code.

That is the memory wall, and it is the reason nearly every LLM kernel is bandwidth-bound rather than compute-bound. Generating a token requires reading every weight the forward pass touches while doing very little arithmetic with each one, so the arithmetic intensity of decode is pinned low no matter how large the model gets; the accounting is worked through in the bandwidth-bound operations article.

Read the last few years of serving optimizations in that light and they stop looking like a grab bag. Larger batches, grouped-query attention, quantized caches, paged cache blocks, kernel fusion, tiled attention that keeps intermediates on-chip — every one of them is a scheme for moving fewer bytes across the HBM boundary. They are all the same idea, wearing different names, cast by the same wall.

What a kernel author actually does about it

The mechanics above reduce to a short procedure, and it is worth running in order rather than reaching straight for the profiler's suggestions.

First, do the division. Count the bytes your kernel is obliged to move, divide by the measured achievable bandwidth from the recipe above, and compare that to its actual runtime. That ratio, and nothing else, tells you whether there is anything left to win.

If you are near the ceiling, stop tuning instructions. No scheduling change rescues a kernel already moving bytes at line rate. The only remaining lever is moving fewer of them: fuse adjacent operations so intermediates never round-trip, drop to a narrower dtype, restructure to create reuse the current formulation is throwing away. That is fusion and tiling territory.

If you are far below it, work down the list. Are the lanes of a warp covering contiguous addresses? Are there enough concurrent requests in flight to cover DRAM latency, or has register pressure crushed residency? Does the traversal produce long runs inside an open row, or does it hop? Is a leading dimension an unfortunate power of two? Is the kernel paying read-write turnaround it could avoid by splitting phases?

Then the mechanical wins: widen loads so each request carries more bytes, issue copies asynchronously so the pipeline stays full, and order block indices so that concurrently-resident blocks touch neighbouring addresses — which helps L2 hit rate and hands the memory controller row locality at the same time. Finally, look at the DRAM-side counters specifically, not only the L1 and L2 ones; a kernel can post an excellent cache hit rate and still be strangling one channel. The tooling is covered in the profiling article, and the surrounding hardware map in the memory hierarchy article.