H100 SM architecture — warp scheduler, tensor cores, TMA, cluster

The SM is the machine you are actually programming

When you launch a grid, the work distributor hands thread blocks to SMs with enough free resources to host them. A block goes to exactly one SM and stays until all its threads retire — blocks never migrate and are never partially resident. That rule is why the SM is the right level for reasoning about performance: the grid tells you how much parallelism you asked for, the SM how much can be in flight.

Everything downstream follows. Occupancy, shared memory and latency hiding are all SM-level, and the profiler reports in the same terms — stall reasons, pipe utilization and achieved occupancy are per-SM averages. So the productive model is to understand one SM completely, then multiply: a kernel that keeps one SM busy keeps a hundred busy, provided the launch has enough blocks to go around.

A map of one SM

The figure maps the pieces and the dependencies between them. Read it as a supply chain rather than a spec sheet: every arrow is somewhere a kernel can stall, and the rest of this article walks them in order.

Four partitions, four warp schedulers

An SM is divided into processing partitions, each with its own warp scheduler, slice of the register file, and execution units. When a block arrives, its warps — groups of 32 threads advancing in lockstep — are spread across those partitions and pinned there for life, so each scheduler works from a fixed roster of resident warps.

Each cycle the scheduler picks one eligible warp and issues its next instruction. Eligible means the operands are ready, the target pipe can accept the instruction, and the warp is not parked on a barrier or a memory return. Warps that fail the test are simply skipped; because every warp’s state already lives in registers, the switch costs nothing. That is the whole latency-hiding story: no out-of-order execution rescues you, only other warps. Instructions within a warp issue in order, so independent work inside one warp helps too, but the primary lever is having enough resident warps that something is always eligible. When nothing is, the scheduler issues nothing — the idle slot a profiler surfaces as a stall.

The register file bounds how many warps stay resident

Registers are the SM’s largest and fastest storage, and they are statically partitioned. The compiler fixes how many registers each thread of a kernel needs, that allocation is reserved for the thread’s whole lifetime, and a warp becomes resident only if its allocation fits. A partition’s register file divided by per-thread demand is therefore a hard ceiling on resident warps.

That creates the central tension of GPU tuning. Generous register use makes each warp faster: more values kept on chip, more independent instructions available, fewer trips to memory. But it shrinks the roster the scheduler picks from, which is the very resource that hides latency. Push too far the other way and the compiler spills excess values to local memory, which despite the name is backed by the cache hierarchy and eventually device memory — usually a worse trade than the registers saved. The allocation also tracks the kernel’s peak demand, so one register-hungry phase taxes everything. Turning these limits into an occupancy figure is its own subject; what matters here is that residency is rationed, not requested.

One SRAM array, two jobs — L1 and shared memory

Each SM has one block of on-chip SRAM serving two roles. Part is the L1 data cache, managed by hardware, absorbing global and local traffic without you asking. The rest is shared memory, an explicitly managed scratchpad addressed by your code and scoped to a thread block. The split is configurable per kernel: you are trading hardware-managed caching for software-managed staging.

The consequence that bites is that shared memory is a residency resource just like registers. A block reserves its allocation for its lifetime, so if each asks for a large tile, fewer blocks fit and the scheduler again has fewer warps. Tiling is therefore never purely about reuse — a bigger tile improves arithmetic intensity but can quietly halve how many blocks are co-resident. The scratchpad’s internal behaviour, including bank structure and conflicting access patterns, is a topic of its own; here it is enough to treat shared memory as a scarce, block-scoped allocation carved from the same array your L1 hits come from.

Tensor cores are a separate pipe

The matrix units are not a faster version of the general arithmetic units. They are a distinct pipe with its own instructions: a warp — or, on Hopper, a cooperating group of warps — issues a matrix-multiply-accumulate that consumes tiles of two inputs and accumulates into a result tile held across the participating threads’ registers. One such instruction replaces a great many scalar multiply-adds, which is why it moves the bottleneck.

Because the matrix pipe retires work so much faster than the vector pipes, the binding constraint on a well-shaped GEMM or attention kernel is almost never the multiply itself. It is operand supply: getting the right tiles on chip, in the layout the instruction expects, early enough that the pipe never waits. Hopper’s warpgroup-level MMA leans into this by reading operands straight out of shared memory instead of requiring every value to be hand-staged into registers, which removes a great deal of address arithmetic and copying from the issue stream. The practical reading: when a matrix kernel underdelivers, look at the feed path before the math. Precision formats and MMA shapes are a separate subject.

Asynchronous copy decouples load from compute

The classic way to fill shared memory was a round trip through registers: each thread issues a global load, waits for it, then stores to the scratchpad. That burns registers, spends issue slots on pure data movement, and — worst — the warp doing the copying is the warp that must then do the math. Load and compute are welded together.

Asynchronous copy breaks the weld. A copy is initiated and the issuing warp moves on; completion is observed later through an asynchronous barrier that threads arrive at and wait on, and data flows global-to-scratchpad without being staged in registers at all. That is what makes real software pipelining possible: while the matrix pipe grinds on tile k, the copies for tile k+1 are already in flight, and the barrier is the only place the two meet — the pipelining pattern itself has its own treatment. Hopper’s Tensor Memory Accelerator takes this further, letting one thread launch a bulk multi-dimensional tile transfer from a precomputed descriptor with address generation done in hardware — that engine has its own article. The SM-level point is structural: copy work leaves the instruction stream, so scheduler slots go to arithmetic.

Thread-block clusters and distributed shared memory

Hopper inserts a scheduling level between the grid and the block: the cluster, a small set of blocks the hardware guarantees will be launched together and co-resident on a group of physically adjacent SMs. That guarantee is what makes the next feature safe — blocks certain to be running at the same time on neighbouring SMs can talk.

They talk through distributed shared memory: within a cluster, a block can address another block’s shared memory directly, and a cluster barrier synchronizes across blocks. Two things follow. A kernel can work on a tile larger than any single SM’s scratchpad would hold, by spreading it across the cluster; and SM-to-SM exchange no longer round-trips through L2 or device memory, which is the difference between an on-chip hop and a trip across the die. The cost is scheduling rigidity — the whole cluster must be placed at once, so a large cluster is harder to fit and can leave SMs idle at the tail of a grid. Clusters are opt-in and generation-specific, so portable code needs a path without them.

Reading a bottleneck off the SM

Holding this structure in your head turns profiler output from a wall of counters into a lookup table. Warps resident but rarely eligible, with stalls dominated by memory dependencies, says the feed path is too slow — check access patterns, reuse, and whether copies are truly asynchronous. Very few warps resident says a residency resource ran out: look at per-thread register demand and spill counters, then per-block shared-memory allocation. A high issue rate beside an idle matrix pipe says instructions are going to address arithmetic and staging rather than math.

The reassuring case is a matrix pipe busy most cycles: the SM is doing what it was built to do, and further gains come from algorithm and precision choices rather than scheduling. What the structure will not give you is permission to guess. Register demand is a compiler outcome, eligibility is a dynamic property, and the scratchpad split is a per-kernel decision — all measurable, none reliably predictable by eye. Use the SM model to know which counter to read, then read it.