What compute-bound actually means
Every kernel does two things: it moves bytes (loads inputs, stores outputs) and it does arithmetic (multiplies and adds). Those happen on two different pieces of hardware — the memory subsystem and the arithmetic units — and they overlap. Whichever finishes last determines the runtime. A kernel is compute-bound when the arithmetic takes longer than the byte movement, so the memory system has slack and the math units are the constraint.
Formally, model the time as t = max(FLOPs / P, bytes / B), where P is peak arithmetic throughput (FLOP/s) and B is peak memory bandwidth (bytes/s). When the first term dominates you are compute-bound and runtime scales with the FLOP count; when the second dominates you are memory-bound and runtime scales with bytes touched. The entire question of which regime you are in reduces to comparing those two terms — and that comparison has a clean closed form, which the next sections build up.
The FLOP ceiling
Compute-bound work runs into a wall you cannot argue with: the chip’s peak floating-point throughput. A datacenter GPU might advertise on the order of ~1000 TFLOP/s of dense bf16 matmul; a modern CPU with AVX-512 or AMX delivers a few TFLOP/s. That number is the product of the number of multiply-add lanes, their clock rate, and how many ops each lane retires per cycle. No software trick exceeds it — it is a property of the silicon.
So when you are genuinely compute-bound, the only levers are (1) reduce the FLOP count — smaller model, fewer tokens, sparsity, cheaper attention — or (2) raise the ceiling by using faster units (tensor cores) or lower precision. You cannot make a fixed matmul finish faster than FLOPs / P allows. Recognizing that a workload sits at the FLOP ceiling is liberating: it tells you to stop tuning memory access and start counting operations.
Arithmetic intensity: the deciding ratio
The single number that decides your regime is arithmetic intensity (AI): FLOPs performed per byte of memory traffic, AI = FLOPs / bytes. It is a property of the operation and its shapes, not of the hardware. A kernel that does a lot of math per byte it loads has high intensity; one that barely touches each byte before moving on has low intensity.
Compare AI against the hardware’s ridge point — the ratio P / B, peak FLOP/s divided by peak bandwidth. If your operation’s AI > P / B, memory can supply data faster than the units consume it, so you are compute-bound. If AI < P / B, the units starve and you are memory-bound. For a GPU with P ≈ 1000 TFLOP/s and B ≈ 3.3 TB/s, the ridge point is roughly 300 FLOP/byte. You need to do about 300 operations on every byte you fetch just to break even.
Matmul is the canonical compute-bound op
Matrix multiplication is where transformers spend most of their FLOPs, and it is the archetype of a compute-bound kernel. For C = A · B with A: [m, k] and B: [k, n], the cost is 2·m·n·k FLOPs (one multiply and one add per inner-product term), while the bytes touched are just the two inputs and one output: ~(mk + kn + mn) elements.
For a square matmul (m = n = k = N), that gives AI ≈ 2N³ / (3N²) = N/3 — intensity grows linearly with matrix size. Small matrices are memory-bound; big ones are firmly compute-bound. This is why high-performance GEMM keeps the matrices large and the tensor cores saturated: reuse each loaded tile across many multiply-adds so the FLOP-to-byte ratio climbs well past the ridge point.
Prefill is compute-bound, decode is not
Autoregressive inference has two phases with opposite personalities. Prefill ingests the whole prompt at once: a sequence of S tokens flows through every layer as a matrix X: [S, d]. Each linear layer is a big matmul with the batch dimension S on one side, so the weights loaded once are reused across all S tokens.
Decode generates one token at a time: the same layers now see X: [1, d], and the matmul degenerates into a matrix-vector product. Each weight is loaded from memory and used for a single multiply-add before being discarded. Prefill has arithmetic intensity on the order of S and sits on the compute roof; decode has intensity near 1 and is starved by memory bandwidth. Same model, same math, opposite bottlenecks — entirely because of the batch dimension.
A worked example: the intensity of a linear layer
Take the FFN up-projection X · W with X: [S, d] and W: [d, 4d]. The FLOPs are 2 · S · d · 4d = 8·S·d². In bf16 the weight matrix is 2 · d · 4d = 8d² bytes, which dominates the traffic when S is modest. So:
AI ≈ 8·S·d² FLOPs / 8·d² bytes = S FLOP/byteThe arithmetic intensity of a weight-bound linear layer is simply the number of tokens you push through it at once. With S = 2048 (prefill), AI ≈ 2048, far above a ~300 ridge point — compute-bound with room to spare. With S = 1 (decode), AI ≈ 1 — deeply memory-bound. The crossover to compute-bound happens right around the ridge point, so batching a few hundred decode requests together is exactly what pushes serving back onto the FLOP roof.
Tensor cores raise the compute roof
The FLOP ceiling is not one number — it depends on which units do the work. General vector lanes (FP32 CUDA cores, CPU SIMD) sit far below the specialized matrix engines: NVIDIA tensor cores, AMD matrix cores, Apple/Intel AMX. These do a small matrix multiply as a single instruction and deliver several times the throughput of scalar/vector math — but only for the matmul shapes they are built for.
That is why compute-bound performance hinges on feeding the tensor cores: dimensions that are multiples of the core’s tile (commonly 8 or 16), contiguous memory, and matrices large enough to amortize setup. A matmul whose k dimension is 130 instead of 128 can quietly drop to a slower path and fall off the high roof. Being compute-bound only pays off if the arithmetic runs on the fastest available units, and that is a shape-and-alignment discipline as much as an algorithmic one.
Lower precision moves the ceiling up
The other way to raise the roof is to shrink the numbers. Halving the bit width roughly doubles peak FLOP/s, because the same silicon area packs twice as many narrower multiply-add lanes: FP32 → bf16/FP16 → FP8 → FP4 each step up the ceiling. This is why mixed-precision and quantized inference are not just memory optimizations — on a compute-bound kernel they directly buy arithmetic throughput.
Lower precision also cuts the bytes moved, so it helps memory-bound kernels too, but the two wins differ: on the compute roof FP8 makes the math faster; on the memory roof it makes the loads smaller. The catch is numerical — narrow formats need scaling and higher-precision accumulation, and validation that quality holds. When it does, dropping a precision level is often the single biggest lever on a compute-bound transformer.