An operation is compute-bound when the arithmetic itself is the bottleneck — the processor’s multiply-add units are the busy resource, and memory can feed them fast enough to keep them fed. This is the regime you want to be in: it means the hardware is doing useful math rather than waiting on data. Large matmuls, transformer prefill, and high-batch training all live here, pressed up against a hard ceiling set by peak FLOP throughput. The mirror image is the memory-bound regime, where the units sit idle waiting on the memory bus — the world of single-token decode. This article stays on the compute-bound side: what sets the ceiling, the arithmetic-intensity threshold that decides which side you land on, and how to tell, concretely, when you are compute-bound versus memory-bound.
What compute-bound actually means
Every kernel does two things: it moves bytes (loads inputs, stores outputs) and it does arithmetic (multiplies and adds). Those happen on two different pieces of hardware — the memory subsystem and the arithmetic units — and they overlap. Whichever finishes last determines the runtime. A kernel is compute-bound when the arithmetic takes longer than the byte movement, so the memory system has slack and the math units are the constraint.
Formally, model the time as t = max(FLOPs / P, bytes / B), where P is peak arithmetic throughput (FLOP/s) and B is peak memory bandwidth (bytes/s). When the first term dominates you are compute-bound and runtime scales with the FLOP count; when the second dominates you are memory-bound and runtime scales with bytes touched. The entire question of which regime you are in reduces to comparing those two terms — and that comparison has a clean closed form, which the next sections build up.
The FLOP ceiling
Compute-bound work runs into a wall you cannot argue with: the chip’s peak floating-point throughput. A datacenter GPU might advertise on the order of ~1000 TFLOP/s of dense bf16 matmul; a modern CPU with AVX-512 or AMX delivers a few TFLOP/s. That number is the product of the number of multiply-add lanes, their clock rate, and how many ops each lane retires per cycle. No software trick exceeds it — it is a property of the silicon.
So when you are genuinely compute-bound, the only levers are (1) reduce the FLOP count — smaller model, fewer tokens, sparsity, cheaper attention — or (2) raise the ceiling by using faster units (tensor cores) or lower precision. You cannot make a fixed matmul finish faster than FLOPs / P allows. Recognizing that a workload sits at the FLOP ceiling is liberating: it tells you to stop tuning memory access and start counting operations.
Arithmetic intensity: the deciding ratio
The single number that decides your regime is arithmetic intensity (AI): FLOPs performed per byte of memory traffic, AI = FLOPs / bytes. It is a property of the operation and its shapes, not of the hardware. A kernel that does a lot of math per byte it loads has high intensity; one that barely touches each byte before moving on has low intensity.
Compare AI against the hardware’s ridge point — the ratio P / B, peak FLOP/s divided by peak bandwidth. If your operation’s AI > P / B, memory can supply data faster than the units consume it, so you are compute-bound. If AI < P / B, the units starve and you are memory-bound. For a GPU with P ≈ 1000 TFLOP/s and B ≈ 3.3 TB/s, the ridge point is roughly 300 FLOP/byte. You need to do about 300 operations on every byte you fetch just to break even.
Matmul is the canonical compute-bound op
Matrix multiplication is where transformers spend most of their FLOPs, and it is the archetype of a compute-bound kernel. For C = A · B with A: [m, k] and B: [k, n], the cost is 2·m·n·k FLOPs (one multiply and one add per inner-product term), while the bytes touched are just the two inputs and one output: ~(mk + kn + mn) elements.
For a square matmul (m = n = k = N), that gives AI ≈ 2N³ / (3N²) = N/3 — intensity grows linearly with matrix size. Small matrices are memory-bound; big ones are firmly compute-bound. This is why high-performance GEMM keeps the matrices large and the tensor cores saturated: reuse each loaded tile across many multiply-adds so the FLOP-to-byte ratio climbs well past the ridge point.
Prefill is compute-bound, decode is not
Autoregressive inference has two phases with opposite personalities. Prefill ingests the whole prompt at once: a sequence of S tokens flows through every layer as a matrix X: [S, d]. Each linear layer is a big matmul with the batch dimension S on one side, so the weights loaded once are reused across all S tokens.
Decode generates one token at a time: the same layers now see X: [1, d], and the matmul degenerates into a matrix-vector product. Each weight is loaded from memory and used for a single multiply-add before being discarded. Prefill has arithmetic intensity on the order of S and sits on the compute roof; decode has intensity near 1 and is starved by memory bandwidth. Same model, same math, opposite bottlenecks — entirely because of the batch dimension.
A worked example: the intensity of a linear layer
Take the FFN up-projection X · W with X: [S, d] and W: [d, 4d]. The FLOPs are 2 · S · d · 4d = 8·S·d². In bf16 the weight matrix is 2 · d · 4d = 8d² bytes, which dominates the traffic when S is modest. So:
AI ≈ 8·S·d² FLOPs / 8·d² bytes = S FLOP/byteThe arithmetic intensity of a weight-bound linear layer is simply the number of tokens you push through it at once. With S = 2048 (prefill), AI ≈ 2048, far above a ~300 ridge point — compute-bound with room to spare. With S = 1 (decode), AI ≈ 1 — deeply memory-bound. The crossover to compute-bound happens right around the ridge point, so batching a few hundred decode requests together is exactly what pushes serving back onto the FLOP roof.
Tensor cores raise the compute roof
The FLOP ceiling is not one number — it depends on which units do the work. General vector lanes (FP32 CUDA cores, CPU SIMD) sit far below the specialized matrix engines: NVIDIA tensor cores, AMD matrix cores, Apple/Intel AMX. These do a small matrix multiply as a single instruction and deliver several times the throughput of scalar/vector math — but only for the matmul shapes they are built for.
That is why compute-bound performance hinges on feeding the tensor cores: dimensions that are multiples of the core’s tile (commonly 8 or 16), contiguous memory, and matrices large enough to amortize setup. A matmul whose k dimension is 130 instead of 128 can quietly drop to a slower path and fall off the high roof. Being compute-bound only pays off if the arithmetic runs on the fastest available units, and that is a shape-and-alignment discipline as much as an algorithmic one.
Lower precision moves the ceiling up
The other way to raise the roof is to shrink the numbers. Halving the bit width roughly doubles peak FLOP/s, because the same silicon area packs twice as many narrower multiply-add lanes: FP32 → bf16/FP16 → FP8 → FP4 each step up the ceiling. This is why mixed-precision and quantized inference are not just memory optimizations — on a compute-bound kernel they directly buy arithmetic throughput.
Lower precision also cuts the bytes moved, so it helps memory-bound kernels too, but the two wins differ: on the compute roof FP8 makes the math faster; on the memory roof it makes the loads smaller. The catch is numerical — narrow formats need scaling and higher-precision accumulation, and validation that quality holds. When it does, dropping a precision level is often the single biggest lever on a compute-bound transformer.
Model FLOPs utilization: how close to the roof
Being compute-bound sets the ceiling; it does not guarantee you reach it. The metric that grades you is MFU (model FLOPs utilization): the ratio of useful model FLOP/s achieved to the hardware’s peak. MFU = achieved_FLOP_per_s / peak_FLOP_per_s. Well-tuned large-model training lands around 40–55%; the gap to 100% is launch overhead, non-matmul work (softmax, norms, elementwise), pipeline bubbles, and imperfect tensor-core packing.
MFU is the right dashboard for compute-bound work because it asks the only question that matters there: of the FLOPs the machine can do, how many are yours? A low MFU on a compute-bound job is fixable — bigger tiles, fused kernels, better overlap. A high MFU means you are near the wall and the only remaining move is to do fewer FLOPs. For memory-bound decode, MFU is intrinsically low and the honest metric is bandwidth utilization instead.
How to tell which regime you are in
Do not guess — measure, then reason. The quickest analytical check is the arithmetic-intensity estimate: count the FLOPs and the bytes for the hot kernel, take the ratio, and compare it to your hardware’s ridge point (P / B). Above it, compute-bound; below it, memory-bound. For a transformer, that usually means ‘is the batch/sequence dimension larger than a few hundred?’
Empirically, a profiler settles it: if the arithmetic units read near peak while memory bandwidth has headroom, you are compute-bound; if bandwidth is pinned near 100% while the math units idle, you are memory-bound. A quick sanity test is to halve the precision — if runtime drops roughly in proportion you were compute-bound; if it barely moves, memory or overhead dominated. Optimizing the wrong bottleneck wastes effort.
What this means on a CPU SLM
On a CPU running a small language model, the same physics apply with the numbers shifted. The FLOP ceiling is far lower (a few TFLOP/s from AVX-512 or AMX, versus hundreds on a GPU), but so is memory bandwidth, so the ridge point — and thus the batch size at which you become compute-bound — is a different value you must measure for your machine.
Two consequences dominate. First, prefill on a CPU is still compute-bound, so it benefits directly from AMX/VNNI matmul instructions and INT8/bf16 kernels that raise the modest CPU ceiling. Second, single-stream decode on a CPU is memory-bound just as on a GPU, so quantizing weights to 4-bit helps mainly by shrinking bytes moved, not by adding math throughput. The practical recipe for a CPU SLM is the familiar one: reach the ceiling for prefill with good matmul kernels, and attack decode as a bandwidth problem.
P / B, roughly 300 FLOP/byte on a modern GPU), so the multiply-add units, not memory, set the runtime. This is the regime you want, and it is governed by a hard FLOP ceiling you can only approach, never beat. Large matmuls, transformer prefill, and high-batch serving live here; single-token decode does not — its intensity is near one, and only batching pushes it back onto the compute roof. Once compute-bound, the levers narrow: do fewer FLOPs, or raise the ceiling with tensor cores and lower precision, and track how close you are with model FLOPs utilization. Above all, measure the arithmetic intensity before you optimize — tuning memory access on a compute-bound kernel, or counting FLOPs on a memory-bound one, is effort spent against the wrong wall.