Roofline Model for LLM Ops

The whole model in one line

Every kernel does some arithmetic and moves some bytes. Call the work W (FLOPs) and the traffic Q (bytes crossing to the memory level you care about). Their ratio is the arithmetic intensity:

I = W / Q          [FLOP per byte]

P(I) = min( P_peak , BW · I )   [attainable FLOP/s]

That is the entire model. The machine can never issue more than P_peak FLOP/s nor feed itself faster than BW bytes per second, so a kernel of intensity I cannot exceed BW · I FLOP/s however good your code is. Plot P(I) on log-log axes and you get a slope-1 line flattening into a horizontal ceiling: a roof. The model, from Williams, Waterman and Patterson, is deliberately crude; its value is being an upper bound you can compute on paper, and upper bounds stop you optimizing the wrong thing.

The ridge point splits the world in two

The bend where slope meets ceiling is the ridge point. Left of it a kernel is bandwidth-bound: it finishes when its bytes arrive, and every FLOP beyond is idle capacity. Right of it, compute-bound. Two rules follow, and they are the only two you need: cutting FLOPs is worthless left of the ridge, cutting bytes right of it. Most wasted LLM optimization work is a right-of-ridge fix on a left-of-ridge kernel. Setting the two terms equal locates it:

I_ridge = P_peak / BW    (machine balance)

Our reference machine throughout: 8 cores at 3.5 GHz, AVX2 (8 fp32 lanes), two FMA ports. An FMA is 2 FLOPs per lane, so 8·2·2 = 32 FLOP/cycle/core and P_peak = 8·3.5e9·32 ≈ 896 GFLOP/s. Dual-channel DDR5-5600 gives 2·5600e6·8 = 89.6 GB/s, so I_ridge = 896/89.6 = 10 FLOP/byte. A datacenter accelerator (1000 TFLOP/s bf16, 3.3 TB/s HBM) sits near 300, thirty times harder to feed — the quiet argument for CPU small-model serving.

Building the two ceilings honestly

A roofline is only as good as its roof, and spec-sheet numbers are usually not the numbers you can reach. Peak FLOP/s assumes every core issues a full-width fused multiply-add every cycle at nominal clock — but sustained AVX-heavy code clocks down, and a kernel that cannot use FMA or fill the vector width loses a large multiple immediately. Peak bandwidth is likewise theoretical; a STREAM triad on real hardware lands at 70–85% of it.

So measure both: a small cache-resident FMA loop for the compute ceiling you can actually hit, STREAM or any large-array copy for the bandwidth you can actually sustain. A roofline from measured ceilings is a diagnostic; one from marketing numbers just tells everybody their kernel is at 8% of peak. The ridge moves with the measured numbers too: softer ceilings push it slightly left.

Worked example: a prefill GEMM

Take a small model with d = 2048, FFN width 8192, fp16, prefilling a 512-token prompt. The up-projection is C[512, 8192] = A[512, 2048] · B[2048, 8192].

FLOPs  W = 2·M·N·K = 2 · 512 · 8192 · 2048 = 17.2 GFLOP
bytes  Q = (512·2048 + 2048·8192 + 512·8192) · 2 B = 44.0 MB
I      = 17.2e9 / 44.0e6 ≈ 390 FLOP/byte

An intensity of 390 sits 39× to the right of our ridge of 10. The bandwidth roof there would be 89.6 · 390 ≈ 35 TFLOP/s, absurdly higher than the machine can compute, so the binding constraint is unambiguously P_peak. Prefill is compute-bound with room to spare, and its levers are vectorization, FMA utilization, thread scaling and tiling for cache reuse — not shrinking weights. Cutting bytes here buys you nothing.

Worked example: a decode-step matvec

Now generate one token. The same layer becomes M = 1: a matrix-vector product against the identical [2048, 8192] weight matrix.

FLOPs  W = 2 · 1 · 8192 · 2048 = 33.6 MFLOP
bytes  Q ≈ 2048·8192·2 B (weights dominate)  = 33.6 MB
I      = 33.6e6 / 33.6e6 ≈ 1.0 FLOP/byte

The result is not an accident of these dimensions but structural: a matvec touches every weight once and does 2 FLOPs with it, so in fp16 (2 bytes per weight) intensity is pinned at 2 FLOP / 2 B = 1 whatever the model size. Intensity 1 against a ridge of 10 means the attainable rate is min(896, 89.6 · 1) = 89.6 GFLOP/s: exactly 10% of peak. Ninety percent of the arithmetic hardware is structurally unreachable during decode, and no kernel tuning changes it — the limit is the weight sweep itself.

The same machine, both workloads, one plot

Drawn on the reference machine, the two land on opposite sides of the bend. Decode sits on the slanted roof, already optimal in the only sense that matters; prefill sits below the flat roof, which is the shape of a real optimization opportunity.

Reading the gap, and the roofs below the roof

The vertical gap from a plotted point to the roof above it is your headroom, in three flavours. No gap means you are done — only a horizontal move helps. A modest gap under the flat roof is classic kernel work: the prefill GEMM above measures 540 GFLOP/s against an 896 roof, 60% of peak — poor vectorization or thread imbalance. A large gap under the slanted roof is the interesting one — you are neither computing nor streaming at capacity, so the limit is elsewhere: dependent-load latency, NUMA traffic, or strided access wasting each cache line.

Sub-ceilings turn that diagnosis into a checklist: the peak without FMA (half), for scalar code at one FMA per cycle (8·3.5e9·2 = 56 GFLOP/s, sixteen times down), on one thread (an eighth). A point on the no-SIMD line is not a memory problem — the compiler failed to vectorize your inner loop. Precision adds its own family of roofs, so always say which one a plot uses.

Moving right: batching, fusion, quantization, tiling

Left of the ridge the only goal is to raise I. Batching is the big lever — with B concurrent sequences the weight bytes are read once and amortized over B rows of work, so I ≈ B in fp16. Our ridge of 10 needs a batch near ten; a ridge of 300 needs hundreds. Fusion removes intermediate round trips: a fused norm-plus-activation does the same FLOPs with less traffic. Quantization moves the point twice — int4 weights cut Q by 4×, lifting decode intensity to 4 FLOP/byte, and raise the compute roof where integer dot-products exist.

Tiling works on a different axis: the same construction applies at every memory level, each with its own bandwidth and slanted roof. Aggregate L2 bandwidth here is 1–2 TB/s, twenty times DRAM, putting the L2 ridge below 1 FLOP/byte. So the DRAM roofline says whether a kernel can go faster, the cache roofline whether blocking is working. For decode the weights dwarf any cache, and every token pays full DRAM price however you tile.

Why CPU SLM decode lives on the far left

Roll the per-layer result up to a whole model and the roofline turns into a throughput prediction. Autoregressive decode reads every parameter once per token, so:

tokens/s ≤ BW / (params · bytes_per_param)

1.5B params, fp16:  3.0 GB / token → 89.6 / 3.0   ≈  30 tok/s
1.5B params, int4:  0.75 GB / token → 89.6 / 0.75 ≈ 119 tok/s

Notice what is absent: FLOPs, core count, clock speed. At batch size one the CPU’s 896 GFLOP/s is nearly irrelevant — the model is a 3 GB array being streamed through the ALUs 30 times a second. This is why the single most effective CPU SLM optimization is weight quantization, why the second is anything that avoids a full sweep (speculative decoding, MoE routing, batching concurrent users), and why buying more cores does almost nothing. The ceiling is the memory bus.

What it misses, and how to use it anyway

Its blind spots matter too. It assumes you can saturate bandwidth given enough demand, but a chain of dependent loads — pointer chasing, tiny sequential kernels, a synchronous decode step — is latency-bound and sits far below the slanted roof with no bandwidth problem to fix. It ignores instruction mix, TLB pressure, synchronization overhead, and bus contention between cores. And it models one kernel at a time, while an inference server is a pipeline where the non-GEMM parts — sampling, tokenization, KV bookkeeping — can dominate a perfectly roofed matmul.

With that caveat the workflow is short: measure the two ceilings once per machine; per hot kernel, count W from the shapes and take Q from hardware counters, not from assumptions about what the cache did; divide, plot, read off the side and the gap. The payoff is arithmetic instead of argument — ‘60 GFLOP/s at intensity 1 on an 89.6 GB/s box’ means 1.5× of headroom, not 15×, so cap the effort and go raise intensity.