Why 8-bit inference, and why it is hard
Weights and activations in a trained transformer are normally fp16 or bf16 — 16 bits each. Storing and moving them dominates both memory and, during decode, bandwidth. Dropping to INT8 halves the footprint and lets modern CPUs and GPUs use integer matmul paths that are markedly faster than floating point. The catch is precision: an 8-bit signed integer has only 256 levels (-127 … 127), so mapping a range of real numbers onto that grid throws away detail.
For most tensors that loss is tolerable, because the values cluster in a narrow band and 256 levels resolve them finely enough. The failure mode is dynamic range: if one value is far larger than the rest, the grid has to stretch to reach it, and every ordinary value collapses toward zero. LLM.int8() is essentially two answers to that dynamic-range problem — a finer granularity of scaling, and an escape hatch for the values that refuse to fit.
Absmax INT8 quantization from first principles
The scheme is absmax (symmetric) quantization. For a vector x you find its largest magnitude, map that to the top of the INT8 range, and scale everything else by the same factor:
absmax(x) = max_k |x_k|
s = 127 / absmax(x) (the scale)
Q(x) = round( s · x ) ∈ {-127, …, 127}
x̂ = Q(x) / s (dequantized approximation)Quantization is round(s · x); dequantization divides back out by s. The error per element is at most half a step, absmax(x) / 254, so a small absmax means a fine grid and a large absmax means a coarse one. Everything downstream turns on keeping each absmax small, which is exactly why the choice of which elements share a scale matters so much.
Vector-wise: a scale per row and per column
Consider the matmul at the heart of every linear layer, C = X W, with activations X: [s, h] (s tokens, h hidden) and weights W: [h, o]. A naive scheme uses one scale for all of X and one for all of W — per-tensor quantization. That forces millions of values to share a single absmax, so the largest element anywhere sets the grid for everything.
LLM.int8() instead uses vector-wise scaling: one scale c_x[i] per row of X (its own absmax), and one scale c_w[j] per column of W. Because the inner product for output C[i,j] runs along row i of X and column j of W, each contracted dimension carries a single, consistent scale — so the whole product can be denormalized afterward with one number. This gives s + o scales instead of two, at negligible cost, and shrinks each absmax dramatically.
A worked quantization example
Take one well-behaved row x = [0.4, -1.2, 3.0, 0.1]. Its absmax is 3.0, so s = 127 / 3.0 ≈ 42.33. Quantizing:
Q(x) = round(42.33 · [0.4, -1.2, 3.0, 0.1])
= round([16.9, -50.8, 127.0, 4.23])
= [17, -51, 127, 4]
x̂ = [17, -51, 127, 4] / 42.33
= [0.402, -1.205, 3.000, 0.094]Every value round-trips to within about 0.012 — three decimal digits of agreement. With a small absmax the 256-level grid is dense enough that even the tiny 0.1 survives. This is the regime plain INT8 handles beautifully, and it covers the overwhelming majority of rows and columns in a transformer. The trouble begins when one entry does not look like its neighbors.
Emergent outlier features at scale
Dettmers et al. found that once a transformer passes roughly 6.7B parameters, a small number of hidden-state feature dimensions begin to carry values 10–20× larger than everything else. These emergent outlier features are not random noise: they concentrate in a few specific dimensions (columns of X, and correspondingly rows of W), they recur in essentially every token’s row, and they spread across most layers as the model grows. The paper flags a dimension as an outlier when its magnitude exceeds a threshold of α = 6.0 and it appears across a large fraction of layers and sequence positions.
They also matter for accuracy: zeroing these few dimensions collapses model performance far more than removing any random set. So they cannot simply be clipped away — they must be represented faithfully. That is the crux the decomposition solves.
Why one outlier crushes a whole vector
Watch what an outlier does to a shared scale. Take x = [0.4, -1.2, 3.0, 60.0], where 60.0 is an outlier feature. Now absmax = 60, so s = 127 / 60 ≈ 2.12 — a 20× coarser grid than before:
Q(x) = round(2.12 · [0.4, -1.2, 3.0, 60.0])
= round([0.85, -2.54, 6.35, 127.0])
= [1, -3, 6, 127]
x̂ = [0.47, -1.42, 2.83, 60.0]The outlier itself is fine, but 0.4 now reconstructs as 0.47 (18% error) and -1.2 as -1.42. Because outliers live in fixed dimensions that recur in every row, row-wise absmax is hijacked in every row at once — the damage is systematic, not sporadic. The fix therefore pulls out whole outlier columns, not stray elements.
The mixed-precision decomposition
The key observation is that a matmul is a sum over the hidden dimension, so it splits cleanly. Let O be the set of outlier dimensions (those few columns of X whose magnitude exceeds α = 6.0). Then:
C = X W
= Σ_{k ∈ O} X[:,k] W[k,:] ← fp16 (outlier dims)
+ Σ_{k ∉ O} X[:,k] W[k,:] ← INT8 (regular dims)The outlier term is a thin fp16 matmul — typically at most a handful of the thousands of hidden dimensions (often ≤ 0.1% of them) — so it stays exact where precision is critical, at trivial compute cost. The regular term, holding the vast bulk of the dimensions, now contains no outliers, so its per-row and per-column absmax values are small and vector-wise INT8 quantizes it accurately. The full product is the sum of the two partial matmuls — one high-precision, one high-throughput.