LLM.int8() (Dettmers et al., 2022) is the trick that lets a large transformer do its heavy matrix multiplies in 8-bit integers at inference time with essentially no loss of accuracy — halving the memory of the weights and, on hardware with integer tensor units, roughly doubling matmul throughput. The core idea is a pair of moves. First, vector-wise absmax quantization: give every row of the activations and every column of the weights its own scale, so a single fat number in one place cannot ruin the whole tensor. Second, a mixed-precision decomposition that handles the one thing plain INT8 cannot: a small set of ‘emergent’ feature dimensions whose values explode past a certain model size. This piece derives the quantization arithmetic, works two numeric examples, and separates LLM.int8() from the weight-only schemes (GPTQ, AWQ) it is often lumped with.
Why 8-bit inference, and why it is hard
Weights and activations in a trained transformer are normally fp16 or bf16 — 16 bits each. Storing and moving them dominates both memory and, during decode, bandwidth. Dropping to INT8 halves the footprint and lets modern CPUs and GPUs use integer matmul paths that are markedly faster than floating point. The catch is precision: an 8-bit signed integer has only 256 levels (-127 … 127), so mapping a range of real numbers onto that grid throws away detail.
For most tensors that loss is tolerable, because the values cluster in a narrow band and 256 levels resolve them finely enough. The failure mode is dynamic range: if one value is far larger than the rest, the grid has to stretch to reach it, and every ordinary value collapses toward zero. LLM.int8() is essentially two answers to that dynamic-range problem — a finer granularity of scaling, and an escape hatch for the values that refuse to fit.
Absmax INT8 quantization from first principles
The scheme is absmax (symmetric) quantization. For a vector x you find its largest magnitude, map that to the top of the INT8 range, and scale everything else by the same factor:
absmax(x) = max_k |x_k|
s = 127 / absmax(x) (the scale)
Q(x) = round( s · x ) ∈ {-127, …, 127}
x̂ = Q(x) / s (dequantized approximation)Quantization is round(s · x); dequantization divides back out by s. The error per element is at most half a step, absmax(x) / 254, so a small absmax means a fine grid and a large absmax means a coarse one. Everything downstream turns on keeping each absmax small, which is exactly why the choice of which elements share a scale matters so much.
Vector-wise: a scale per row and per column
Consider the matmul at the heart of every linear layer, C = X W, with activations X: [s, h] (s tokens, h hidden) and weights W: [h, o]. A naive scheme uses one scale for all of X and one for all of W — per-tensor quantization. That forces millions of values to share a single absmax, so the largest element anywhere sets the grid for everything.
LLM.int8() instead uses vector-wise scaling: one scale c_x[i] per row of X (its own absmax), and one scale c_w[j] per column of W. Because the inner product for output C[i,j] runs along row i of X and column j of W, each contracted dimension carries a single, consistent scale — so the whole product can be denormalized afterward with one number. This gives s + o scales instead of two, at negligible cost, and shrinks each absmax dramatically.
A worked quantization example
Take one well-behaved row x = [0.4, -1.2, 3.0, 0.1]. Its absmax is 3.0, so s = 127 / 3.0 ≈ 42.33. Quantizing:
Q(x) = round(42.33 · [0.4, -1.2, 3.0, 0.1])
= round([16.9, -50.8, 127.0, 4.23])
= [17, -51, 127, 4]
x̂ = [17, -51, 127, 4] / 42.33
= [0.402, -1.205, 3.000, 0.094]Every value round-trips to within about 0.012 — three decimal digits of agreement. With a small absmax the 256-level grid is dense enough that even the tiny 0.1 survives. This is the regime plain INT8 handles beautifully, and it covers the overwhelming majority of rows and columns in a transformer. The trouble begins when one entry does not look like its neighbors.
Emergent outlier features at scale
Dettmers et al. found that once a transformer passes roughly 6.7B parameters, a small number of hidden-state feature dimensions begin to carry values 10–20× larger than everything else. These emergent outlier features are not random noise: they concentrate in a few specific dimensions (columns of X, and correspondingly rows of W), they recur in essentially every token’s row, and they spread across most layers as the model grows. The paper flags a dimension as an outlier when its magnitude exceeds a threshold of α = 6.0 and it appears across a large fraction of layers and sequence positions.
They also matter for accuracy: zeroing these few dimensions collapses model performance far more than removing any random set. So they cannot simply be clipped away — they must be represented faithfully. That is the crux the decomposition solves.
Why one outlier crushes a whole vector
Watch what an outlier does to a shared scale. Take x = [0.4, -1.2, 3.0, 60.0], where 60.0 is an outlier feature. Now absmax = 60, so s = 127 / 60 ≈ 2.12 — a 20× coarser grid than before:
Q(x) = round(2.12 · [0.4, -1.2, 3.0, 60.0])
= round([0.85, -2.54, 6.35, 127.0])
= [1, -3, 6, 127]
x̂ = [0.47, -1.42, 2.83, 60.0]The outlier itself is fine, but 0.4 now reconstructs as 0.47 (18% error) and -1.2 as -1.42. Because outliers live in fixed dimensions that recur in every row, row-wise absmax is hijacked in every row at once — the damage is systematic, not sporadic. The fix therefore pulls out whole outlier columns, not stray elements.
The mixed-precision decomposition
The key observation is that a matmul is a sum over the hidden dimension, so it splits cleanly. Let O be the set of outlier dimensions (those few columns of X whose magnitude exceeds α = 6.0). Then:
C = X W
= Σ_{k ∈ O} X[:,k] W[k,:] ← fp16 (outlier dims)
+ Σ_{k ∉ O} X[:,k] W[k,:] ← INT8 (regular dims)The outlier term is a thin fp16 matmul — typically at most a handful of the thousands of hidden dimensions (often ≤ 0.1% of them) — so it stays exact where precision is critical, at trivial compute cost. The regular term, holding the vast bulk of the dimensions, now contains no outliers, so its per-row and per-column absmax values are small and vector-wise INT8 quantizes it accurately. The full product is the sum of the two partial matmuls — one high-precision, one high-throughput.
Recombining: the dequantization arithmetic
The INT8 half is where the vector-wise scales pay off. Quantize the regular columns of X with row scales c_x and of W with column scales c_w, do the integer matmul into an int32 accumulator, then denormalize with the outer product:
A[i,j] = Σ_k Xq[i,k] · Wq[k,j] (int32 accumulate)
C_int8[i,j] ≈ (c_x[i] · c_w[j] / 127^2) · A[i,j]
C = dequant(C_int8) + C_fp16One outer product c_x ⊗ c_w (shape [s, o]) rescales the entire integer result in a single elementwise pass, because each output element was contracted along exactly one row-scale and one column-scale. Add the small fp16 outlier product on top and the reconstruction matches the original fp16 matmul to within noise — which is why LLM.int8() preserves zero-shot accuracy even at 175B parameters, with no retraining or calibration.
Not GPTQ, not AWQ: a different problem
LLM.int8() is frequently filed next to GPTQ and AWQ, but they solve different problems. GPTQ and AWQ are post-training, weight-only quantizers: offline, using a calibration set, they compress weights to 3–4 bits (GPTQ via second-order error compensation; AWQ by scaling up salient weight channels the activations care about). At inference the low-bit weights are dequantized on the fly and the matmul runs in fp16 — activations are never quantized.
LLM.int8() is inference-time and both-operand: it quantizes weights and activations to INT8 and executes the matmul in integer arithmetic, handling activation outliers dynamically by splitting them into fp16. It needs no calibration and no retraining. In short: GPTQ/AWQ shrink the stored weights for cheaper loading; LLM.int8() makes the actual matmul run in INT8 while protecting the outlier dimensions that would otherwise break it.
CPU-SLM implications and pitfalls
For small models on CPUs the news is mostly good. INT8 halves the weight memory and lets AVX-512 VNNI (or ARM dot-product) instructions do integer matmul at roughly double fp16 throughput — a real win for memory-bound decode on commodity hardware. And crucially, the emergent-outlier phenomenon is a scale effect: below about 6.7B parameters outliers are rare or absent, so a small language model often runs fine with plain vector-wise INT8 and no decomposition at all.
The pitfalls are practical. The mixed-precision path needs a gather/scatter of outlier columns plus a separate fp16 matmul, and on hardware without fast INT8 units that overhead can erase the speedup even as memory still drops. Vector-wise scaling also assumes outliers stay column-aligned — if they smeared across dimensions, the trick would fail. Measure on your target: the memory saving is reliable; the speedup is hardware-dependent.
s = 127 / absmax), so the integer product denormalizes cleanly through the outer product c_x ⊗ c_w / 127². And a mixed-precision decomposition splits the sum over the hidden dimension: the few emergent outlier dimensions that appear past ~6.7B parameters go through an exact fp16 matmul, the rest through INT8, and the two are added back. That is what separates it from GPTQ and AWQ — those are offline, weight-only compressors that keep the matmul in fp16, whereas LLM.int8() quantizes both operands and handles activation outliers on the fly. For sub-6.7B CPU models the outliers usually vanish, so plain vector-wise INT8 is often enough — but confirm the speedup on your hardware, because the memory win is certain and the throughput win is not.