Activation Quantization

The quantization map: real numbers to integers

Quantization replaces a floating-point tensor with low-bit integers plus a small amount of metadata. The affine (asymmetric) scheme maps a real value x to an integer q using a scale s (a positive float) and a zero-point z (an integer):

quantize:    q = clamp( round(x / s) + z,  q_min,  q_max )
dequantize:  x_hat = s · (q - z)

For signed int8, q_min = -128 and q_max = 127. The symmetric special case sets z = 0 and uses a range [-127, 127], so q = clamp(round(x / s), -127, 127) and x_hat = s · q. The only free parameter is then the scale, usually chosen from the tensor’s dynamic range: s = max(|x|) / 127. Every value gets rounded to the nearest multiple of s, so s is the resolution — the gap between representable numbers. Choose s too large and small values vanish; too small and large values clip. The entire difficulty of activation quantization is choosing this one number well when the data keeps moving.

Weights are static; activations are not

Weights are frozen after training. You can compute max(|W|) once, pick a scale, and reuse it forever — the distribution never changes between one inference and the next. Weight matrices are also fairly well-behaved: after training they tend to be roughly bell-shaped and bounded, so a single scale per matrix (or per output channel) captures them with little error. Int8 weights, and even int4, are routinely deployed with negligible quality loss.

Activations are the intermediate results X = f(input) — the output of a LayerNorm, an attention block, a GELU. They exist only during the forward pass and are different for every input. A tensor X: [N, d] (N tokens, d channels) computed for one prompt has a different range than the next prompt’s. You cannot look at the true tensor before you compute it. So the fundamental problem is: pick a scale for numbers you have not seen yet, that vary per input, per token, and per channel — and be wrong by as little as possible.

Static vs dynamic quantization

There are two ways to obtain activation scales, trading accuracy against speed.

Static quantization fixes the scales ahead of time. You run a calibration pass over a few hundred representative samples, record the observed activation ranges at each layer, and freeze a scale (often a percentile or a moving-average max, not the raw max, to resist stray spikes). At inference the scale is a constant — no per-input work — so integer kernels run at full speed. The risk is distribution shift: an input whose activations exceed the calibrated range gets clipped.

Dynamic quantization computes the scale on the fly from the actual tensor: for each activation, take its real max(|x|) right before the matmul and quantize with it. This is always well-matched to the data, so accuracy is higher, but it costs a reduction over the tensor every layer and complicates fused kernels. A common compromise: dynamic per-token scales for activations (cheap, one max per row) with static per-channel weight scales.

Granularity: per-tensor, per-token, per-channel

The other lever is how many scales you use. One scale for the whole tensor is cheapest and coarsest; more scales fit the data better but cost more metadata and complicate the kernel.

Per-tensor: a single s for all of X: [N, d]. One outlier anywhere inflates it for everything. Per-token: one scale per row, s_n for token n. This is the natural granularity for activations, and it is nearly free in the GEMM. Consider Y = X W, Y[n,j] = Σ_k X[n,k] W[k,j]. With per-token scaling X[n,k] = s_n · Xq[n,k], the factor pulls straight out: Y[n,j] = s_n · Σ_k Xq[n,k] W[k,j]. Per-channel (one scale per feature k) is the natural fit for the outlier problem — but for activations k is the reduction dimension, so s_k sits inside the sum and will not factor out of an integer matmul. That mismatch is central to what follows.

The outlier-channel phenomenon

Here is what actually breaks activation quantization. In transformers — especially once they pass a few billion parameters — a small, fixed set of feature channels carries values 10× to 100× larger than the rest. These outliers are systematic: they appear in the same channel indices across nearly all tokens, they emerge during training, and they matter for accuracy, so you cannot simply clip them away. LLM.int8() first documented this: past roughly the 6.7B-parameter scale, a handful of dimensions dominate the activation magnitude.

Now the granularity trap bites. Per-tensor and per-token scales are both driven by the max magnitude in their group, so an outlier of 25 sets s = 25/127 ≈ 0.197 — and every ordinary value near 0.1 collapses to 0 or one quantization step. The precision that should have gone to the bulk of the signal is spent representing a few giant numbers. The granularity that would isolate the outliers — per-channel — is exactly the one that does not factor out of the activation matmul. That is the bind.

A worked example

Take one activation row with a single outlier channel:

x = [ 0.10, -0.30, 0.20, 25.00, 0.15, -0.05 ]
symmetric int8:  s = max(|x|)/127 = 25.00/127 = 0.19685

Quantize the ordinary values against that outlier-driven scale:

0.20  -> round(0.20/0.19685)=round(1.02)=1  -> x_hat = 0.197   (err ~ -0.003, but only 1 level!)
0.10  -> round(0.51)=1                      -> x_hat = 0.197   (err ~ +97%)
0.15  -> round(0.76)=1                      -> x_hat = 0.197
-0.05 -> round(-0.25)=0                     -> x_hat = 0.000   (destroyed)
-0.30 -> round(-1.52)=-2                    -> x_hat = -0.394  (err ~ +31%)

Three distinct small values (0.10, 0.15, 0.20) all round to the same code — the signal is gone. Now quantize the same row without the outlier: max(|x|)=0.30, s = 0.30/127 = 0.00236, and 0.20 -> round(84.7)=85 -> 0.2008, essentially exact. One channel out of thousands cost the layer almost all of its int8 precision. Per-token scaling would not help here: the outlier lives inside the token being scaled.

Why W8A8 needs activation quantization at all

You might ask why not just keep activations in fp16 and quantize only weights (a W8A16 scheme). W8A16 does cut the model’s memory footprint and helps the memory-bound decode phase, and it sidesteps the outlier problem entirely. But it leaves the arithmetic in floating point: the matmul is int8 weight against fp16 activation, so you cannot use the hardware’s fast integer GEMM units.

W8A8 — 8-bit weights and 8-bit activations — is what unlocks true int8 matrix multiply: roughly 2× the throughput and half the memory traffic of fp16 on hardware with int8 tensor units, which is exactly the win you want in the compute-bound prefill phase and on int8-friendly CPUs. The catch is that both operands must be integers, so the activations must be quantized too — and that drags in the outlier problem in full. W8A8 is therefore the scheme that makes activation quantization unavoidable rather than optional.