Activation Quantization

The quantization map: real numbers to integers

Quantization replaces a floating-point tensor with low-bit integers plus a small amount of metadata. The affine (asymmetric) scheme maps a real value x to an integer q using a scale s (a positive float) and a zero-point z (an integer):

quantize:    q = clamp( round(x / s) + z,  q_min,  q_max )
dequantize:  x_hat = s · (q - z)

For signed int8, q_min = -128 and q_max = 127. The symmetric special case sets z = 0 and uses a range [-127, 127], so q = clamp(round(x / s), -127, 127) and x_hat = s · q. The only free parameter is then the scale, usually chosen from the tensor’s dynamic range: s = max(|x|) / 127. Every value gets rounded to the nearest multiple of s, so s is the resolution — the gap between representable numbers. Choose s too large and small values vanish; too small and large values clip. The entire difficulty of activation quantization is choosing this one number well when the data keeps moving.

Weights are static; activations are not

Weights are frozen after training. You can compute max(|W|) once, pick a scale, and reuse it forever — the distribution never changes between one inference and the next. Weight matrices are also fairly well-behaved: after training they tend to be roughly bell-shaped and bounded, so a single scale per matrix (or per output channel) captures them with little error. Int8 weights, and even int4, are routinely deployed with negligible quality loss.

Activations are the intermediate results X = f(input) — the output of a LayerNorm, an attention block, a GELU. They exist only during the forward pass and are different for every input. A tensor X: [N, d] (N tokens, d channels) computed for one prompt has a different range than the next prompt’s. You cannot look at the true tensor before you compute it. So the fundamental problem is: pick a scale for numbers you have not seen yet, that vary per input, per token, and per channel — and be wrong by as little as possible.

Static vs dynamic quantization

There are two ways to obtain activation scales, trading accuracy against speed.

Static quantization fixes the scales ahead of time. You run a calibration pass over a few hundred representative samples, record the observed activation ranges at each layer, and freeze a scale (often a percentile or a moving-average max, not the raw max, to resist stray spikes). At inference the scale is a constant — no per-input work — so integer kernels run at full speed. The risk is distribution shift: an input whose activations exceed the calibrated range gets clipped.

Dynamic quantization computes the scale on the fly from the actual tensor: for each activation, take its real max(|x|) right before the matmul and quantize with it. This is always well-matched to the data, so accuracy is higher, but it costs a reduction over the tensor every layer and complicates fused kernels. A common compromise: dynamic per-token scales for activations (cheap, one max per row) with static per-channel weight scales.

Granularity: per-tensor, per-token, per-channel

The other lever is how many scales you use. One scale for the whole tensor is cheapest and coarsest; more scales fit the data better but cost more metadata and complicate the kernel.

Per-tensor: a single s for all of X: [N, d]. One outlier anywhere inflates it for everything. Per-token: one scale per row, s_n for token n. This is the natural granularity for activations, and it is nearly free in the GEMM. Consider Y = X W, Y[n,j] = Σ_k X[n,k] W[k,j]. With per-token scaling X[n,k] = s_n · Xq[n,k], the factor pulls straight out: Y[n,j] = s_n · Σ_k Xq[n,k] W[k,j]. Per-channel (one scale per feature k) is the natural fit for the outlier problem — but for activations k is the reduction dimension, so s_k sits inside the sum and will not factor out of an integer matmul. That mismatch is central to what follows.

The outlier-channel phenomenon

Here is what actually breaks activation quantization. In transformers — especially once they pass a few billion parameters — a small, fixed set of feature channels carries values 10× to 100× larger than the rest. These outliers are systematic: they appear in the same channel indices across nearly all tokens, they emerge during training, and they matter for accuracy, so you cannot simply clip them away. LLM.int8() first documented this: past roughly the 6.7B-parameter scale, a handful of dimensions dominate the activation magnitude.

Now the granularity trap bites. Per-tensor and per-token scales are both driven by the max magnitude in their group, so an outlier of 25 sets s = 25/127 ≈ 0.197 — and every ordinary value near 0.1 collapses to 0 or one quantization step. The precision that should have gone to the bulk of the signal is spent representing a few giant numbers. The granularity that would isolate the outliers — per-channel — is exactly the one that does not factor out of the activation matmul. That is the bind.

A worked example

Take one activation row with a single outlier channel:

x = [ 0.10, -0.30, 0.20, 25.00, 0.15, -0.05 ]
symmetric int8:  s = max(|x|)/127 = 25.00/127 = 0.19685

Quantize the ordinary values against that outlier-driven scale:

0.20  -> round(0.20/0.19685)=round(1.02)=1  -> x_hat = 0.197   (err ~ -0.003, but only 1 level!)
0.10  -> round(0.51)=1                      -> x_hat = 0.197   (err ~ +97%)
0.15  -> round(0.76)=1                      -> x_hat = 0.197
-0.05 -> round(-0.25)=0                     -> x_hat = 0.000   (destroyed)
-0.30 -> round(-1.52)=-2                    -> x_hat = -0.394  (err ~ +31%)

Three distinct small values (0.10, 0.15, 0.20) all round to the same code — the signal is gone. Now quantize the same row without the outlier: max(|x|)=0.30, s = 0.30/127 = 0.00236, and 0.20 -> round(84.7)=85 -> 0.2008, essentially exact. One channel out of thousands cost the layer almost all of its int8 precision. Per-token scaling would not help here: the outlier lives inside the token being scaled.

Why W8A8 needs activation quantization at all

You might ask why not just keep activations in fp16 and quantize only weights (a W8A16 scheme). W8A16 does cut the model’s memory footprint and helps the memory-bound decode phase, and it sidesteps the outlier problem entirely. But it leaves the arithmetic in floating point: the matmul is int8 weight against fp16 activation, so you cannot use the hardware’s fast integer GEMM units.

W8A8 — 8-bit weights and 8-bit activations — is what unlocks true int8 matrix multiply: roughly 2× the throughput and half the memory traffic of fp16 on hardware with int8 tensor units, which is exactly the win you want in the compute-bound prefill phase and on int8-friendly CPUs. The catch is that both operands must be integers, so the activations must be quantized too — and that drags in the outlier problem in full. W8A8 is therefore the scheme that makes activation quantization unavoidable rather than optional.

SmoothQuant: move the difficulty into the weights

SmoothQuant resolves the bind with an algebraic trick. Since Y = X W is unchanged if you divide the activation channels by a vector and multiply the corresponding weight rows by the same vector, insert a per-channel smoothing factor s_j:

X W = ( X · diag(s)^-1 ) ( diag(s) · W )
     =        X_hat        ·      W_hat
choose  s_j = max(|X_j|)^α / max(|W_j|)^(1-α)      (α ~ 0.5)

Dividing by s_j shrinks the outlier channels of X until the activations are smooth and per-tensor/per-token int8 quantizes them cleanly. The magnitude does not disappear — it is migrated into W_hat, but weights are static, well-behaved, and quantize fine per-channel, so they absorb it easily. The exponent α tunes how much difficulty each side carries. The result is a mathematically equivalent network whose activations are now quantization-friendly, making accurate W8A8 practical — the pointer to follow when the outlier example above stops you.

CPU and small-model implications

On a CPU running a small language model, int8 is not a luxury — it is often the difference between usable and not. Commodity CPUs have int8 dot-product instructions (AVX-512 VNNI, ARM dot-product) that run several times faster than fp32 and move half the bytes, so a W8A8 SLM can hit interactive latencies where fp32 would crawl. Memory bandwidth is usually the binding constraint on CPU decode, and halving activation width directly relieves it.

The practical recipe that survives contact with real models: static per-channel scales for weights (cheap, accurate, done offline); dynamic per-token scales for activations (one max per row, factors out of the GEMM); and a SmoothQuant-style pre-pass to tame outlier channels so the per-token int8 range is not hijacked. Calibrate on data that resembles production, use percentile clipping rather than raw max, and keep the numerically sensitive spots — often the LayerNorm and the final projection — in higher precision if the accuracy budget is tight.

Pitfalls and what to measure

The failure modes are specific. Calibrating on the raw max lets a single freak activation set a scale that clips everything else — prefer a 99.9th-percentile or moving-average estimator. Ignoring outlier channels and hoping per-token scaling saves you: it will not, because the outlier sits inside the token. Calibration set too narrow: static scales tuned on one domain clip on another, so calibrate broadly. Symmetric quantization on a skewed, post-GELU activation wastes half the integer range on values that never occur — asymmetric (with a zero-point) fits one-sided distributions better.

Do not trust weight-only int8 results to predict W8A8 quality; the activation path is where accuracy is actually lost. Measure end-to-end task metrics, not just mean-squared quantization error, and inspect per-layer activation histograms to find which layers hold the outliers. The layers that break are usually few — fix those, quantize the rest aggressively, and W8A8 becomes a clean, fast win.