SmoothQuant Math

The goal: W8A8, not just smaller weights

Quantization replaces high-precision numbers with low-precision integers plus a scale. For a tensor X, INT8 quantization picks a scale Δ so that X ≈ Δ · round(X / Δ), with the rounded values living in [-127, 127]. The payoff depends on what you quantize. Weight-only quantization (W8A16, W4A16) stores weights as integers but dequantizes them to FP16 before the matmul, so the arithmetic is still floating point — a memory and bandwidth win, not compute.

SmoothQuant aims at W8A8: both operands of the linear layer are INT8, so the general matrix multiply (GEMM) executes as an integer operation on hardware INT8 tensor cores, roughly doubling throughput versus FP16. That is a compute win, not just a storage win. The catch is that you now have to quantize activations too — and activations, unlike weights, are hostile to naive INT8. Understanding why is the whole reason SmoothQuant exists.

The activation-outlier problem

Weights in a trained transformer are well-behaved: their values sit in a narrow, roughly Gaussian range, so a single per-tensor (or per-output-channel) INT8 scale quantizes them with little error. Activations are not. In large language models, a handful of input channels carry values tens to hundreds of times larger than the rest, and those outlier channels are persistent: the same channel indices blow up across essentially every token.

This wrecks INT8. A per-tensor activation scale must be large enough to represent an outlier of magnitude, say, 100, which forces Δ ≈ 100/127. The ordinary channels, with values near 1, then round to 0 or 1 — almost all their information is destroyed. The distribution is simply too wide for 8 bits when one scale must cover everything. Giving each channel its own scale would fix the accuracy — but, as the next section shows, a per-input-channel activation scale is exactly the one thing INT8 GEMM cannot use.

Why per-channel activation scales don’t factor out

Consider a linear layer Y = X W with activations X: [N, C_in] (N tokens, C_in input channels) and weights W: [C_in, C_out]. Each output is a sum over the shared dimension: Y_ik = Σ_j X_ij · W_jk. For INT8 GEMM to work, the scales must factor out of that summation so the inner loop is pure integer multiply-accumulate.

Two placements factor out cleanly. A per-token activation scale Δ_i (one per row of X) pulls out on the i side: Y_ik = Δ_i Σ_j X̄_ij W_jk. A per-output-channel weight scale Δ_k pulls out on the k side. Both are outside the sum over j, so they are applied once, cheaply, after the integer GEMM. But the outliers live along j — specific input channels — and a per-input-channel scale Δ_j sits inside the summation, entangled with W_jk. It cannot be hoisted out, so it is incompatible with a single integer matmul. Activations are therefore stuck with per-tensor or per-token scales — which the outliers destroy.

The core idea: migrate difficulty into the weights

Here is the pivot. A per-input-channel scaling can be absorbed for free — not into the activations, but into the weights, where a per-input-channel factor is just a per-row rescale folded offline at load time. So instead of quantizing the spiky activations in place, SmoothQuant migrates the spikiness out of the activations and into the weights, which have precision to spare.

Introduce a smoothing vector s of length C_in, one entry per input channel. Divide activation channel j by s_j and multiply weight row j by the same s_j:

X̂ = X · diag(s)^-1     (shrink outlier activation channels)
Ŵ = diag(s) · W        (grow the matching weight rows)

Y = X̂ Ŵ = X diag(s)^-1 diag(s) W = X W   (unchanged)

The two diag(s) factors cancel, so the layer computes the same output. But now the smoothed activations X̂ have their outlier channels divided down toward the bulk, easy to quantize per-token, while the extra range lands in Ŵ, which quantizes fine per-output-channel.

The smoothing identity, channel by channel

It is worth seeing the cancellation at a single output element, because it makes clear that the shared index j is the same on both sides — the identity is not an approximation, it is exact algebra:

Y_ik = Σ_j X_ij W_jk
     = Σ_j (X_ij / s_j)(s_j W_jk)
     = Σ_j X̂_ij Ŵ_jk

Every term keeps the same s_j in numerator and denominator, so no approximation is introduced by the smoothing step itself — the only error comes from the subsequent INT8 rounding of X̂ and Ŵ, which is now far smaller because both are well-conditioned. Two details make this practically free. The diag(s)^-1 on activations is fused into the preceding operation — the LayerNorm or previous linear’s output scaling — so it costs nothing at runtime, and Ŵ = diag(s) W is computed once, offline, before the weights are quantized. At inference you simply feed already-smoothed, already-quantized tensors into an ordinary INT8 GEMM.

Choosing s: the migration strength alpha

How aggressively should each channel be smoothed? Push s_j too high and the weights inherit outliers of their own; too low and the activations stay spiky. SmoothQuant balances the two with a migration strength α ∈ [0, 1]:

s_j = max(|X_j|)^α / max(|W_j|)^(1 - α)

Here max(|X_j|) is the largest magnitude seen in activation channel j (measured on a small calibration set) and max(|W_j|) is the largest magnitude in weight row j. Read the exponents as a dial. At α = 0, s_j = 1/max(|W_j|) — all difficulty stays in the activations. At α = 1, everything is pushed into the weights. The default α = 0.5 splits the range as a geometric mean: the smoothed activation-channel max and the smoothed weight-row max both become √(max|X_j| · max|W_j|), so the two sides are equally hard to quantize. Models with harsher activation outliers benefit from a higher α, migrating more into the weights — but the sweet spot is a per-model calibration choice, not a universal constant.

A worked numeric example

Take one outlier input channel j. On the calibration set its activation magnitude peaks at max(|X_j|) = 100, while the corresponding weight row is tame, max(|W_j|) = 0.1. Naive per-tensor INT8 here is hopeless: the scale is dragged up to cover 100, and channels of magnitude ~1 collapse to a single bin.

Apply SmoothQuant with α = 0.5:

s_j = 100^0.5 / 0.1^0.5 = 10 / 0.3162 ≈ 31.6

smoothed activation max = 100 / 31.6   ≈ 3.16
smoothed weight-row max = 0.1 × 31.6  ≈ 3.16

Both sides land on √(100 · 0.1) = √10 ≈ 3.16 — the symmetry that α = 0.5 guarantees. The activation outlier has shrunk from 100 to about 3.16, roughly a 32× reduction, bringing it back among the ordinary channels so a per-token INT8 scale represents the whole row faithfully. The weight row grew from 0.1 to 3.16, still comfortably inside INT8’s per-output-channel range. The output is unchanged before rounding; after rounding, the error is a fraction of what naive quantization would have produced.

Why this makes INT8 GEMM viable

Stack the pieces and the payoff is concrete. After smoothing, activations quantize well per-token (factoring out on the token side) and weights quantize well per-output-channel (factoring out on the output side). Neither scale sits inside the summation over j, so the inner product is a pure INT8 multiply-accumulate into an INT32 accumulator, followed by a single dequantization:

Y_ik ≈ Δ^x_i · Δ^w_k · Σ_j X̂^int_ij Ŵ^int_jk

The heavy inner sum runs entirely in integers on INT8 tensor cores; the two scalar scales Δ^x_i and Δ^w_k are applied once per output element, outside the loop. Because the smoothing is folded away, there is no runtime overhead beyond the GEMM itself. The result is the full W8A8 dividend — roughly 2× faster matmuls and about half the memory footprint of FP16 — without the accuracy cliff that naive activation quantization falls off.

Not GPTQ, not AWQ: joint weight-and-activation quantization

It is easy to lump SmoothQuant with GPTQ and AWQ, but they solve different problems. GPTQ is weight-only: it quantizes weights to low bit-widths (often 4-bit) using second-order error compensation, and activations stay in FP16. AWQ is also weight-only and a close cousin of SmoothQuant — it too uses activation statistics and per-channel scaling to protect salient weight channels — but its output is W4A16.

The sharp distinction is the A. Weight-only methods (W4A16, W8A16) reduce memory and bandwidth, but the matmul still runs in FP16: the quantized weights are dequantized on the fly, so the win is storage and load time, not arithmetic throughput. SmoothQuant keeps activations in INT8 too, so the GEMM is a genuine integer operation on tensor cores — a compute-and-throughput win. That is the axis that separates them: weight-only shrinks the model; SmoothQuant speeds up the math. The choice follows the bottleneck — memory-bound decode favors weight-only quantization, while compute-bound prefill and batched serving reward the INT8 GEMM that W8A8 unlocks.

Practical notes and pitfalls

A few things decide whether SmoothQuant helps in practice. Calibration matters: the per-channel max(|X_j|) is estimated from a small sample of representative inputs, so an unrepresentative set gives wrong smoothing factors and leaves outliers un-migrated — a few hundred target-distribution sequences is usually enough. Alpha is a knob, not a constant: 0.5 is a strong default, but outlier-heavy models may need it higher, so sweep a few values against a validation metric.

Fold the scales in the right place: the activation diag(s)^-1 must be absorbed into a preceding LayerNorm or linear — applying it as a separate elementwise op would erase part of the speedup. Finally, remember what SmoothQuant does not do: it does not reduce bit-width below 8, and it does not remove outliers, it relocates them. If a model’s weights already sit near the edge of INT8 range, migrating even more magnitude into them can backfire — which is precisely why the balanced α exists. Used within its regime, it is one of the cleanest routes to real INT8 inference on transformers.