SmoothQuant Math

The goal: W8A8, not just smaller weights

Quantization replaces high-precision numbers with low-precision integers plus a scale. For a tensor X, INT8 quantization picks a scale Δ so that X ≈ Δ · round(X / Δ), with the rounded values living in [-127, 127]. The payoff depends on what you quantize. Weight-only quantization (W8A16, W4A16) stores weights as integers but dequantizes them to FP16 before the matmul, so the arithmetic is still floating point — a memory and bandwidth win, not compute.

SmoothQuant aims at W8A8: both operands of the linear layer are INT8, so the general matrix multiply (GEMM) executes as an integer operation on hardware INT8 tensor cores, roughly doubling throughput versus FP16. That is a compute win, not just a storage win. The catch is that you now have to quantize activations too — and activations, unlike weights, are hostile to naive INT8. Understanding why is the whole reason SmoothQuant exists.

The activation-outlier problem

Weights in a trained transformer are well-behaved: their values sit in a narrow, roughly Gaussian range, so a single per-tensor (or per-output-channel) INT8 scale quantizes them with little error. Activations are not. In large language models, a handful of input channels carry values tens to hundreds of times larger than the rest, and those outlier channels are persistent: the same channel indices blow up across essentially every token.

This wrecks INT8. A per-tensor activation scale must be large enough to represent an outlier of magnitude, say, 100, which forces Δ ≈ 100/127. The ordinary channels, with values near 1, then round to 0 or 1 — almost all their information is destroyed. The distribution is simply too wide for 8 bits when one scale must cover everything. Giving each channel its own scale would fix the accuracy — but, as the next section shows, a per-input-channel activation scale is exactly the one thing INT8 GEMM cannot use.

Why per-channel activation scales don’t factor out

Consider a linear layer Y = X W with activations X: [N, C_in] (N tokens, C_in input channels) and weights W: [C_in, C_out]. Each output is a sum over the shared dimension: Y_ik = Σ_j X_ij · W_jk. For INT8 GEMM to work, the scales must factor out of that summation so the inner loop is pure integer multiply-accumulate.

Two placements factor out cleanly. A per-token activation scale Δ_i (one per row of X) pulls out on the i side: Y_ik = Δ_i Σ_j X̄_ij W_jk. A per-output-channel weight scale Δ_k pulls out on the k side. Both are outside the sum over j, so they are applied once, cheaply, after the integer GEMM. But the outliers live along j — specific input channels — and a per-input-channel scale Δ_j sits inside the summation, entangled with W_jk. It cannot be hoisted out, so it is incompatible with a single integer matmul. Activations are therefore stuck with per-tensor or per-token scales — which the outliers destroy.

The core idea: migrate difficulty into the weights

Here is the pivot. A per-input-channel scaling can be absorbed for free — not into the activations, but into the weights, where a per-input-channel factor is just a per-row rescale folded offline at load time. So instead of quantizing the spiky activations in place, SmoothQuant migrates the spikiness out of the activations and into the weights, which have precision to spare.

Introduce a smoothing vector s of length C_in, one entry per input channel. Divide activation channel j by s_j and multiply weight row j by the same s_j:

X̂ = X · diag(s)^-1     (shrink outlier activation channels)
Ŵ = diag(s) · W        (grow the matching weight rows)

Y = X̂ Ŵ = X diag(s)^-1 diag(s) W = X W   (unchanged)

The two diag(s) factors cancel, so the layer computes the same output. But now the smoothed activations X̂ have their outlier channels divided down toward the bulk, easy to quantize per-token, while the extra range lands in Ŵ, which quantizes fine per-output-channel.

The smoothing identity, channel by channel

It is worth seeing the cancellation at a single output element, because it makes clear that the shared index j is the same on both sides — the identity is not an approximation, it is exact algebra:

Y_ik = Σ_j X_ij W_jk
     = Σ_j (X_ij / s_j)(s_j W_jk)
     = Σ_j X̂_ij Ŵ_jk

Every term keeps the same s_j in numerator and denominator, so no approximation is introduced by the smoothing step itself — the only error comes from the subsequent INT8 rounding of X̂ and Ŵ, which is now far smaller because both are well-conditioned. Two details make this practically free. The diag(s)^-1 on activations is fused into the preceding operation — the LayerNorm or previous linear’s output scaling — so it costs nothing at runtime, and Ŵ = diag(s) W is computed once, offline, before the weights are quantized. At inference you simply feed already-smoothed, already-quantized tensors into an ordinary INT8 GEMM.

Choosing s: the migration strength alpha

How aggressively should each channel be smoothed? Push s_j too high and the weights inherit outliers of their own; too low and the activations stay spiky. SmoothQuant balances the two with a migration strength α ∈ [0, 1]:

s_j = max(|X_j|)^α / max(|W_j|)^(1 - α)

Here max(|X_j|) is the largest magnitude seen in activation channel j (measured on a small calibration set) and max(|W_j|) is the largest magnitude in weight row j. Read the exponents as a dial. At α = 0, s_j = 1/max(|W_j|) — all difficulty stays in the activations. At α = 1, everything is pushed into the weights. The default α = 0.5 splits the range as a geometric mean: the smoothed activation-channel max and the smoothed weight-row max both become √(max|X_j| · max|W_j|), so the two sides are equally hard to quantize. Models with harsher activation outliers benefit from a higher α, migrating more into the weights — but the sweet spot is a per-model calibration choice, not a universal constant.

A worked numeric example

Take one outlier input channel j. On the calibration set its activation magnitude peaks at max(|X_j|) = 100, while the corresponding weight row is tame, max(|W_j|) = 0.1. Naive per-tensor INT8 here is hopeless: the scale is dragged up to cover 100, and channels of magnitude ~1 collapse to a single bin.

Apply SmoothQuant with α = 0.5:

s_j = 100^0.5 / 0.1^0.5 = 10 / 0.3162 ≈ 31.6

smoothed activation max = 100 / 31.6   ≈ 3.16
smoothed weight-row max = 0.1 × 31.6  ≈ 3.16

Both sides land on √(100 · 0.1) = √10 ≈ 3.16 — the symmetry that α = 0.5 guarantees. The activation outlier has shrunk from 100 to about 3.16, roughly a 32× reduction, bringing it back among the ordinary channels so a per-token INT8 scale represents the whole row faithfully. The weight row grew from 0.1 to 3.16, still comfortably inside INT8’s per-output-channel range. The output is unchanged before rounding; after rounding, the error is a fraction of what naive quantization would have produced.