Weight-only quantization and where it breaks

Large-model inference is dominated by the cost of loading weights, not by arithmetic, so shrinking weights from 16 bits to 4 is the highest-leverage win for a CPU or a small GPU. Weight-only quantization keeps activations in floating point and stores each weight as a low-bit integer plus a shared scale. For a group of weights the quantizer is Q(w) = Δ · round(w / Δ), with step size Δ = max(|w|) / 2^(N-1) for N bits. The rounding introduces an error of up to Δ/2 per weight.

The trouble is that Δ is set by the largest magnitude in the group. A handful of weights that sit on important channels get the same coarse step as everything else, and round-to-nearest smears them. Because those few channels contribute a large share of the layer’s output, their error propagates and perplexity jumps. The naive fix — keep the important weights in FP16 and quantize the rest — works numerically but produces mixed-precision matrices that are slow and awkward on real hardware. AWQ’s contribution is getting the same protection with a uniform 4-bit layout.

Advertisement

Saliency is an activation property, not a weight property

The first question is which weights to protect. A tempting answer is ‘the ones with the largest magnitude,’ but that is wrong. What a layer produces is y = W x; the contribution of input channel j to the output is W[:, j] · x_j. A weight column can be small yet still dominate the output if the activation x_j flowing through it is consistently large.

AWQ therefore ranks input channels by the magnitude of the activations that pass through them, measured as the average |x_j| over a small calibration set. Empirically, a very small subset of channels — roughly 0.1% to 1% — carries outsized activation, and protecting just those recovers almost all the lost accuracy. This is the ‘activation-aware’ in the name: the metric that decides which weights are salient is read off the activations, so calibration data (a few hundred sequences) is needed to estimate the per-channel activation statistics. Weight magnitude alone tells you nothing about which channels the network actually leans on.

Advertisement

The equivalent transform: scale up W, scale down x

Once you know which input channels are salient, you want to reduce their quantization error without touching the layer’s output. AWQ does this with a per-input-channel scale s_j > 1. Multiply column j of the weight matrix by s_j and divide the corresponding activation by the same s_j:

y = W x
  = Σ_j  W[:, j] · x_j
  = Σ_j  (W[:, j] · s_j) · (x_j / s_j)

Let  W' = W · diag(s),   x' = diag(s)^-1 · x
Then y = W' x'   (exactly, in full precision)

In exact arithmetic the two scalings cancel and the output is unchanged. The point is that quantization happens after the transform: we store Q(W'), not Q(W). The activation scaling 1/s_j can be folded backward into the previous layer’s weights (for example the preceding LayerNorm or linear), so at inference time there is no extra runtime division — the model runs as ordinary quantized matmuls. The transform is free at inference and only changes what gets rounded.

Why scaling shrinks the error that matters

Look at one salient weight w multiplied by activation x. The output error from quantizing it is roughly the step size times a rounding fraction times the activation:

Err( Q(w) · x )  ≈  Δ · RoundErr · |x|
   where RoundErr ∈ [0, 0.5],  E[RoundErr] ≈ 0.25

After scaling channel by s:
Err( Q(w · s) · (x / s) )  ≈  Δ' · RoundErr · |x| / s

The key is what happens to Δ. Because only a few channels are scaled and the group’s maximum magnitude is set by other, larger weights, scaling one salient channel up by a modest s barely moves max(|w|), so Δ' ≈ Δ. The activation term, however, is divided by s. Net effect: the error contributed by the salient channel drops by a factor of about 1/s. You are spending a little precision on the many unimportant channels (whose relative error rises slightly) to buy a large error reduction on the few that dominate the output. That asymmetry is exactly why it is a net win.

A worked numeric example

Take a 4-bit group (N = 4, so 2^(N-1) = 8) whose largest weight magnitude is 1.0. The step is Δ = 1.0 / 8 = 0.125. Suppose a salient channel has a small weight w = 0.1 but a large average activation |x| = 10.

Before scaling:
  Err ≈ Δ · 0.25 · |x| = 0.125 · 0.25 · 10 = 0.3125

Scale this channel by s = 2:  w → 0.2,  x → 5
  group max still ≈ 1.0  ⇒  Δ' ≈ 0.125
  Err ≈ 0.125 · 0.25 · 5 = 0.15625

The salient channel’s output error is halved — matching the 1/s prediction — while the rest of the group is essentially untouched because Δ did not change. Push s too high, though, and two things go wrong: the scaled salient weight starts to raise max(|w|) (inflating Δ for the whole group), and the un-scaled channels now sit lower in the range and lose relative precision. There is an optimum s that trades these off, which is exactly what AWQ searches for.

Searching the scale — no backprop required

AWQ does not learn the scales by gradient descent. It parameterizes the per-channel scale as a simple function of the measured activation magnitude and searches a single knob. A common form is:

s_j = (s_X,j) ^ α,     α ∈ [0, 1]
  s_X,j = average |x_j| over the calibration set

choose α* = argmin_α  || W x  -  Q(W · diag(s)) · diag(s)^-1 x ||

At α = 0 every scale is 1 (plain quantization); at α = 1 scales track activation magnitude fully. AWQ sweeps a small grid of α values, and for each one it quantizes the layer, runs the calibration activations through, and measures the mean-squared error between the original FP16 output and the quantized output. The α with the lowest error wins. Everything here is a forward pass — no gradients, no optimizer, no weight updates. That makes AWQ cheap (minutes, a few hundred calibration samples), deterministic, and robust: because it fits one scalar per layer rather than tuning individual weights, it does not overfit the calibration set and it generalizes across domains.