AWQ Math

Weight-only quantization and where it breaks

Large-model inference is dominated by the cost of loading weights, not by arithmetic, so shrinking weights from 16 bits to 4 is the highest-leverage win for a CPU or a small GPU. Weight-only quantization keeps activations in floating point and stores each weight as a low-bit integer plus a shared scale. For a group of weights the quantizer is Q(w) = Δ · round(w / Δ), with step size Δ = max(|w|) / 2^(N-1) for N bits. The rounding introduces an error of up to Δ/2 per weight.

The trouble is that Δ is set by the largest magnitude in the group. A handful of weights that sit on important channels get the same coarse step as everything else, and round-to-nearest smears them. Because those few channels contribute a large share of the layer’s output, their error propagates and perplexity jumps. The naive fix — keep the important weights in FP16 and quantize the rest — works numerically but produces mixed-precision matrices that are slow and awkward on real hardware. AWQ’s contribution is getting the same protection with a uniform 4-bit layout.

Saliency is an activation property, not a weight property

The first question is which weights to protect. A tempting answer is ‘the ones with the largest magnitude,’ but that is wrong. What a layer produces is y = W x; the contribution of input channel j to the output is W[:, j] · x_j. A weight column can be small yet still dominate the output if the activation x_j flowing through it is consistently large.

AWQ therefore ranks input channels by the magnitude of the activations that pass through them, measured as the average |x_j| over a small calibration set. Empirically, a very small subset of channels — roughly 0.1% to 1% — carries outsized activation, and protecting just those recovers almost all the lost accuracy. This is the ‘activation-aware’ in the name: the metric that decides which weights are salient is read off the activations, so calibration data (a few hundred sequences) is needed to estimate the per-channel activation statistics. Weight magnitude alone tells you nothing about which channels the network actually leans on.

The equivalent transform: scale up W, scale down x

Once you know which input channels are salient, you want to reduce their quantization error without touching the layer’s output. AWQ does this with a per-input-channel scale s_j > 1. Multiply column j of the weight matrix by s_j and divide the corresponding activation by the same s_j:

y = W x
  = Σ_j  W[:, j] · x_j
  = Σ_j  (W[:, j] · s_j) · (x_j / s_j)

Let  W' = W · diag(s),   x' = diag(s)^-1 · x
Then y = W' x'   (exactly, in full precision)

In exact arithmetic the two scalings cancel and the output is unchanged. The point is that quantization happens after the transform: we store Q(W'), not Q(W). The activation scaling 1/s_j can be folded backward into the previous layer’s weights (for example the preceding LayerNorm or linear), so at inference time there is no extra runtime division — the model runs as ordinary quantized matmuls. The transform is free at inference and only changes what gets rounded.

Why scaling shrinks the error that matters

Look at one salient weight w multiplied by activation x. The output error from quantizing it is roughly the step size times a rounding fraction times the activation:

Err( Q(w) · x )  ≈  Δ · RoundErr · |x|
   where RoundErr ∈ [0, 0.5],  E[RoundErr] ≈ 0.25

After scaling channel by s:
Err( Q(w · s) · (x / s) )  ≈  Δ' · RoundErr · |x| / s

The key is what happens to Δ. Because only a few channels are scaled and the group’s maximum magnitude is set by other, larger weights, scaling one salient channel up by a modest s barely moves max(|w|), so Δ' ≈ Δ. The activation term, however, is divided by s. Net effect: the error contributed by the salient channel drops by a factor of about 1/s. You are spending a little precision on the many unimportant channels (whose relative error rises slightly) to buy a large error reduction on the few that dominate the output. That asymmetry is exactly why it is a net win.

A worked numeric example

Take a 4-bit group (N = 4, so 2^(N-1) = 8) whose largest weight magnitude is 1.0. The step is Δ = 1.0 / 8 = 0.125. Suppose a salient channel has a small weight w = 0.1 but a large average activation |x| = 10.

Before scaling:
  Err ≈ Δ · 0.25 · |x| = 0.125 · 0.25 · 10 = 0.3125

Scale this channel by s = 2:  w → 0.2,  x → 5
  group max still ≈ 1.0  ⇒  Δ' ≈ 0.125
  Err ≈ 0.125 · 0.25 · 5 = 0.15625

The salient channel’s output error is halved — matching the 1/s prediction — while the rest of the group is essentially untouched because Δ did not change. Push s too high, though, and two things go wrong: the scaled salient weight starts to raise max(|w|) (inflating Δ for the whole group), and the un-scaled channels now sit lower in the range and lose relative precision. There is an optimum s that trades these off, which is exactly what AWQ searches for.

Searching the scale — no backprop required

AWQ does not learn the scales by gradient descent. It parameterizes the per-channel scale as a simple function of the measured activation magnitude and searches a single knob. A common form is:

s_j = (s_X,j) ^ α,     α ∈ [0, 1]
  s_X,j = average |x_j| over the calibration set

choose α* = argmin_α  || W x  -  Q(W · diag(s)) · diag(s)^-1 x ||

At α = 0 every scale is 1 (plain quantization); at α = 1 scales track activation magnitude fully. AWQ sweeps a small grid of α values, and for each one it quantizes the layer, runs the calibration activations through, and measures the mean-squared error between the original FP16 output and the quantized output. The α with the lowest error wins. Everything here is a forward pass — no gradients, no optimizer, no weight updates. That makes AWQ cheap (minutes, a few hundred calibration samples), deterministic, and robust: because it fits one scalar per layer rather than tuning individual weights, it does not overfit the calibration set and it generalizes across domains.

Shapes, granularity, and complexity

For a linear layer the weight is W : [C_out, C_in] and the activation is x : [C_in] (per token). The scale vector is s : [C_in] — one entry per input channel — applied as W · diag(s) along the input dimension and diag(s)^-1 on the activation. Quantization itself is usually per-group: input channels are chopped into groups (commonly 128), each with its own Δ and zero-point, which keeps the step size tight without storing a scale per weight.

The search cost is small: for each candidate α you do one quantize-and-evaluate pass over the calibration batch, so the total is (number of grid points) × (a handful of forward passes) per layer — independent of model depth in any expensive way and trivially parallel across layers. Storage at inference is the quantized integers plus per-group scales and zero-points; the activation scales are folded into neighboring weights, so the deployed model is just a standard 4-bit checkpoint. There is no runtime overhead versus plain weight-only quantization — AWQ changes the numbers, not the compute graph.

How AWQ differs from GPTQ

GPTQ attacks the same 4-bit target from a different angle: error compensation. It quantizes weights one column at a time and, after fixing each column, adjusts the remaining, not-yet-quantized weights to cancel the error just introduced. The adjustment uses second-order information — the Hessian of the layer’s reconstruction loss, which for a linear layer is H = 2 X X^T from the calibration activations:

δ = - (w_q - w) / [H^-1]_qq  ·  H^-1[:, q]   (update remaining weights)

So GPTQ needs the inverse Hessian, a Cholesky factorization, and a careful greedy ordering; it corrects quantization error after the fact. AWQ never computes a Hessian and never updates a weight to compensate. It uses activation statistics only to identify saliency and choose a scale, then does plain round-to-nearest on the transformed weights. The contrast is clean: GPTQ = second-order error feedback; AWQ = activation-saliency plus a protective scaling transform. AWQ is simpler and faster to run, has no matrix inversion to go unstable, and tends to generalize better because it fits one scalar per group instead of nudging thousands of individual weights toward the calibration set.

CPU-SLM implications and common pitfalls

For a small language model on a CPU, AWQ is close to ideal: 4-bit weights cut the memory-bandwidth bill that dominates decode, the transform folds away so kernels stay simple integer matmuls, and there is no mixed precision to special-case. The main pitfalls are about the ingredients. Calibration data matters: the activation magnitudes that define saliency are estimated from it, so a calibration set wildly off-distribution from deployment can pick the wrong channels — though AWQ is more forgiving here than methods that fit weights directly. Over-scaling is the other trap: too large an s inflates the group’s Δ and hurts the channels you were not protecting, which is why the α search exists — do not hand-set aggressive scales. Watch group size too: larger groups save a little memory but widen Δ and can erase the benefit. Finally, AWQ is weight-only; if a model’s trouble is genuine activation outliers under activation quantization, that is a different problem needing a different tool.