Perplexity: the default metric and what it measures

Perplexity is the exponentiated average negative log-likelihood the model assigns to a held-out corpus. For a token sequence of length N, with the model predicting each token from its predecessors:

PPL = exp( -(1/N) Σ_i log p(x_i | x_<i) )

lower PPL  →  the model was, on average, less ‘surprised’

It is popular for good reasons: it needs no labels, runs on any text, is a single scalar, and correlates broadly with model quality across scale. For quantization it is the natural first probe — run the FP16 model and the quantized model over the same corpus (WikiText, C4, a slice of your own domain data) and compare. But notice what the formula does: it takes a mean of a log-likelihood over every token. The overwhelming majority of tokens in natural text are easy and highly predictable (‘the’, closing brackets, common continuations). Those easy tokens dominate the average, so perplexity is a measure of bulk fluency, not of the model’s behavior on the handful of pivotal tokens where a decision is actually made.

Advertisement

Perplexity delta — and why it hides behavior changes

The quantization figure of merit is usually the perplexity delta, reported as an absolute gap or a relative one:

ΔPPL      = PPL_quant − PPL_fp
ΔPPL_rel  = (PPL_quant − PPL_fp) / PPL_fp    (often < 1%)

A common result is ‘4-bit costs +0.1 perplexity’ and everyone relaxes. The problem is averaging and cancellation. Quantization makes the model slightly better on some tokens and slightly worse on others; the mean can stay almost flat while the distribution of per-token errors widens dramatically. Perplexity also has no notion of task success: getting the final answer digit of an arithmetic problem wrong changes one token’s log-probability by a rounding error, yet flips the answer from correct to wrong. And it is measured under teacher forcing — every prefix is the ground-truth text, never the model’s own output — so it never sees the compounding drift of autoregressive generation, where one degraded token conditions the next. A tiny ΔPPL is necessary but nowhere near sufficient.

Advertisement

Downstream accuracy: the tasks that actually matter

Because perplexity is a proxy, you have to measure the thing you care about directly: downstream task accuracy. Run the FP16 and quantized models through the same task suite and compare scores — multiple-choice knowledge (MMLU-style), grade-school and competition math (GSM8K), code generation (HumanEval pass@1), instruction-following, and, crucially, your own production tasks with your own rubric.

Two disciplines make this trustworthy. First, use generative, not just likelihood-ranked, evaluation where the real workload is generative: a multiple-choice metric that scores the log-probability of ‘A’ vs ‘B’ can look fine while free-form generation degrades, because ranking four options never exercises long autoregressive rollouts. Second, watch the variance: a 1-point MMLU drop may be noise, but a 6-point GSM8K drop with unchanged perplexity is the signature of a reasoning regression that perplexity structurally cannot see. Task accuracy is the ground truth quantization must answer to; everything else is a cheaper early warning.

Error norms: per-layer mean squared error

The output metrics tell you whether quality dropped; to learn where, measure the numerical error inside the network. The simplest is mean squared error between a tensor and its quantized-then-dequantized reconstruction. For a weight (or activation) tensor x with n elements and quantizer Q(·):

e     = x − Q(x)              (the quantization residual)
MSE   = (1/n) Σ_i (x_i − Q(x_i))^2  = (1/n) ||e||^2

You can compute MSE on the raw weights, but it is far more informative on the layer outputs under real activations: feed calibration data through the FP16 network, capture each layer’s output, do the same for the quantized network, and take the per-layer MSE. This is exactly the objective that modern post-training methods (GPTQ, AWQ and kin) minimize — they choose scales and rounding to reduce reconstruction error propagated through the layer, not just on the isolated weights. Raw MSE has one weakness: its scale depends on how large the tensor’s values are, so a big MSE on a high-magnitude layer is not directly comparable to a small MSE on a low-magnitude one. That is what SQNR fixes.

SQNR: signal-to-quantization-noise ratio (with a worked example)

SQNR normalizes the error by the signal it corrupts, giving a scale-free fidelity number in decibels — higher is better:

P_signal = Σ_i x_i^2          P_noise = Σ_i (x_i − Q(x_i))^2

SQNR(dB) = 10 · log10( P_signal / P_noise )
         = 20 · log10( ||x|| / ||x − Q(x)|| )

rule of thumb for B-bit uniform quantization:
SQNR ≈ 6.02 · B + 1.76  dB      (~6 dB per added bit)

Worked example. Take one linear layer’s weight tensor. In FP16 its signal power is P_signal = Σ w^2 = 128.0. Quantize to INT4 and measure the residual power P_noise = Σ e^2 = 0.32. Then SQNR = 10 · log10(128 / 0.32) = 10 · log10(400) ≈ 26.0 dB. The 4-bit rule of thumb predicts 6.02×4 + 1.76 ≈ 25.8 dB — the tensor is quantizing essentially ideally. Now suppose a neighboring layer measures only 14 dB at the same bit width. That layer is far worse than uniform theory allows, a fingerprint of outliers stretching the scale so the many normal-sized weights land in coarse bins. SQNR turns ‘this layer is fragile’ into a comparable number across layers, tensors, and bit widths.

KL divergence of the output distributions

MSE and SQNR live inside the network; the last angle looks at the output distribution itself. At each position the model emits a probability distribution over the vocabulary. Let P be the FP16 model’s softmax and Q the quantized model’s, over the same context. The Kullback–Leibler divergence measures how much they disagree:

KL(P || Q) = Σ_v P(v) · log( P(v) / Q(v) )   ≥ 0

average over positions → a single per-token divergence

KL is strictly sharper than perplexity or top-1 agreement because it compares the whole distribution, not just the argmax. Two models can pick the same next token while assigning very different mass to the alternatives — and that runner-up mass is precisely what governs sampling diversity, calibration, and how the model behaves at higher temperature. A near-zero average KL is strong evidence the quantized model is behaviorally faithful; a KL that spikes on specific token types (numbers, code punctuation, rare vocabulary) localizes the damage that a flat perplexity delta smeared into invisibility. It is also the natural objective if you later distill the quantized model back toward its FP16 teacher.