You quantized a model from FP16 to 4-bit, it still loads, it still talks, and the perplexity barely moved. Ship it? Not yet. Perplexity is the cheapest, most seductive, and most misleading number in the quantization toolbox: it averages away exactly the rare, high-stakes behaviors — a multi-step proof, a needle buried at token 30,000, a strict JSON schema — that decide whether a compressed model is usable. Evaluating quantization properly means triangulating from several angles at once: a language-modeling metric (perplexity), task metrics (downstream accuracy), signal-fidelity metrics measured inside the network (per-layer MSE and SQNR), and a distributional metric on the outputs (KL divergence). This piece defines each, gives the SQNR and MSE formulas with a worked example, shows how per-layer sensitivity finds the fragile weights, and ends with a checklist you can actually run.
Perplexity: the default metric and what it measures
Perplexity is the exponentiated average negative log-likelihood the model assigns to a held-out corpus. For a token sequence of length N, with the model predicting each token from its predecessors:
PPL = exp( -(1/N) Σ_i log p(x_i | x_<i) )
lower PPL → the model was, on average, less ‘surprised’It is popular for good reasons: it needs no labels, runs on any text, is a single scalar, and correlates broadly with model quality across scale. For quantization it is the natural first probe — run the FP16 model and the quantized model over the same corpus (WikiText, C4, a slice of your own domain data) and compare. But notice what the formula does: it takes a mean of a log-likelihood over every token. The overwhelming majority of tokens in natural text are easy and highly predictable (‘the’, closing brackets, common continuations). Those easy tokens dominate the average, so perplexity is a measure of bulk fluency, not of the model’s behavior on the handful of pivotal tokens where a decision is actually made.
Perplexity delta — and why it hides behavior changes
The quantization figure of merit is usually the perplexity delta, reported as an absolute gap or a relative one:
ΔPPL = PPL_quant − PPL_fp
ΔPPL_rel = (PPL_quant − PPL_fp) / PPL_fp (often < 1%)A common result is ‘4-bit costs +0.1 perplexity’ and everyone relaxes. The problem is averaging and cancellation. Quantization makes the model slightly better on some tokens and slightly worse on others; the mean can stay almost flat while the distribution of per-token errors widens dramatically. Perplexity also has no notion of task success: getting the final answer digit of an arithmetic problem wrong changes one token’s log-probability by a rounding error, yet flips the answer from correct to wrong. And it is measured under teacher forcing — every prefix is the ground-truth text, never the model’s own output — so it never sees the compounding drift of autoregressive generation, where one degraded token conditions the next. A tiny ΔPPL is necessary but nowhere near sufficient.
Downstream accuracy: the tasks that actually matter
Because perplexity is a proxy, you have to measure the thing you care about directly: downstream task accuracy. Run the FP16 and quantized models through the same task suite and compare scores — multiple-choice knowledge (MMLU-style), grade-school and competition math (GSM8K), code generation (HumanEval pass@1), instruction-following, and, crucially, your own production tasks with your own rubric.
Two disciplines make this trustworthy. First, use generative, not just likelihood-ranked, evaluation where the real workload is generative: a multiple-choice metric that scores the log-probability of ‘A’ vs ‘B’ can look fine while free-form generation degrades, because ranking four options never exercises long autoregressive rollouts. Second, watch the variance: a 1-point MMLU drop may be noise, but a 6-point GSM8K drop with unchanged perplexity is the signature of a reasoning regression that perplexity structurally cannot see. Task accuracy is the ground truth quantization must answer to; everything else is a cheaper early warning.
Error norms: per-layer mean squared error
The output metrics tell you whether quality dropped; to learn where, measure the numerical error inside the network. The simplest is mean squared error between a tensor and its quantized-then-dequantized reconstruction. For a weight (or activation) tensor x with n elements and quantizer Q(·):
e = x − Q(x) (the quantization residual)
MSE = (1/n) Σ_i (x_i − Q(x_i))^2 = (1/n) ||e||^2You can compute MSE on the raw weights, but it is far more informative on the layer outputs under real activations: feed calibration data through the FP16 network, capture each layer’s output, do the same for the quantized network, and take the per-layer MSE. This is exactly the objective that modern post-training methods (GPTQ, AWQ and kin) minimize — they choose scales and rounding to reduce reconstruction error propagated through the layer, not just on the isolated weights. Raw MSE has one weakness: its scale depends on how large the tensor’s values are, so a big MSE on a high-magnitude layer is not directly comparable to a small MSE on a low-magnitude one. That is what SQNR fixes.
SQNR: signal-to-quantization-noise ratio (with a worked example)
SQNR normalizes the error by the signal it corrupts, giving a scale-free fidelity number in decibels — higher is better:
P_signal = Σ_i x_i^2 P_noise = Σ_i (x_i − Q(x_i))^2
SQNR(dB) = 10 · log10( P_signal / P_noise )
= 20 · log10( ||x|| / ||x − Q(x)|| )
rule of thumb for B-bit uniform quantization:
SQNR ≈ 6.02 · B + 1.76 dB (~6 dB per added bit)Worked example. Take one linear layer’s weight tensor. In FP16 its signal power is P_signal = Σ w^2 = 128.0. Quantize to INT4 and measure the residual power P_noise = Σ e^2 = 0.32. Then SQNR = 10 · log10(128 / 0.32) = 10 · log10(400) ≈ 26.0 dB. The 4-bit rule of thumb predicts 6.02×4 + 1.76 ≈ 25.8 dB — the tensor is quantizing essentially ideally. Now suppose a neighboring layer measures only 14 dB at the same bit width. That layer is far worse than uniform theory allows, a fingerprint of outliers stretching the scale so the many normal-sized weights land in coarse bins. SQNR turns ‘this layer is fragile’ into a comparable number across layers, tensors, and bit widths.
KL divergence of the output distributions
MSE and SQNR live inside the network; the last angle looks at the output distribution itself. At each position the model emits a probability distribution over the vocabulary. Let P be the FP16 model’s softmax and Q the quantized model’s, over the same context. The Kullback–Leibler divergence measures how much they disagree:
KL(P || Q) = Σ_v P(v) · log( P(v) / Q(v) ) ≥ 0
average over positions → a single per-token divergenceKL is strictly sharper than perplexity or top-1 agreement because it compares the whole distribution, not just the argmax. Two models can pick the same next token while assigning very different mass to the alternatives — and that runner-up mass is precisely what governs sampling diversity, calibration, and how the model behaves at higher temperature. A near-zero average KL is strong evidence the quantized model is behaviorally faithful; a KL that spikes on specific token types (numbers, code punctuation, rare vocabulary) localizes the damage that a flat perplexity delta smeared into invisibility. It is also the natural objective if you later distill the quantized model back toward its FP16 teacher.
Per-layer sensitivity: finding the fragile weights
Not all layers deserve the same bit width. Sensitivity analysis ranks how much each layer’s quantization hurts the whole model, so you can spend bits where they matter. The cheap version is the per-layer SQNR or output MSE from above — low-SQNR layers are suspects. The rigorous version is a leave-one-out ablation: quantize exactly one layer, keep the rest in FP16, and measure the end-to-end change in KL, perplexity, or task score. Repeat per layer to get a sensitivity ranking.
The findings are remarkably consistent across models. A small number of layers — often the first and last blocks, and the layers carrying outlier activation channels — dominate the damage, while most of the network tolerates aggressive compression. That structure is exactly what mixed-precision exploits: keep the sensitive few at 8-bit (or FP16), push the tolerant majority to 4-bit or lower, and recover most of the quality at most of the compression. Sensitivity analysis is what turns quantization from a single global knob into a targeted, per-layer budget.
Why low perplexity loss doesn’t preserve reasoning or long context
Here is the crux the metrics conspire to hide. Perplexity is a per-token, teacher-forced average over mostly-easy text. Reasoning and long-context retrieval are the opposite: sequential and brittle. A chain-of-thought answer is correct only if every step is correct; a single quantization-induced slip in an intermediate token derails the rest, and because generation is autoregressive the error compounds rather than averaging out. Perplexity, scored on the ground-truth chain, never witnesses that cascade.
Long context is hit through a second door. Quantizing the KV cache injects noise into every stored key and value, and a correct retrieval at position 30,000 depends on one attention score standing clearly above thousands of competitors. A little noise on many keys is enough to let the wrong token win — yet on short calibration text, where nothing important sits far back, perplexity looks pristine. This is why a model with +0.1 perplexity can quietly lose 10 points of GSM8K or fail needle-in-a-haystack recall. If reasoning or long context is your workload, you must measure them directly; no language-modeling average will surface the regression.
A practical evaluation checklist
Put the angles together into a routine you run on every quantized build, cheapest first so failures short-circuit early:
| Check | Metric | Red flag |
|---|---|---|
| Bulk fluency | ΔPPL on in-domain text | relative ΔPPL > ~1–2% |
| Per-layer fidelity | SQNR / output MSE per layer | any layer far below 6·B+1.76 dB |
| Distributional faithfulness | avg KL(P || Q) over positions | KL spikes on numbers / code / rare tokens |
| Task accuracy | MMLU / GSM8K / HumanEval, generative | drop beyond run-to-run noise |
| Reasoning | multi-step math & chain-of-thought | correct-answer rate falls, PPL flat |
| Long context | needle-in-haystack at full length | recall collapses past mid-context |
Three rules keep it honest. Evaluate generatively, the way you deploy, not only by likelihood ranking. Test at the target sequence length, including a quantized KV cache if you use one, since short calibration text hides long-context failure. And use sensitivity analysis to spend bits: when a check fails, the per-layer SQNR ranking usually points straight at the one or two layers that need a higher precision. For a CPU-served small model, where every bit of compression buys real throughput, this discipline is what separates a genuine win from a silent regression.