GPTQ quantizes a trained transformer’s weights to 3 or 4 bits without any retraining, and it does so with a surprisingly principled idea: treat each linear layer as a little least-squares problem and compensate for every rounding error you make. When you snap one weight to the nearest grid point, you nudge the weights you have not touched yet so the layer’s output barely moves. That single move — error compensation driven by a second-order (Hessian) model of the layer — is what lets GPTQ reach 4-bit, and often 3-bit, with only a small accuracy loss where naive round-to-nearest falls apart. This piece builds the objective from first principles, derives the compensating update, works a tiny numeric example by hand, explains the Cholesky trick that makes it fast and stable, and draws the line between GPTQ and its cousins AWQ and SmoothQuant.
Post-training quantization and the per-layer objective
Post-training quantization (PTQ) takes an already-trained model and lowers the precision of its weights — from 16-bit floats to 4-bit integers, say — using only a small calibration set of a few hundred forward passes, and no gradient descent on the task. The danger is obvious: rounding millions of weights independently to the nearest grid point injects error into every layer, and those errors compound through the network.
GPTQ makes the problem tractable by attacking it one layer at a time. For a single linear layer with weight matrix W and calibration inputs X: [d_in, N] (activations collected from real data), it asks for the quantized matrix Ŵ that keeps the layer’s output as close as possible to the original. It does not try to keep the weights themselves close — it keeps the thing that actually matters, WX, close. That reframing, from ‘round the weights’ to ‘preserve the outputs on real inputs,’ is the whole game.
The reconstruction objective, written out
Concretely, GPTQ minimizes the squared output error of each layer over the calibration data:
argmin_Ŵ || W·X − Ŵ·X ||_2^2
W, Ŵ : [d_row, d_col] X : [d_col, N]Two facts make this manageable. First, the rows of W do not interact in the output — output feature i depends only on row i of W — so the whole matrix problem decouples into independent per-row problems, each of the form min || w·X − ŵ·X ||^2 for a single row vector w. Second, that per-row objective is a plain quadratic in w. Expanding it, the term that depends on the weight perturbation δ = ŵ − w is δ (X X^T) δ^T. The matrix X X^T is a Gram matrix of the inputs — and a quadratic’s curvature matrix is its Hessian. That is the door into the second-order machinery.
Why it is second-order: the Hessian
Because the per-row error is quadratic in the weights, its Hessian is constant — it does not depend on w at all:
H = ∂^2 E / ∂w^2 = 2 · X X^T (H : [d_col, d_col])The constant 2 rides along harmlessly and cancels in every ratio below, so most treatments just write H = X X^T. What matters is the off-diagonal structure: H_ij measures how correlated input channels i and j are across the calibration set. Correlation is exactly what makes compensation possible — if channel i and channel j tend to fire together, an error you introduce by quantizing w_i can be partly cancelled by adjusting w_j.
This is the lineage of Optimal Brain Surgeon / Optimal Brain Damage and its quantization descendant OBQ (Optimal Brain Quantization): use the second-order term to decide which weight to touch next and how to repair the damage. GPTQ is a fast, scalable reworking of OBQ’s update.
Greedy quantization with error compensation
OBQ quantizes the weights of a row one at a time, and after each one it updates all the remaining (not-yet-quantized) weights to absorb the error just introduced. When you quantize weight w_q, the rounding error is e_q = w_q − quant(w_q), and the optimal correction to the surviving weights is:
δ = − ( e_q / [H^{-1}]_qq ) · H^{-1}_{:,q}Read that as: spread the error out along column q of the inverse Hessian, scaled so it exactly offsets the perturbation at q. The same [H^{-1}]_qq term also gives OBQ its greedy pick rule — quantize next whichever weight has the smallest e_q^2 / [H^{-1}]_qq, i.e. the one whose rounding does least damage. Crucially, the weights already frozen are left alone; only the free ones move. Each step trades a little freedom for a lot of error reduction, and the running H^{-1} tells you exactly how.
A worked two-weight example
Take a row with two weights, w = [0.62, 0.30], and a Hessian from correlated inputs, H = [[4, 2], [2, 3]]. Quantize w_1 to a grid of step 0.5: the nearest point is 0.5, so q_1 = 0.5 and the perturbation is δ_1 = q_1 − w_1 = −0.12. With only two variables, the general rule collapses to a clean special case — fix δ_1 and minimize the quadratic over δ_2:
δ_2 = −(H_12 / H_22) · δ_1 = −(2/3)(−0.12) = 0.08So w_2 moves from 0.30 to 0.38 to soak up the damage. The payoff: the output error δ H δ^T falls from H_11·δ_1^2 = 0.0576 (no compensation) to (H_11 − H_12^2/H_22)·δ_1^2 = 0.0384 — a 33% reduction from a single free weight. Scale that to thousands of input channels and you see why error compensation, not smarter rounding, is what buys the extra bits.
GPTQ’s speedups: fixed order and lazy batches
OBQ’s greedy selection is accurate but expensive: every row picks its own order, so nothing can be shared, and the cost scales badly to matrices with thousands of columns. GPTQ’s first insight is that on large layers the order barely matters — quantizing columns in a fixed left-to-right sweep loses almost nothing versus the greedy pick. Dropping the per-row order means every row shares one column order, so the whole matrix can be quantized in lockstep and the inverse-Hessian information reused across all rows at once.
The second insight is lazy batch updates. Applying the compensating update to every remaining column after each single column is memory-bandwidth-bound — lots of data moved for little arithmetic. Instead GPTQ processes columns in blocks (commonly 128), fully updating weights inside the block immediately and deferring the update to columns outside the block until the block finishes. Same math, far better arithmetic intensity, and the layer quantizes in minutes rather than hours.
The Cholesky trick for the inverse Hessian
The sequential updates need particular rows of H^{-1}, and OBQ recomputes them by repeatedly shrinking the inverse as columns are removed. Done naively over thousands of steps that accumulates floating-point error, and the Hessian can even drift non-positive-definite. GPTQ’s fix is elegant: the exact sequence of rows the fixed-order sweep needs from H^{-1} is precisely what a Cholesky factorization of H^{-1} hands you, computed once up front with a numerically stable library routine.
Two guards keep it well-posed. A small dampening term is added to the diagonal — H ← H + λI, with λ around 1% of the mean diagonal — before inverting, which regularizes near-singular Hessians from dead or redundant channels. And columns with a zero diagonal (channels the calibration data never excited) are simply skipped. The result is one stable factorization driving the entire layer, replacing a long chain of error-prone rank-one updates.
Group-size and act-order
Two knobs shape the accuracy/size trade. Group-size controls how many input channels share one quantization scale and zero-point. A single scale for a whole row (per-channel) is cheapest to store but coarse; a scale per group of, say, 128 columns (g=128) tracks local weight statistics far better, at the cost of a little metadata. Smaller groups mean lower error and slightly larger files — 128 is the common sweet spot, and it is often the difference between a usable and a broken 3-bit model.
Act-order (a.k.a. desc_act) changes the sweep order: instead of plain left-to-right, quantize columns in order of decreasing activation magnitude — the diagonal of H. The intuition is that the most influential channels should be quantized first, while the largest pool of still-free weights remains to compensate for them. Combined with grouping it noticeably improves accuracy, especially at 3-bit; the cost is a channel permutation that inference kernels must account for.
Not AWQ, not SmoothQuant: what GPTQ actually does
All three fight quantization error, but through different mechanisms, and the contrast sharpens what GPTQ is. GPTQ is weight-only and its lever is error compensation: it accepts the rounding, then uses the inverse Hessian to move other weights so the output error is minimized. Nothing is rescaled — the damage is actively repaired.
AWQ (Activation-aware Weight Quantization) instead protects the small fraction of weight channels that act on high-magnitude activations. It searches for per-channel scales that enlarge those salient channels before quantizing (and shrink them after), so their relative precision survives — but it does no cross-weight compensation and computes no Hessian. SmoothQuant targets a different problem entirely: quantizing activations as well as weights (W8A8). Activation outliers are hard to quantize, so it migrates that difficulty into the weights with a per-channel smoothing scale, making both sides quantization-friendly. In short: GPTQ compensates, AWQ protects salient channels by scaling, SmoothQuant rebalances weight-vs-activation difficulty.
Practical notes and pitfalls for small CPU models
For small language models on CPU, GPTQ is a strong default because it is weight-only and the payoff is direct: 4-bit weights cut the memory footprint roughly 4× versus fp16, and since CPU decode is memory-bandwidth-bound, moving fewer bytes per token often means faster generation, not just a smaller file. 4-bit with g=128 and act-order typically lands within a hair of the full-precision perplexity; 3-bit is reachable but far more sensitive and usually wants grouping and act-order to hold up.
The pitfalls are mostly about calibration and honesty. Use calibration data that resembles your real inputs — the Hessian is only as representative as X, and an off-distribution calibration set quietly degrades everything. Too few samples give a noisy, near-singular Hessian (lean on the dampening term). And remember the error is minimized per layer, on the calibration set — it is a local proxy, not an end-to-end guarantee, so always confirm the quantized model on a real downstream task rather than trusting the reconstruction loss alone.