Post-training quantization and the per-layer objective
Post-training quantization (PTQ) takes an already-trained model and lowers the precision of its weights — from 16-bit floats to 4-bit integers, say — using only a small calibration set of a few hundred forward passes, and no gradient descent on the task. The danger is obvious: rounding millions of weights independently to the nearest grid point injects error into every layer, and those errors compound through the network.
GPTQ makes the problem tractable by attacking it one layer at a time. For a single linear layer with weight matrix W and calibration inputs X: [d_in, N] (activations collected from real data), it asks for the quantized matrix Ŵ that keeps the layer’s output as close as possible to the original. It does not try to keep the weights themselves close — it keeps the thing that actually matters, WX, close. That reframing, from ‘round the weights’ to ‘preserve the outputs on real inputs,’ is the whole game.
The reconstruction objective, written out
Concretely, GPTQ minimizes the squared output error of each layer over the calibration data:
argmin_Ŵ || W·X − Ŵ·X ||_2^2
W, Ŵ : [d_row, d_col] X : [d_col, N]Two facts make this manageable. First, the rows of W do not interact in the output — output feature i depends only on row i of W — so the whole matrix problem decouples into independent per-row problems, each of the form min || w·X − ŵ·X ||^2 for a single row vector w. Second, that per-row objective is a plain quadratic in w. Expanding it, the term that depends on the weight perturbation δ = ŵ − w is δ (X X^T) δ^T. The matrix X X^T is a Gram matrix of the inputs — and a quadratic’s curvature matrix is its Hessian. That is the door into the second-order machinery.
Why it is second-order: the Hessian
Because the per-row error is quadratic in the weights, its Hessian is constant — it does not depend on w at all:
H = ∂^2 E / ∂w^2 = 2 · X X^T (H : [d_col, d_col])The constant 2 rides along harmlessly and cancels in every ratio below, so most treatments just write H = X X^T. What matters is the off-diagonal structure: H_ij measures how correlated input channels i and j are across the calibration set. Correlation is exactly what makes compensation possible — if channel i and channel j tend to fire together, an error you introduce by quantizing w_i can be partly cancelled by adjusting w_j.
This is the lineage of Optimal Brain Surgeon / Optimal Brain Damage and its quantization descendant OBQ (Optimal Brain Quantization): use the second-order term to decide which weight to touch next and how to repair the damage. GPTQ is a fast, scalable reworking of OBQ’s update.
Greedy quantization with error compensation
OBQ quantizes the weights of a row one at a time, and after each one it updates all the remaining (not-yet-quantized) weights to absorb the error just introduced. When you quantize weight w_q, the rounding error is e_q = w_q − quant(w_q), and the optimal correction to the surviving weights is:
δ = − ( e_q / [H^{-1}]_qq ) · H^{-1}_{:,q}Read that as: spread the error out along column q of the inverse Hessian, scaled so it exactly offsets the perturbation at q. The same [H^{-1}]_qq term also gives OBQ its greedy pick rule — quantize next whichever weight has the smallest e_q^2 / [H^{-1}]_qq, i.e. the one whose rounding does least damage. Crucially, the weights already frozen are left alone; only the free ones move. Each step trades a little freedom for a lot of error reduction, and the running H^{-1} tells you exactly how.
A worked two-weight example
Take a row with two weights, w = [0.62, 0.30], and a Hessian from correlated inputs, H = [[4, 2], [2, 3]]. Quantize w_1 to a grid of step 0.5: the nearest point is 0.5, so q_1 = 0.5 and the perturbation is δ_1 = q_1 − w_1 = −0.12. With only two variables, the general rule collapses to a clean special case — fix δ_1 and minimize the quadratic over δ_2:
δ_2 = −(H_12 / H_22) · δ_1 = −(2/3)(−0.12) = 0.08So w_2 moves from 0.30 to 0.38 to soak up the damage. The payoff: the output error δ H δ^T falls from H_11·δ_1^2 = 0.0576 (no compensation) to (H_11 − H_12^2/H_22)·δ_1^2 = 0.0384 — a 33% reduction from a single free weight. Scale that to thousands of input channels and you see why error compensation, not smarter rounding, is what buys the extra bits.
GPTQ’s speedups: fixed order and lazy batches
OBQ’s greedy selection is accurate but expensive: every row picks its own order, so nothing can be shared, and the cost scales badly to matrices with thousands of columns. GPTQ’s first insight is that on large layers the order barely matters — quantizing columns in a fixed left-to-right sweep loses almost nothing versus the greedy pick. Dropping the per-row order means every row shares one column order, so the whole matrix can be quantized in lockstep and the inverse-Hessian information reused across all rows at once.
The second insight is lazy batch updates. Applying the compensating update to every remaining column after each single column is memory-bandwidth-bound — lots of data moved for little arithmetic. Instead GPTQ processes columns in blocks (commonly 128), fully updating weights inside the block immediately and deferring the update to columns outside the block until the block finishes. Same math, far better arithmetic intensity, and the layer quantizes in minutes rather than hours.
The Cholesky trick for the inverse Hessian
The sequential updates need particular rows of H^{-1}, and OBQ recomputes them by repeatedly shrinking the inverse as columns are removed. Done naively over thousands of steps that accumulates floating-point error, and the Hessian can even drift non-positive-definite. GPTQ’s fix is elegant: the exact sequence of rows the fixed-order sweep needs from H^{-1} is precisely what a Cholesky factorization of H^{-1} hands you, computed once up front with a numerically stable library routine.
Two guards keep it well-posed. A small dampening term is added to the diagonal — H ← H + λI, with λ around 1% of the mean diagonal — before inverting, which regularizes near-singular Hessians from dead or redundant channels. And columns with a zero diagonal (channels the calibration data never excited) are simply skipped. The result is one stable factorization driving the entire layer, replacing a long chain of error-prone rank-one updates.