FP4 Quantization

Post-training weight quant, not training

First, scope. There are two very different things called ‘FP4.’ One is FP4 training — running the forward and backward passes in 4-bit arithmetic, where you fight exploding gradients, stochastic rounding, and dynamic scaling. That is a different article. This one is post-training weight quantization (PTQ): the model is already trained in FP16/BF16, its weights are frozen, and you simply want to store and load them in 4 bits so the model occupies a quarter of the memory.

Two things follow. First, there is no training loop to absorb the error — whatever precision you lose in rounding, the model just tolerates at inference, so the entire game is minimizing quantization error for a fixed, known set of numbers. Second, a trained weight matrix has a predictable shape: values cluster tightly around zero and thin out in the tails, close to a normal distribution. That single statistical fact is the lever behind everything that follows. Uniform int4 ignores it; NF4 and FP4 exploit it.

The 4-bit budget: sixteen numbers

Four bits gives 2^4 = 16 distinct codes — that is the whole budget. Quantization is a two-part scheme: a codebook of 16 target values, and a per-block scale that stretches that codebook to cover the real weights. Each weight is stored as a 4-bit index (0–15); to reconstruct it you look up the codebook value and multiply by the block scale.

The naive choice is int4: sixteen evenly spaced levels, e.g. {-8, -7, …, 0, …, 7}, scaled to the weight range. A uniform grid is right when the data is uniformly distributed — every region equally likely, every level equally useful. But weights are the opposite. Picture 1000 weights drawn from N(0, σ): the vast majority land within ±σ of zero, and only a handful reach the tails. A uniform grid spends its sixteen levels evenly across [-max, +max] — lavishing resolution on the sparse tails while the dense center, where almost all the mass lives, gets only a few levels. The grid is in the wrong place.

Why non-uniform beats uniform int4

The fix is to spend levels where the weights are: pack them densely near zero and space them out toward the tails. That is what makes a non-uniform grid win. The formal way to say it: quantization error is the expected squared distance from a weight to its nearest codebook value, E[(w - Q(w))^2], weighted by the probability of each weight. Minimizing that expectation means putting more codebook points where p(w) is high.

Concretely, if 60% of your weights fall in [-0.1, 0.1], a grid with several levels inside that band represents most of the model finely, while one or two coarse levels out at ±1.0 handle the rare outliers — who cares if a one-in-a-thousand weight is rounded crudely? A uniform int4 grid does the reverse: it might place only one level in that dense central band and waste half its levels on tail regions almost no weight ever visits. For Gaussian-shaped weights, matching the grid to the distribution is worth roughly a full bit of effective precision — 4-bit NF4 can rival 5-bit uniform quantization on quality.

NF4: the quantile construction

NF4 (4-bit NormalFloat), from the QLoRA paper, makes ‘match the grid to the distribution’ precise via quantile quantization. The idea: choose the 16 codebook values so each bin holds an equal share of a standard normal’s probability mass. If every bin captures 1/16 of the weights, no bin is wasted and none overloaded — the information-theoretically optimal placement for normally-distributed data.

Construction: take the standard normal’s quantile function (inverse CDF) Q(p), split [0,1] into 16 equal-probability slices, and place a codebook value at the midpoint of each: roughly c_i = Q((i + 0.5) / 16), then rescale so the outermost values sit at ±1. NF4 does this asymmetrically so an exact 0.0 is representable (important for zero-padding). The resulting 16 levels, normalized to [-1, 1]:

-1.000  -0.696  -0.525  -0.395  -0.284  -0.185  -0.091  0.000
 0.080   0.161   0.246   0.338   0.441   0.563   0.723  1.000

Notice the spacing: seven levels crammed into [-0.7, 0] where density is highest, then big jumps out to ±1. The grid traces the bell curve.

FP4 (E2M1): the floating-point grid

FP4 reaches a non-uniform grid a different way — it is a tiny floating-point format, E2M1: 1 sign bit, 2 exponent bits, 1 mantissa bit. Like all floats, its representable values are denser near zero and get exponentially sparser as magnitude grows, because each exponent step doubles the scale while the single mantissa bit gives two steps within each octave. The positive E2M1 grid is:

0,  0.5,  1.0,  1.5,  2.0,  3.0,  4.0,  6.0   (and negatives)

So FP4 also concentrates resolution near zero — better than uniform int4 for Gaussian weights — but its spacing comes from the exponent’s powers of two, not the normal distribution’s actual quantiles. That makes it a slightly worse fit than NF4 for weights, but FP4 has a decisive practical edge: it is a hardware number format. Newer tensor cores multiply FP4 values natively, so FP4 buys the memory saving and faster matmuls, whereas NF4 is a storage/codebook trick typically dequantized back to a compute type before the matmul. NF4 optimizes accuracy per bit; FP4 optimizes for silicon.

Block-wise scaling

Both codebooks live on a normalized [-1, 1] range, so to use them you map real weights into that range with a scale. The standard choice is absmax: s = max(|w|) over a block, store each weight as Q(w / s), and reconstruct as s · codebook[index].

The catch is that one scale for an entire tensor is fragile: a single large outlier weight inflates max(|w|), shrinking every other weight toward zero after normalization and collapsing them onto a few codebook levels. The fix is block-wise scaling — chop the tensor into small blocks (NF4 uses 64 weights per block) and give each its own absmax scale. Now an outlier only pollutes its own block of 64, and the rest quantize cleanly. Smaller blocks track the local range more faithfully and reduce error, but each costs one stored scale, so block size trades accuracy against overhead — because those scales are not free.

Double quantization: quantizing the scales

Count the overhead. With a block size of 64 and one 32-bit float scale per block, the scales alone add 32 / 64 = 0.5 bits per weight. On top of a 4-bit weight that is a 12.5% tax — a real chunk of the memory budget spent on bookkeeping.

Double quantization (also from QLoRA) attacks it by quantizing the scales too. The block scales are themselves just numbers with a distribution, so group them (256 per block) and quantize each 32-bit scale down to 8 bits, keeping one 32-bit scale-of-scales per group. The new overhead per weight:

before:  32 / 64                 = 0.500 bits/weight
after:    8 / 64  +  32 / (64*256) = 0.125 + 0.002
                                   = 0.127 bits/weight

That is a ~0.37 bit-per-weight saving for essentially no accuracy loss — on a 65B model, about 3 GB reclaimed. NF4’s effective footprint drops from 4.5 to roughly 4.13 bits per weight. A neat recursion: the trick that compresses the weights is applied once more to the metadata.

A worked memory example

Put numbers on it for a 7-billion-parameter model. The dominant cost is the weights, so we tally bytes per parameter and multiply.

FP16            7e9 * 16 bits / 8   = 14.00 GB   (baseline)
int8            7e9 *  8 bits / 8   =  7.00 GB
NF4 weights     7e9 *  4 bits / 8   =  3.50 GB
  + block scales 7e9 * 0.5 bits / 8 = +0.44 GB   -> 3.94 GB
  + double quant 7e9 * 0.127 / 8    = +0.11 GB   -> 3.61 GB

So the honest number for NF4 with double quantization is about 3.6 GB — a 14 / 3.6 ≈ 3.9× reduction over FP16, comfortably inside an 8 GB laptop or small GPU. The lesson: don’t quote the bare 4-bit figure (3.5 GB) and forget the scales — the metadata is real, which is why double quantization exists. And activations, the KV cache, and a few sensitive layers stay in higher precision, so live inference memory sits above the static weight figure.

The 4-bit accuracy / memory trade

Where does 4-bit land on the quality curve? The drop from 16-bit to 8-bit weights is nearly free — perplexity barely moves. From 8-bit to 4-bit, with a distribution-aware grid like NF4 plus block scaling, the loss is small but measurable: a point or two of relative accuracy on many tasks, more on the hardest. Below 4 bits quality falls off a cliff without fancier methods, which is why 4 bits is the current sweet spot for weight-only PTQ.

4-bit NF4 holds up because everything above works together: the grid matches the weights (quantile placement), the scale is local (block-wise), and the metadata is cheap (double quantization). Take any one away and quality sags. The trade is usually worth it: a sliver of accuracy for a 4× memory cut, which for a memory-bandwidth-bound decoder often means faster inference too, since loading a quarter of the bytes per token is a quarter of the bottleneck.

Practical notes and pitfalls

A few things bite in practice. Not all layers are equal: attention and MLP weights quantize well, but embeddings, the LM head, and layernorm parameters are sensitive — many recipes keep them in higher precision. Outliers dominate: a small number of large-magnitude weights cause most of the error, which is why block-wise scaling and outlier-aware methods matter.

Weight-only vs full: NF4 and this whole discussion quantize weights; activations stay in FP16/BF16 and the matmul dequantizes weights on the fly. That is why it helps memory and bandwidth more than raw compute, and why the 4× figure is weights only — activations and the KV cache are separate budgets. And measure, don’t assume: the quality hit is model- and task-dependent, so evaluate on your own workload rather than trusting a headline perplexity number. Used with these cautions, 4-bit NF4 is the default way to fit a capable model into commodity memory.