FP4 training pushes the operands of a transformer’s matrix multiplies down to four bits — a single sign bit, two exponent bits, one mantissa bit — and asks the network to still learn. Four bits give you exactly sixteen numbers to represent every weight and activation, so the whole discipline is about making those sixteen numbers land where the data actually is. This is not post-training quantization, where you shrink a finished model and accept a little accuracy loss (that is tm_fp4_quant); it is doing the forward and backward passes themselves in 4-bit float, on hardware like NVIDIA Blackwell whose tensor cores run FP4 at roughly twice FP8 throughput. It is also a full step below tm_fp8_training: FP8 has 8 bits and 256 values to play with, FP4 has 16, and that gap is why FP4 needs a stack of tricks — fine-grained block scaling, stochastic rounding, and rotations — that FP8 can mostly skip. This piece walks the format, the math, and where the 4 bits are allowed to go.
The core bet: gradients in four bits
Every low-precision training scheme makes one bet: that the signal in weights, activations, and gradients survives being squeezed into a coarse grid, as long as you keep a faithful copy of the state that must not drift. FP4 makes the most aggressive version of that bet. The expensive part of training a transformer is the general matrix multiplies (GEMMs) inside the linear layers — forward, and the two in the backward pass. Those GEMMs are what FP4 targets, because that is where the FLOPs and the memory traffic live.
The catch is that 4 bits is not ‘a bit worse than 8’ — it is a different regime. With so few representable values, naive rounding destroys small updates and clips large ones, and error that would be noise in FP8 becomes bias that steers the whole run. So FP4 training keeps a high-precision master copy of the weights, does the delicate arithmetic (accumulation, the optimizer, normalization) in BF16 or FP32, and spends its budget making the 4-bit operands themselves trustworthy.
The E2M1 format: sixteen numbers
The 4-bit float used for training is E2M1: 1 sign bit, 2 exponent bits, 1 mantissa bit. The exponent bias is 2^(2-1) - 1 = 1. Working the encoding out gives the mantissa fraction as 0 or 0.5 (one bit), and four exponent fields, one of which (E=0) is subnormal:
subnormal (E=0): 2^(1-bias) * (M/2) -> {0, 0.5}
normal (E=1): 2^0 * (1 + M/2) -> {1.0, 1.5}
normal (E=2): 2^1 * (1 + M/2) -> {2.0, 3.0}
normal (E=3): 2^2 * (1 + M/2) -> {4.0, 6.0}So the positive magnitudes are {0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0}, and with the sign bit that is 16 codes total. Note what is missing: E2M1 spends every code on a finite number — there is no inf, no NaN. The largest magnitude is 6.0, the smallest nonzero is 0.5, and the gap between adjacent values is never smaller than 0.5. That grid is the entire vocabulary FP4 has for numbers.
The precision-and-range wall of four bits
Two limits fall out of that grid immediately. Range: the ratio of largest to smallest nonzero magnitude is only 6.0 / 0.5 = 12. FP8’s E4M3 spans roughly 448 / 0.0019 ≈ 240,000; FP4 spans twelve. Anything more than ~12× larger than the smallest value you care about either clips to 6.0 or flushes to zero.
Precision: with at most one mantissa bit, the step between neighbors is huge — from 4.0 to 6.0 is a single jump of 50%, so a value of 5.0 rounds to 4.0 or 6.0, a 20% error either way. Neural network weights and activations, though, are roughly bell-shaped with heavy tails: most values small and clustered, a few large outliers. A single scale for a whole tensor cannot serve both — set it for the outliers and the bulk collapses toward zero; set it for the bulk and the outliers clip. Everything that follows is a way around this one wall.
Micro-scaling: MXFP4 and NVFP4 block scales
The escape is to stop using one scale per tensor and use one scale per small block of contiguous values — micro-scaling. Each block gets a multiplier chosen for its own local magnitude, so an outlier in one block cannot wreck the resolution of a quiet block next door.
Two formats dominate. MXFP4 (the Open Compute standard) uses blocks of 32 values sharing an 8-bit E8M0 scale — a pure power of two, 2^(s-127), no mantissa. Because the scale is a power of two, applying it is an exact exponent shift with no rounding. NVFP4 (NVIDIA’s Blackwell format) goes finer: blocks of 16 with an E4M3 (8-bit float) per-block scale, plus one FP32 per-tensor scale on top. The E4M3 scale can land between powers of two, fitting each block tighter, at the cost of a second, non-exact multiply. Smaller blocks and richer scales cost more metadata but track the data more faithfully — the central FP4 trade-off.
The scaling math, worked
Quantizing a block x is: pick a scale so the block’s largest magnitude maps near FP4’s ceiling of 6.0, divide, round each element onto the E2M1 grid, and remember the scale for dequantization.
amax = max_i |x_i| # block max magnitude
scale = 2^ceil(log2(amax / 6.0)) # E8M0 power-of-two
q_i = round_to_E2M1( x_i / scale ) # onto the FP4 grid
x_hat = q_i * scale # dequantizedWorked: a block with amax = 22.0. Then 22/6 ≈ 3.67, ceil(log2 3.67) = 2, so scale = 2^2 = 4. An element x = 22 becomes 22/4 = 5.5 → rounds to 6.0, dequantized 6.0 × 4 = 24. An element x = 3.0 becomes 0.75 → 0.5, back to 2.0. The block max is preserved well; mid-range values carry visible error — which is exactly why a smaller block or a float scale, covering a narrower spread, quantizes more faithfully.
Stochastic rounding: keeping updates unbiased
Round-to-nearest has a fatal flaw for training: a weight update smaller than half the local step always rounds back to where it started, so the weight never moves and learning stalls — the coarser the grid, the worse it bites, and FP4 is very coarse. Stochastic rounding fixes it by making the rounding direction random, weighted by distance. For x between grid neighbors a < x < b:
round(x) = b with probability p = (x - a)/(b - a)
= a with probability 1 - p
E[round(x)] = a*(1-p) + b*p = a + (b-a)*p = x # unbiasedThe expected rounded value equals x exactly, so error has zero mean — it becomes noise the optimizer averages out rather than a systematic drift. A tiny update now nudges the weight to the next grid point with a small probability, and across thousands of steps those probabilistic nudges accumulate into real motion. Stochastic rounding is what lets small gradients survive a grid of sixteen points.
Hadamard rotations: spreading the outliers
Block scaling helps, but a single giant outlier still forces its whole block to a coarse scale. The trick is to rotate the data so no single coordinate is an outlier. Multiply by an orthogonal Hadamard matrix H (entries ±1/√n, with H H^T = I). A Hadamard transform mixes every coordinate into every output, so a spike concentrated in one channel gets smeared across all of them — the rotated distribution is closer to Gaussian, with a much smaller max-to-median ratio inside each block.
The reason it is free is matmul invariance. Because H H^T = I, insert H on the activations and H^T on the weights and the product is unchanged:
(X H)(H^T W) = X (H H^T) W = X WYou quantize XH and H^TW — the tamed, outlier-free versions — instead of the raw operands, and the exact GEMM result is recovered. The fast Walsh–Hadamard transform costs only O(n log n), cheap beside the matmul it protects.
Which matmuls go FP4, which stay high
FP4 is applied surgically. A linear layer during training has three GEMMs: the forward Y = X W, the input gradient dX = dY W^T, and the weight gradient dW = X^T dY. These dominate the compute, so their operands are the candidates for 4-bit — typically the forward and input-gradient GEMMs first, with the gradient tensors (which have the widest, spikiest range) often kept in FP8 or BF16 because FP4’s range of 12 cannot hold them.
Crucially, the accumulation of each dot product happens in FP32 inside the tensor core, even when the inputs are FP4 — you multiply 4-bit values but sum them in full precision. And several things never go to FP4 at all: the master weights and optimizer state (BF16/FP32), layernorm and softmax, residual adds, the embedding and final projection, and gradient reductions across devices. The rule of thumb: quantize the big matmul operands; keep anything that accumulates, normalizes, or holds long-term state in high precision.
FP4 training vs FP8 training vs FP4 post-training quant
These three sound alike and are not. FP8 training (tm_fp8_training) uses 8-bit floats — E4M3 for forward, E5M2 for gradients — giving 256 values and a dynamic range in the hundred-thousands. That headroom means FP8 often needs only per-tensor scaling and no rotations at all. FP4 has 16 values and a range of 12, so everything gets harder: block scaling becomes mandatory, stochastic rounding becomes necessary, and Hadamard rotations move from optional to routine.
FP4 post-training quantization (tm_fp4_quant) also uses E2M1, but it compresses an already-trained model for inference — one pass, no gradients, no backward, no stochastic rounding, and the goal is to minimize accuracy loss on a fixed network. FP4 training, this article’s subject, does the learning itself in 4 bits: it must keep updates unbiased over millions of steps, which is why it needs the whole probabilistic-rounding, master-weight, mixed-precision machinery that a one-shot quantizer never touches.
Practical realities and pitfalls
FP4 training earns its complexity on the right hardware: Blackwell’s FP4 tensor cores roughly double FP8 throughput and halve operand memory, so a run that is matmul-bound gets faster and a model that was memory-bound fits in less. But the gains are conditional. The most common failure is treating FP4 like FP8 — reusing a per-tensor scale, skipping rotations — and watching the loss diverge or plateau early because outliers clipped and small gradients vanished.
The recurring pitfalls: quantizing the master weights in place instead of updating a high-precision copy; accumulating in FP4 instead of FP32; putting gradients in FP4 when their range needs FP8; and choosing a block size too large for the outlier structure. On the CPU-SLM side FP4 is mostly an inference story — consumer chips lack FP4 matmul units — but the same E2M1 grid and block-scaling ideas underpin the 4-bit weight formats that let a small language model run in a couple of gigabytes.