Why eight bits is a different problem than sixteen
Mixed-precision BF16 works because BF16 is 1-8-7: one sign bit, eight exponent bits, seven mantissa bits. Those eight exponent bits are the same as FP32’s, so BF16 spans roughly 10^-38 to 10^38. You almost never overflow or underflow, which is why BF16 needs little more than a master weight copy to just work. It trades precision (7 mantissa bits) for range, and for training that trade is nearly free.
FP8 has no such luxury. With only eight bits total, splitting them between exponent and mantissa is a genuine dilemma: more exponent bits buy range but cost precision, and vice-versa. No single 8-bit float can simultaneously cover gradients that span many orders of magnitude and resolve weights finely. The resolution is to define two formats and assign each to the tensors whose statistics it suits, then rescale every tensor into the sliver of range the chosen format actually represents. FP8 is therefore not ‘BF16 but smaller’ — it is a scaling problem you opt into on purpose to win back another 2x in memory and throughput.
The two formats: E4M3 and E5M2 bit layouts
Both formats spend one bit on sign and split the remaining seven between exponent (E) and mantissa (M). The names say the split:
E4M3: S | EEEE | MMM sign(1) exp(4) mantissa(3) bias 7
E5M2: S | EEEEE | MM sign(1) exp(5) mantissa(2) bias 15
value = (-1)^S × 2^(exp - bias) × (1.MMM) [normal]
value = (-1)^S × 2^(1 - bias) × (0.MMM) [subnormal]E4M3 gives up one exponent bit to gain a mantissa bit, so it resolves values more finely but reaches a smaller maximum. To claw back range it bends the IEEE convention: it has no infinities, and only the single pattern S.1111.111 is reserved for NaN, freeing the rest of the top exponent for real numbers. E5M2 keeps the IEEE-style layout — a full infinity/NaN exponent (11111) — and its extra exponent bit roughly squares the dynamic range at the cost of coarser steps. Three mantissa bits versus two sounds tiny, but it is the difference between ~12.5% and ~25% relative spacing between representable magnitudes.
Representable ranges, worked out from the bits
Plug the extremes into the formulas. For E4M3 (bias 7): the largest finite value uses exponent field 1111 (= 15, so 2^8) with the largest non-NaN mantissa 1.110 (= 1.75), giving 2^8 × 1.75 = 448. The smallest normal is 2^(1-7) = 2^-6 ≈ 0.0156, and subnormals reach down to 2^-9 ≈ 0.00195.
For E5M2 (bias 15): the largest finite value uses exponent 11110 (= 30, so 2^15) with mantissa 1.11 (= 1.75), giving 2^15 × 1.75 = 57344. The smallest normal is 2^-14 and subnormals reach 2^-16.
format max finite min normal min subnormal mantissa steps
E4M3 448 2^-6 2^-9 3 bits (~12.5% rel.)
E5M2 57344 2^-14 2^-16 2 bits (~25% rel.)The punchline: E4M3 tops out at 448 — a value a BF16 tensor shrugs off — so anything above it saturates unless you scale first. E5M2 reaches ~57k and dips far lower, so it tolerates the wild spread of gradient magnitudes but resolves each one crudely.
Format by role: E4M3 for weights and activations, E5M2 for gradients
The assignment falls straight out of those ranges. Weights and activations in a trained network are relatively well-behaved: after normalization they cluster within a bounded band, so they do not need enormous range, and precision matters because small relative errors here feed directly into the forward pass. That is exactly E4M3’s profile — more mantissa, less range.
Gradients are the opposite. During backprop they span many orders of magnitude within a single tensor and shift as training proceeds; a few large components sit far above a long tail of tiny ones. Clipping that tail to zero or saturating the peaks is more damaging than representing each value a little coarsely, so gradients want range over precision — E5M2, with its five exponent bits reaching to 2^-16 and 57344. This is why FP8 training is described as E4M3 forward and E5M2 backward: the forward GEMM consumes E4M3 weights and activations, while the backward GEMMs that produce input- and weight-gradients cast their gradient operands to E5M2. Matching format to a tensor’s statistics is the whole game.
Per-tensor scaling: moving values into the window
Because each format’s representable window is narrow, you cannot just truncate a BF16 tensor to FP8 — its amax (the maximum absolute value) may sit far outside the window. The fix is a per-tensor scale factor. Before casting, multiply the whole tensor by a scalar s chosen so its amax lands near the format’s maximum, using the full range without saturating:
s = FP8_max / amax(X) (e.g. FP8_max = 448 for E4M3)
X_fp8 = cast_fp8( X × s )
# ...matmul consumes X_fp8, accumulating in FP32...
Y = Y_fp8_accum / s (descale to recover true magnitude)Each tensor carries its own scale, tracked in higher precision, and the scale is divided back out after the matmul (or folded into the next op). This is a per-tensor operation, not per-element: one shared scalar keeps the cast cheap and the tensor-core inputs a clean 8 bits. Getting s right is the entire stability question — too small wastes range and buries small values below the subnormal floor; too large saturates the peaks to the max value.
Delayed scaling and the amax history buffer
There is a chicken-and-egg wrinkle. To pick s from the current tensor’s amax you must first scan the whole tensor — but you want to produce the scaled FP8 output in the same kernel that generates the tensor, before you have seen all of it. Computing amax in a separate pass costs an extra read of the data, which erodes the bandwidth win FP8 is supposed to deliver.
Delayed scaling sidesteps this. Instead of the current amax, you keep a rolling amax history — a small buffer of the amax observed on this tensor over the last N iterations (commonly 16 to 1024). The scale for the current step is computed from the max (or a percentile) of that history, so it is available before the tensor is produced and no extra pass is needed. The current step’s amax is recorded into the buffer for future steps. Because training statistics drift slowly, a stale-by-one-step scale is almost always close enough. The trade is a rare lag: if a tensor suddenly spikes, the history-based scale can under-shoot for a step and saturate — which is why the buffer favors recent maxima and why frameworks expose the window length as a knob.