FP8 Training on H100+ Hardware

Why eight bits is a different problem than sixteen

Mixed-precision BF16 works because BF16 is 1-8-7: one sign bit, eight exponent bits, seven mantissa bits. Those eight exponent bits are the same as FP32’s, so BF16 spans roughly 10^-38 to 10^38. You almost never overflow or underflow, which is why BF16 needs little more than a master weight copy to just work. It trades precision (7 mantissa bits) for range, and for training that trade is nearly free.

FP8 has no such luxury. With only eight bits total, splitting them between exponent and mantissa is a genuine dilemma: more exponent bits buy range but cost precision, and vice-versa. No single 8-bit float can simultaneously cover gradients that span many orders of magnitude and resolve weights finely. The resolution is to define two formats and assign each to the tensors whose statistics it suits, then rescale every tensor into the sliver of range the chosen format actually represents. FP8 is therefore not ‘BF16 but smaller’ — it is a scaling problem you opt into on purpose to win back another 2x in memory and throughput.

The two formats: E4M3 and E5M2 bit layouts

Both formats spend one bit on sign and split the remaining seven between exponent (E) and mantissa (M). The names say the split:

E4M3:  S | EEEE | MMM     sign(1) exp(4) mantissa(3)   bias 7
E5M2:  S | EEEEE | MM    sign(1) exp(5) mantissa(2)   bias 15

value = (-1)^S × 2^(exp - bias) × (1.MMM)     [normal]
value = (-1)^S × 2^(1 - bias)   × (0.MMM)     [subnormal]

E4M3 gives up one exponent bit to gain a mantissa bit, so it resolves values more finely but reaches a smaller maximum. To claw back range it bends the IEEE convention: it has no infinities, and only the single pattern S.1111.111 is reserved for NaN, freeing the rest of the top exponent for real numbers. E5M2 keeps the IEEE-style layout — a full infinity/NaN exponent (11111) — and its extra exponent bit roughly squares the dynamic range at the cost of coarser steps. Three mantissa bits versus two sounds tiny, but it is the difference between ~12.5% and ~25% relative spacing between representable magnitudes.

Representable ranges, worked out from the bits

Plug the extremes into the formulas. For E4M3 (bias 7): the largest finite value uses exponent field 1111 (= 15, so 2^8) with the largest non-NaN mantissa 1.110 (= 1.75), giving 2^8 × 1.75 = 448. The smallest normal is 2^(1-7) = 2^-6 ≈ 0.0156, and subnormals reach down to 2^-9 ≈ 0.00195.

For E5M2 (bias 15): the largest finite value uses exponent 11110 (= 30, so 2^15) with mantissa 1.11 (= 1.75), giving 2^15 × 1.75 = 57344. The smallest normal is 2^-14 and subnormals reach 2^-16.

format   max finite   min normal   min subnormal   mantissa steps
E4M3     448          2^-6         2^-9            3 bits  (~12.5% rel.)
E5M2     57344        2^-14        2^-16           2 bits  (~25%   rel.)

The punchline: E4M3 tops out at 448 — a value a BF16 tensor shrugs off — so anything above it saturates unless you scale first. E5M2 reaches ~57k and dips far lower, so it tolerates the wild spread of gradient magnitudes but resolves each one crudely.

Format by role: E4M3 for weights and activations, E5M2 for gradients

The assignment falls straight out of those ranges. Weights and activations in a trained network are relatively well-behaved: after normalization they cluster within a bounded band, so they do not need enormous range, and precision matters because small relative errors here feed directly into the forward pass. That is exactly E4M3’s profile — more mantissa, less range.

Gradients are the opposite. During backprop they span many orders of magnitude within a single tensor and shift as training proceeds; a few large components sit far above a long tail of tiny ones. Clipping that tail to zero or saturating the peaks is more damaging than representing each value a little coarsely, so gradients want range over precision — E5M2, with its five exponent bits reaching to 2^-16 and 57344. This is why FP8 training is described as E4M3 forward and E5M2 backward: the forward GEMM consumes E4M3 weights and activations, while the backward GEMMs that produce input- and weight-gradients cast their gradient operands to E5M2. Matching format to a tensor’s statistics is the whole game.

Per-tensor scaling: moving values into the window

Because each format’s representable window is narrow, you cannot just truncate a BF16 tensor to FP8 — its amax (the maximum absolute value) may sit far outside the window. The fix is a per-tensor scale factor. Before casting, multiply the whole tensor by a scalar s chosen so its amax lands near the format’s maximum, using the full range without saturating:

s = FP8_max / amax(X)      (e.g. FP8_max = 448 for E4M3)
X_fp8 = cast_fp8( X × s )
# ...matmul consumes X_fp8, accumulating in FP32...
Y = Y_fp8_accum / s        (descale to recover true magnitude)

Each tensor carries its own scale, tracked in higher precision, and the scale is divided back out after the matmul (or folded into the next op). This is a per-tensor operation, not per-element: one shared scalar keeps the cast cheap and the tensor-core inputs a clean 8 bits. Getting s right is the entire stability question — too small wastes range and buries small values below the subnormal floor; too large saturates the peaks to the max value.

Delayed scaling and the amax history buffer

There is a chicken-and-egg wrinkle. To pick s from the current tensor’s amax you must first scan the whole tensor — but you want to produce the scaled FP8 output in the same kernel that generates the tensor, before you have seen all of it. Computing amax in a separate pass costs an extra read of the data, which erodes the bandwidth win FP8 is supposed to deliver.

Delayed scaling sidesteps this. Instead of the current amax, you keep a rolling amax history — a small buffer of the amax observed on this tensor over the last N iterations (commonly 16 to 1024). The scale for the current step is computed from the max (or a percentile) of that history, so it is available before the tensor is produced and no extra pass is needed. The current step’s amax is recorded into the buffer for future steps. Because training statistics drift slowly, a stale-by-one-step scale is almost always close enough. The trade is a rare lag: if a tensor suddenly spikes, the history-based scale can under-shoot for a step and saturate — which is why the buffer favors recent maxima and why frameworks expose the window length as a knob.

A worked scaling example

Take a forward activation tensor destined for E4M3, with observed amax(X) = 1024. E4M3 caps at 448, so a direct cast saturates every value above 448 to the max — a catastrophic loss of information across the tensor’s top end. Instead compute the scale:

amax(X)   = 1024
FP8_max   = 448          (E4M3)
s         = 448 / 1024   = 0.4375

X × s : amax becomes 1024 × 0.4375 = 448   → fits exactly at the top of range
X_fp8     = cast_e4m3( X × 0.4375 )

matmul(X_fp8, W_fp8) → accumulate in FP32 → Y_accum
Y         = Y_accum / (s_X × s_W)     descale by both operands’ scales

Now the whole tensor lives inside E4M3’s range with the peak using the full resolution. Under delayed scaling the 1024 would come not from this step but from the amax-history buffer, and 1024 would be pushed into the buffer for the next iteration. A gradient tensor with amax 3.1e-5 shows the mirror case: cast raw to E5M2 it sits near the subnormal floor and most of it rounds to zero; scaled up by s = 57344 / 3.1e-5 ≈ 1.85e9 it spreads across E5M2’s range before the cast.

Why FP8 halves memory and bandwidth and speeds the tensor cores

The wins are mechanical. An FP8 value is one byte; a BF16 value is two. Every tensor stored in FP8 — activations held for the backward pass, weights streamed into a GEMM — occupies half the space and moves half the bytes across the memory bus and the interconnect. Since large-model training is very often memory-bandwidth-bound, halving the bytes moved translates fairly directly into speed, and it lets larger batches or longer sequences fit in the same HBM.

On compute, H100-class tensor cores run FP8 matmuls at roughly double the throughput of BF16 — the same silicon processes twice as many 8-bit multiply-accumulates per cycle. The crucial detail is that only the GEMM inputs are 8-bit: the products are accumulated in FP32 (or FP16) inside the tensor core, so a long dot product does not compound 8-bit rounding error term by term. FP8 thus attacks both bottlenecks at once — less data to move and faster math — which is why it is the headline feature of the Transformer Engine on Hopper and later. The 2x is real but conditional: it shows up when the GEMMs dominate and the scaling overhead stays small.

Accuracy: master weights, high-precision accumulation, and what stays wide

FP8 is used surgically, not everywhere. A run keeps master weights in FP32 (or BF16) and an FP8 copy only for the matmul operands; the optimizer updates the high-precision master, and the small update steps — often far below FP8’s resolution — would vanish if applied to an 8-bit weight directly. Accumulation inside every GEMM stays in FP32. Reductions and numerically delicate ops — softmax, LayerNorm, the attention logits, the loss — run in BF16 or FP32, because summing many small 8-bit terms is exactly where 8 bits fail.

Practitioners also keep the most sensitive layers wide: the first and last layers, embeddings, and occasionally the output projection are often left in BF16. Done this way, FP8 training reaches loss curves within noise of a BF16 baseline. Done carelessly — casting a tensor without a scale, or pushing softmax into FP8 — it diverges or silently degrades. The 8-bit format is the easy part; the scaling discipline around it is what makes the run converge.