What mixed precision actually is
A pure fp32 training step stores every weight, activation, and gradient as a 4-byte float and does every multiply-add in fp32. That is safe but wasteful: at the memory wall, 2 bytes per element moves twice the tensor per second that 4 bytes does, and modern tensor cores deliver several times more FLOPs in bf16/fp16 than in fp32. For a model that costs millions of accelerator-hours, the format choice is a budget line.
Mixed precision keeps most tensors in 16-bit for the forward and backward passes — the activations, the weight copies used in matmuls, the gradients — but retains a 32-bit copy of the quantities that cannot tolerate rounding: the optimizer’s master weights, its moment estimates, and a handful of reductions. ‘Mixed’ is literal: within one step, the same weight exists as both an fp32 master and a 16-bit compute copy, and the runtime moves between them on a schedule. Getting that schedule right is the entire subject.
Three formats and what their bits buy
All three formats split a float into sign, exponent (range), and mantissa (precision). fp32 spends 8 bits on exponent and 23 on mantissa: range up to ~3.4×10ⁿ⁸ and about 7 decimal digits of precision. fp16 spends 5 exponent bits and 10 mantissa bits: a much smaller range — largest normal value 65,504, smallest normal ~6×10⁻⁵ — and about 3.3 decimal digits. bf16 keeps fp32’s 8 exponent bits but has only 7 mantissa bits: the same range as fp32, but only ~2–3 decimal digits of precision.
That contrast is the whole story of 16-bit training. fp16 gives you more mantissa (finer precision) but a dangerously narrow range; bf16 gives you fp32’s range but coarser precision. Which failure you fear — values falling off the range cliff, or values losing their low-order digits — decides which format you reach for and which safety nets you must string up around it.
The fp32 master weights
The first failure of pure 16-bit training is silent: updates round to zero. Consider a weight of magnitude 1.0 receiving an update of 1×10⁻⁷ (a small gradient times a small learning rate). In fp16 the gap between representable numbers near 1.0 is about 10⁻₃ — so 1.0 + 1e-7 rounds back to exactly 1.0. The weight is frozen while the loss curve merely looks slow.
The fix is to keep the optimizer’s copy of every parameter in fp32 — the master weights — together with Adam’s first and second moments. Each step casts the masters down to 16-bit for compute, but the update w ← w − lr·m̂/(√v̂+ε) is applied to the fp32 master, where a 10⁻⁷ increment is representable and accumulates across thousands of steps. The cost is memory: 4 bytes of master plus 8 bytes of moments per parameter dwarf the 2-byte compute copy, which is exactly why ZeRO and FSDP shard optimizer state first.
Loss scaling for fp16
fp16’s narrow range has a second edge: gradients are often too small to represent. Gradient magnitudes in deep transformers concentrate around 10⁻⁶–10⁻₃, and much of that distribution sits below fp16’s smallest normal value, so those gradients underflow to zero before the optimizer ever sees them. bf16 does not have this problem — its exponent range matches fp32 — so loss scaling is an fp16-only device.
The trick exploits linearity of the backward pass. Multiply the loss by a scale factor L (say 2^16 = 65,536) and, by the chain rule, every gradient is multiplied by L too — sliding the whole distribution up into fp16’s representable band. Gradients are then divided by L in fp32 before clipping and the optimizer step, so the math is unchanged; only the numerics are rescued. Dynamic loss scaling automates the choice: on any overflow (an inf/nan in the scaled gradients) skip the step and halve L; after a few thousand clean steps, double it. L rides just under the overflow ceiling, maximizing headroom against underflow.
The accumulate-in-fp32 rule
The single most important numerical rule of mixed precision is: do the multiply in 16-bit, but accumulate the sum in fp32. A matmul is a chain of multiply-adds, and while each 16-bit product is fine, adding thousands of them into a 16-bit running total loses the small terms — once the accumulator grows large, adding a small product rounds away entirely. Tensor cores are built for exactly this: they take 16-bit inputs and accumulate into an fp32 register by design, returning a 16-bit result only at the end.
The same rule governs every long reduction in the model, not just matmuls. A layernorm variance over thousands of elements, a softmax denominator, and above all the gradient all-reduce across data-parallel replicas — each is a long sum whose rounding error compounds linearly with the number of terms. Summing hundreds of replicas’ gradients in 16-bit produces a biased gradient mean, a silent quality tax rather than a crash, so those reductions run in fp32 (or bf16 with care).
bf16 vs fp16: range against precision
The two 16-bit formats are not interchangeable, and the difference decides how much machinery you need. fp16 has finer precision but a range so narrow that activations in attention can spike past 65,504 to inf, and gradients underflow below its floor — so fp16 training requires loss scaling and careful overflow handling. bf16 trades mantissa for range: it inherits fp32’s exponent, so gradients never underflow and activations rarely overflow, and no loss scaling is needed at all — a whole failure class and its telemetry simply deleted.
bf16’s cost is precision: ~2–3 decimal digits versus fp16’s ~3.3. In practice the gradient noise from mini-batching already exceeds bf16’s rounding noise, so the lost mantissa bits rarely change the loss curve. That is why large-model pretraining standardized on bf16: robustness beats precision when you are training for weeks. fp16 persists on hardware without bf16 support and in some fine-tuning and inference stacks where its extra mantissa helps.
Which ops stay in fp32
Not everything is safe to cast down, and the exceptions cluster around two shapes: operations that exponentiate and operations that reduce over many elements. Both amplify small numerical errors, so they stay in fp32 regardless of the compute format.
The standard fp32 ‘islands’ are: softmax (exp overflows easily, and the max-subtraction stability trick assumes precision), layernorm statistics (variance of thousands of activations), the loss computation itself, and long reductions including the gradient all-reduce and often the residual-stream adds that carry signal through dozens of layers. Matmul accumulation is already fp32 inside the tensor core. Everything else — the matmul inputs, the elementwise activations, the attention scores before softmax — runs in 16-bit. The taxonomy is not arbitrary: an op stays fp32 exactly when its output depends on the faithful sum or exponential of many small numbers.