Mixed-precision training is the standard recipe for training transformers fast: run the heavy matmuls in 16-bit (fp16 or bf16) to double memory bandwidth and unlock tensor cores, while keeping a small set of numerically load-bearing quantities in 32-bit so the optimization stays correct. The whole design is a single idea applied everywhere — speed where precision is cheap, precision where it is load-bearing. This overview covers the pieces: the floating-point formats and what their bits buy, the fp32 master weights that keep tiny updates from vanishing, loss scaling that rescues fp16 gradients from underflow, the rule that reductions accumulate in fp32, the range-versus-precision trade between bf16 and fp16, and the short list of ops that stay fp32 no matter what. Each of these has its own deep dive in this series; here we assemble the map.
What mixed precision actually is
A pure fp32 training step stores every weight, activation, and gradient as a 4-byte float and does every multiply-add in fp32. That is safe but wasteful: at the memory wall, 2 bytes per element moves twice the tensor per second that 4 bytes does, and modern tensor cores deliver several times more FLOPs in bf16/fp16 than in fp32. For a model that costs millions of accelerator-hours, the format choice is a budget line.
Mixed precision keeps most tensors in 16-bit for the forward and backward passes — the activations, the weight copies used in matmuls, the gradients — but retains a 32-bit copy of the quantities that cannot tolerate rounding: the optimizer’s master weights, its moment estimates, and a handful of reductions. ‘Mixed’ is literal: within one step, the same weight exists as both an fp32 master and a 16-bit compute copy, and the runtime moves between them on a schedule. Getting that schedule right is the entire subject.
Three formats and what their bits buy
All three formats split a float into sign, exponent (range), and mantissa (precision). fp32 spends 8 bits on exponent and 23 on mantissa: range up to ~3.4×10ⁿ⁸ and about 7 decimal digits of precision. fp16 spends 5 exponent bits and 10 mantissa bits: a much smaller range — largest normal value 65,504, smallest normal ~6×10⁻⁵ — and about 3.3 decimal digits. bf16 keeps fp32’s 8 exponent bits but has only 7 mantissa bits: the same range as fp32, but only ~2–3 decimal digits of precision.
That contrast is the whole story of 16-bit training. fp16 gives you more mantissa (finer precision) but a dangerously narrow range; bf16 gives you fp32’s range but coarser precision. Which failure you fear — values falling off the range cliff, or values losing their low-order digits — decides which format you reach for and which safety nets you must string up around it.
The fp32 master weights
The first failure of pure 16-bit training is silent: updates round to zero. Consider a weight of magnitude 1.0 receiving an update of 1×10⁻⁷ (a small gradient times a small learning rate). In fp16 the gap between representable numbers near 1.0 is about 10⁻₃ — so 1.0 + 1e-7 rounds back to exactly 1.0. The weight is frozen while the loss curve merely looks slow.
The fix is to keep the optimizer’s copy of every parameter in fp32 — the master weights — together with Adam’s first and second moments. Each step casts the masters down to 16-bit for compute, but the update w ← w − lr·m̂/(√v̂+ε) is applied to the fp32 master, where a 10⁻⁷ increment is representable and accumulates across thousands of steps. The cost is memory: 4 bytes of master plus 8 bytes of moments per parameter dwarf the 2-byte compute copy, which is exactly why ZeRO and FSDP shard optimizer state first.
Loss scaling for fp16
fp16’s narrow range has a second edge: gradients are often too small to represent. Gradient magnitudes in deep transformers concentrate around 10⁻⁶–10⁻₃, and much of that distribution sits below fp16’s smallest normal value, so those gradients underflow to zero before the optimizer ever sees them. bf16 does not have this problem — its exponent range matches fp32 — so loss scaling is an fp16-only device.
The trick exploits linearity of the backward pass. Multiply the loss by a scale factor L (say 2^16 = 65,536) and, by the chain rule, every gradient is multiplied by L too — sliding the whole distribution up into fp16’s representable band. Gradients are then divided by L in fp32 before clipping and the optimizer step, so the math is unchanged; only the numerics are rescued. Dynamic loss scaling automates the choice: on any overflow (an inf/nan in the scaled gradients) skip the step and halve L; after a few thousand clean steps, double it. L rides just under the overflow ceiling, maximizing headroom against underflow.
The accumulate-in-fp32 rule
The single most important numerical rule of mixed precision is: do the multiply in 16-bit, but accumulate the sum in fp32. A matmul is a chain of multiply-adds, and while each 16-bit product is fine, adding thousands of them into a 16-bit running total loses the small terms — once the accumulator grows large, adding a small product rounds away entirely. Tensor cores are built for exactly this: they take 16-bit inputs and accumulate into an fp32 register by design, returning a 16-bit result only at the end.
The same rule governs every long reduction in the model, not just matmuls. A layernorm variance over thousands of elements, a softmax denominator, and above all the gradient all-reduce across data-parallel replicas — each is a long sum whose rounding error compounds linearly with the number of terms. Summing hundreds of replicas’ gradients in 16-bit produces a biased gradient mean, a silent quality tax rather than a crash, so those reductions run in fp32 (or bf16 with care).
bf16 vs fp16: range against precision
The two 16-bit formats are not interchangeable, and the difference decides how much machinery you need. fp16 has finer precision but a range so narrow that activations in attention can spike past 65,504 to inf, and gradients underflow below its floor — so fp16 training requires loss scaling and careful overflow handling. bf16 trades mantissa for range: it inherits fp32’s exponent, so gradients never underflow and activations rarely overflow, and no loss scaling is needed at all — a whole failure class and its telemetry simply deleted.
bf16’s cost is precision: ~2–3 decimal digits versus fp16’s ~3.3. In practice the gradient noise from mini-batching already exceeds bf16’s rounding noise, so the lost mantissa bits rarely change the loss curve. That is why large-model pretraining standardized on bf16: robustness beats precision when you are training for weeks. fp16 persists on hardware without bf16 support and in some fine-tuning and inference stacks where its extra mantissa helps.
Which ops stay in fp32
Not everything is safe to cast down, and the exceptions cluster around two shapes: operations that exponentiate and operations that reduce over many elements. Both amplify small numerical errors, so they stay in fp32 regardless of the compute format.
The standard fp32 ‘islands’ are: softmax (exp overflows easily, and the max-subtraction stability trick assumes precision), layernorm statistics (variance of thousands of activations), the loss computation itself, and long reductions including the gradient all-reduce and often the residual-stream adds that carry signal through dozens of layers. Matmul accumulation is already fp32 inside the tensor core. Everything else — the matmul inputs, the elementwise activations, the attention scores before softmax — runs in 16-bit. The taxonomy is not arbitrary: an op stays fp32 exactly when its output depends on the faithful sum or exponential of many small numbers.
Autocast: the recipe as a dispatch table
You almost never wire these rules by hand. In PyTorch AMP the entire policy lives behind autocast, which is precisely a per-operation dispatch table: it intercepts each op and decides whether to run it in the low-precision format or promote it to fp32. Matmuls and convolutions run in 16-bit; softmax, layernorm, and loss functions are on the fp32 list; and when an fp32 tensor and a 16-bit tensor meet, autocast inserts the cast for you.
Loss scaling is a separate object — a GradScaler in PyTorch, or the scaler baked into DeepSpeed and Megatron — that wraps the backward pass and the optimizer step. Together they compress the whole recipe into two lines of user code: enter an autocast region for the forward pass, and scale/unscale around backward(). Understanding what those two lines do is what lets you debug a run at 3 a.m. and tell a numerics problem from a learning-rate problem — two incidents with identical symptoms and opposite remedies.
One training step, end to end
Assembling the pieces, a single fp16 step runs like this. (1) Cast the fp32 masters to fp16 compute copies. (2) Forward pass on tensor cores — fp16 inputs, fp32 accumulators, fp16 outputs — with autocast routing softmax and layernorm through fp32. (3) Compute the loss in fp32 and multiply by the current scale L. (4) Backward pass: every gradient emerges scaled by L, so a raw gradient of 3×10⁻⁷ travels as ~0.02, comfortably normal.
(5) Upcast gradients to fp32, divide by L, and scan for inf/nan. (6a) Clean step: clip by global norm, let Adam update the fp32 masters, increment the good-step counter. (6b) Overflow: skip the optimizer step entirely and halve L. (7) All-reduce gradients in fp32. The bf16 variant deletes steps 3, 5’s unscale, and 6b — collapsing to masters plus fp32 islands plus fp32 reductions, which is why bf16 won the pretraining default.
Pitfalls, and what this means on CPU
A few failure modes recur. If you resume from a checkpoint you must persist and restore the loss scale: resuming at the default 65,536 after the run had settled at 8,192 buys an immediate overflow cascade. A loss scale that keeps collapsing toward 1 is a symptom — usually an unstable activation, not a learning-rate issue — and forcing it high just papers over the real spike. And a single unguarded fp16 op (a home-grown softmax, a custom kernel) can NaN a whole run in one backward pass; keep exponentials and long sums in fp32.
For CPU and small-language-model work the calculus shifts. CPUs lack tensor cores, so 16-bit rarely speeds up compute; the win is memory footprint, which matters most at inference, where bf16 or int8 weights halve or quarter the model’s RAM. Training a small model on CPU is usually simplest in plain fp32; when 16-bit helps at all, bf16 is the safer choice because it needs no loss scaling and tolerates the coarse, un-accelerated arithmetic of a general-purpose core.