Train a network in pure FP16 and it often diverges — not because the math is wrong, but because half of the gradients quietly become zero. FP16 simply cannot represent the tiny magnitudes that small gradients naturally take, so they underflow, and a parameter that receives a zero gradient never learns. Loss scaling is the one-line fix that makes FP16 training work: multiply the loss by a constant S before backpropagation, let the chain rule carry that factor into every gradient so they all shift up into representable range, then divide the gradients back down by S before the optimizer touches them. This piece works through the FP16 range math that creates the problem, the scale-then-unscale algorithm that solves it, the difference between a fixed scale and a dynamic one that adapts itself, a worked numeric trace, and why BF16 sidesteps the entire issue — which is exactly why it now dominates.
The underflow problem in one sentence
FP16 (half precision) is attractive for training: it halves memory and, on tensor-core hardware, roughly doubles throughput versus FP32. But it buys that with a narrow dynamic range. The catch is not the loss or the activations — those sit comfortably in FP16’s range. It is the gradients. During backpropagation, gradients of the loss with respect to early-layer weights are products of many small numbers, and they routinely land in the range 1e-8 to 1e-10.
FP16 cannot represent numbers that small. Anything below roughly 2^-24 ≈ 6e-8 rounds to exactly zero. So a gradient that should nudge a weight by a hair instead reads as no signal at all — the weight is frozen. Multiply this across the many parameters whose gradients live in that tiny range and a large fraction of the network stops learning. The model stalls or diverges, and no amount of tuning the learning rate fixes it, because the information was destroyed the moment the gradient was rounded to zero. Loss scaling exists to rescue those numbers before rounding happens.
The FP16 number line and the underflow cliff
A FP16 value is 1 sign bit, 5 exponent bits, and 10 mantissa bits. Work the range out from the exponent field:
FP16 = [sign:1][exponent:5][mantissa:10], bias = 15
largest normal = 2^15 * (2 - 2^-10) ≈ 65504
smallest normal = 2^-14 ≈ 6.10e-5
smallest subnormal = 2^-14 * 2^-10 = 2^-24 ≈ 5.96e-8
underflow cliff = anything < ~2^-25 rounds to 0Below the smallest normal (2^-14), FP16 keeps going with subnormals — values with a leading zero that trade mantissa precision for a little extra reach, down to 2^-24. Past that is the cliff: there is no representable positive number between 2^-25-ish and zero, so everything there collapses to zero. A gradient of 3e-9 is roughly 2^-28 — well over the edge, and gone. The whole game of loss scaling is to shift the gradient distribution to the right, off the cliff, using the headroom that sits unused up near 65504.
The trick: scale the loss, and the chain rule does the rest
The elegance is that you do not touch the gradients directly. You multiply the scalar loss by a constant S before calling backward(). Because differentiation is linear, that single factor propagates uniformly to every gradient in the graph:
L’ = S · L
∂L’/∂w = ∂(S · L)/∂w = S · (∂L/∂w) for every parameter wEvery gradient is now S times larger than it would have been — the entire distribution slides up by the same factor. Choose S = 2^15 = 32768 and a gradient that was 2^-27 (underflowed to zero) becomes 2^-27 × 2^15 = 2^-12 ≈ 2.4e-4, comfortably inside FP16’s normal range and represented with full precision. Crucially, using a power of two makes the scaling exact in floating point — it only changes the exponent, never the mantissa, so no rounding error is introduced by the scaling itself. The gradients are now safe to store in FP16 through the backward pass.
Unscale before you step — and before you clip
Scaled gradients are correct for surviving the backward pass, but they are S times too big for the actual weight update. If the optimizer stepped on them directly, the effective learning rate would be S times too large and training would explode. So you unscale: divide the gradients by S before the optimizer step, typically as they are copied into the FP32 master weights the optimizer maintains.
The subtle part is gradient clipping. Clipping thresholds a gradient by its true norm (say, clip to norm 1.0). If you clip the scaled gradients, the norm is S times too large and the clip fires far too aggressively — every step gets crushed to the threshold. So the canonical order is fixed: backward(S·L) → unscale grads → clip on the unscaled grads → optimizer.step(). Anything that reads the gradient magnitude — clipping, gradient-norm logging, weight decay computed on gradients — must see the true, unscaled values. Scaling is a temporary disguise the gradients wear only for the trip through FP16 storage.
Static loss scaling: pick one number
The simplest scheme is static (fixed) loss scaling: choose a single constant S — commonly 128, 1024, or 2^15 — and use it for the whole run. It works when it works, and it is trivial to implement. The difficulty is picking the value, because it is a tightrope between two failure modes.
Set S too low and small gradients still underflow to zero — you have not lifted the distribution far enough off the cliff. Set it too high and the large gradients at the top of the distribution overflow: grad × S > 65504 becomes inf, which then poisons the whole update with inf/nan. The safe window depends on the model and even drifts during training as gradient magnitudes shrink toward convergence — a scale that was fine at step 0 may start overflowing later, or become too conservative. Static scaling therefore demands manual tuning and offers no safety net, which is exactly why the dynamic scheme was invented.
Dynamic loss scaling: let S find itself
Dynamic loss scaling removes the guesswork by treating S as a value the training loop discovers automatically. The principle: push S as high as possible to protect the smallest gradients, and back off only when the largest gradients overflow. Two rules run every step:
Backoff on overflow. After the backward pass, inspect the scaled gradients for inf or nan. If any appear, S was too high this step: skip the optimizer step entirely (the gradients are corrupt — do not update the weights) and multiply S by a backoff factor, usually 0.5. Growth on a clean streak. If some number of consecutive steps (often 2000) pass with no overflow, gently raise S by a growth factor, usually 2, and reset the counter. The result is a self-tuning value that hovers just below the overflow threshold — the largest scale that is currently safe — and re-adapts for free as gradient magnitudes change over training. This is what frameworks ship by default.
A worked dynamic-scaling trace
Follow a run that starts, as PyTorch’s GradScaler does, at S = 2^16 = 65536 with backoff 0.5, growth 2, and a growth interval of 2000 steps. The first few steps show the automatic search settling in:
step S result action
---- ------- ------------------------- ------------------------
0 65536 inf in grads (overflow) skip step, S *= 0.5
1 32768 inf in grads (overflow) skip step, S *= 0.5
2 16384 clean step, +1 clean
... 16384 clean x2000 grow: S *= 2
2002 32768 inf in grads (overflow) skip step, S *= 0.5
2003 16384 clean step, counter resetRead the shape of it: the first two steps are “wasted” probes — the initial 65536 was too hot, so S halves twice until 16384 is safe, and only skipped weight updates, never corrupted ones, result. Then it holds, and after 2000 clean steps it tries doubling — discovers 32768 overflows — and drops straight back. It has found the largest safe power of two and will keep gently re-probing forever. A handful of skipped steps early in training is a negligible price for never having to hand-tune the scale.
Why BF16 makes the whole problem disappear
The cleanest fix for gradient underflow is to stop using a format that underflows. BF16 (bfloat16) is also 16 bits, but it splits them differently: 1 sign, 8 exponent, and only 7 mantissa bits. Those 8 exponent bits are the same width as FP32’s, so BF16 has essentially the same dynamic range as FP32 — smallest normal around 1.18e-38, versus FP16’s 6e-5.
A gradient of 3e-9 that vanished in FP16 is nowhere near BF16’s underflow floor — it is represented without drama. So BF16 needs no loss scaling at all: no scale to tune, no overflow checks, no skipped steps, no dynamic bookkeeping. The cost is precision: 7 mantissa bits give roughly 2–3 decimal digits, fewer than FP16’s 10 bits. In practice that coarser precision is far more forgiving than FP16’s range cliff, because deep-learning training is robust to a little gradient noise but not to gradients being deleted. This is why modern training — anywhere the hardware supports it — overwhelmingly prefers BF16, and loss scaling has quietly become a legacy concern.
Practicalities and the CPU-SLM angle
Loss scaling is fundamentally a tensor-core / GPU concern. FP16 pays off only where the hardware has fast half-precision matrix units; a CPU has no such units, so CPU training of a small language model runs in FP32 or, on newer chips, BF16 — and in both cases loss scaling is simply not in the picture. That is the practical takeaway for this series: if you are training or fine-tuning a small model on CPU, reach for BF16 (when available) precisely because it inherits FP32’s range and skips this entire apparatus.
A few gotchas worth remembering when you do meet FP16. The overflow check must run on the scaled gradients, before unscaling, since that is where inf appears. A persistent stream of overflows usually signals a genuine instability (a bad learning rate, an exploding layer) rather than a scaling problem — dynamic scaling will keep halving S into the ground trying to compensate, which is a useful diagnostic. And loss scaling is orthogonal to the FP32 master-weight copy that mixed-precision training keeps: scaling protects gradients in transit; the master weights protect the slow accumulation of tiny updates. You need both, for different reasons.
2^-24 ≈ 6e-8) underflow to zero and their parameters stop learning. Loss scaling fixes this by multiplying the loss by a constant S before backprop; by the chain rule every gradient scales up by the same S, shifting the whole distribution off the underflow cliff into representable range. You then unscale — divide by S — before gradient clipping and before the optimizer step, so the update and the clip norm are correct. Static scaling picks one S and walks a tightrope between underflow and overflow; dynamic scaling raises S until the gradients overflow to inf, then halves it and skips that step, self-tuning to the largest safe value. BF16, with 8 exponent bits and FP32-like range, underflows so rarely that it needs no scaling at all — which, for CPU and modern training alike, is why it wins.