bfloat16 — ‘brain float’ — is the format that quietly made large-model training routine. Its trick is almost embarrassingly simple: take a 32-bit float and throw away the bottom 16 bits of the fraction. What survives is a 16-bit number that keeps FP32’s entire dynamic range while carrying a quarter of its precision. That single choice — spend the saved bits on range, not on mantissa — is why BF16 slots into a training loop with no loss scaling, no overflow drama, and almost none of the babysitting FP16 demands. But the precision you gave up does not vanish; it resurfaces in large sums and tiny weight updates, and the standard recipe — FP32 master weights, FP32 matmul accumulation, and sometimes stochastic rounding — exists to pay that bill. This piece works through the bit layout, the range arithmetic, a rounding example you can check by hand, and where those 7 mantissa bits help and where they hurt.
The one-line idea: FP32 with the tail chopped off
A float32 lays out its 32 bits as 1 sign, 8 exponent, 23 mantissa. bfloat16 keeps the sign and exponent exactly and truncates the mantissa from 23 bits to 7, giving 1 + 8 + 7 = 16 bits. Nothing about the exponent field changes — same width, same bias of 127, same encoding of infinities and NaNs.
Converting FP32 → BF16 is therefore close to a no-op: take the top 16 bits of the 32-bit word (rounding on the boundary) and you are done. No re-scaling of the exponent, and no risk that an FP32-representable value overflows or underflows in BF16, because both formats reach the same magnitudes. Contrast float16 (FP16), which spends its bits as 1 + 5 + 10 — fewer exponent bits, a much narrower range. BF16 made the opposite bet, and for training it is the right one.
The bit layout, and a worked encoding
Here are the three formats side by side, and one number pushed through the BF16 encoder by hand.
format sign exponent mantissa total
FP32 1 8 23 32 bits
BF16 1 8 7 16 bits
FP16 1 5 10 16 bits
encode π = 3.14159 into BF16
3.14159 = 1.570796 × 2^1 (normalize)
sign = 0
exponent = 1 + 127 = 128 = 1000 0000
mantissa = .570796 → 7 bits = 100 1001
value = 1.5703125 × 2^1 = 3.140625
error = 3.14159 - 3.140625 ≈ 0.001 (≈ 0.03%)The stored value is 0 10000000 1001001. The significand is always 1.fffffff in binary — the leading 1 is implicit — so 7 stored bits buy 8 bits of significand. Because they are the top 7 fraction bits of the value FP32 would store, BF16 rounding is just ‘keep the high bits, round on bit 8’ — why hardware casts between the formats almost for free.
Range math: why 8 exponent bits reach FP32’s ceiling
Dynamic range is set entirely by the exponent field, and BF16 shares FP32’s. With an 8-bit exponent and bias 127, the stored exponent runs 0–255, with 0 and 255 reserved (subnormals/zero and infinity/NaN). So the usable normalized exponents are 1–254, i.e. actual exponents -126 to +127.
max normal = (2 - 2^-7) × 2^127 ≈ 3.39 × 10^38
min normal = 1.0 × 2^-126 ≈ 1.18 × 10^-38
min subnormal = 2^-126 × 2^-7 = 2^-133 ≈ 9.2 × 10^-40Those are, to the digit, FP32’s limits — it differs only in the (2 - 2^-23) factor at the top, a rounding hair. FP16, with 5 exponent bits, tops out at just 65504 and bottoms out near 6 × 10^-5. That gap — 10^38 versus 10^4 at the top, 10^-38 versus 10^-5 at the bottom — is why BF16 is comfortable where FP16 is fragile.
Why matching FP32’s range makes BF16 drop-in
Training a transformer produces values across an enormous span: activations of order 1 and — the dangerous end — gradients of 10^-6 or smaller for deep layers early on. In FP16, anything below ≈ 6 × 10^-5 flushes to zero, and a gradient that underflows to zero is a weight that never learns. FP16 survives this only with loss scaling: multiply the loss by a large factor (say 2^15) before the backward pass to lift small gradients into the representable band, then divide it back out before the optimizer step — usually with dynamic tuning to dodge the overflow the trick invites.
BF16 deletes this entire mechanism. Because its smallest normal is ~10^-38, gradients that would vanish in FP16 are represented directly: no scale to pick, no overflow to guard. You cast to BF16 and train. That operational simplicity, more than raw speed, is why BF16 became the default.
The precision cost: 7 mantissa bits
Range is free; precision is what you paid. With 8 significand bits, BF16’s machine epsilon — the gap between 1.0 and the next representable value — is 2^-7 = 0.0078125, roughly 2–3 significant decimal digits. FP16’s 11 significand bits give epsilon 2^-10 ≈ 0.00098 (3–4 digits), and FP32’s 2^-23 ≈ 1.2 × 10^-7 gives 7.
Crucially, floating-point precision is relative: the spacing between BF16 values scales with magnitude. In [1, 2) the step is 2^-7; in [1024, 2048) it is 2^3 = 8. So the fractional error of any single rounded value stays near 2^-8 ≈ 0.4% everywhere — harmless for one number. The trouble comes when you add millions of them, or add a number far smaller than the one you add it to — where the missing 16 mantissa bits send the bill.
Where it bites: large accumulations
Consider a dot product or a reduction over thousands of terms — the heart of every matmul and gradient sum. As a running total grows, its BF16 spacing grows with it. Once the accumulator reaches 1024, the representable gap is 8, so any term smaller than 4 rounds away to nothing. This is swamping: small contributions are silently dropped because they land below the accumulator’s current resolution.
Sum a million values of ~1.0 in pure BF16 and the total stalls far short of 1,000,000, because past a point each new +1 falls below the running sum’s rounding step and vanishes. The fix is not more mantissa bits but a wider accumulator: keep the operands in BF16, let the running sum live in FP32 — what tensor-core hardware does, and the key to accurate low-precision matmul.
Where it bites: small weight updates
The second failure mode is the optimizer step. A weight update is w ← w - η · g, and late in training η · g is routinely 10^3–10^6 times smaller than w. Watch what BF16 does to it:
w = 1.0
η · g = 0.002 (the update)
BF16 step near 1.0 = 2^-7 = 0.0078125
half-step = 0.00390625
0.002 < 0.00390625 → round-to-nearest gives 1.0
w (after) = 1.0 (the update DISAPPEARED)Because 0.002 is less than half of BF16’s step at 1.0, 1.0 + 0.002 rounds straight back to 1.0. Do this every step and the weight freezes — not because the gradient is zero, but because the update is forever smaller than the storage resolution. Thousands of real nudges accumulate to nothing, and a model that should keep improving flatlines.
The fixes: FP32 master weights and stochastic rounding
The standard cure is FP32 master weights. The optimizer keeps a full-precision copy of every parameter; the update w - η · g is applied to that FP32 copy, where 0.002 is comfortably representable and accumulates faithfully over many steps. A BF16 copy is cast off the master only for the forward and backward matmuls. Small updates land in the master and are never lost; the BF16 weights are a fast, disposable shadow.
The lighter-weight alternative is stochastic rounding: round probabilistically in proportion to distance instead of always to nearest. Here 1.0 + 0.002 rounds up to 1.0078125 with probability 0.002 / 0.0078125 ≈ 0.26 and stays at 1.0 otherwise. Any single step is wrong, but the expected value is exact, so the update survives in aggregate — letting you keep weights in BF16 without an FP32 master, at the cost of some added variance.
FP32 accumulation inside the matmul
The accumulation fix is baked into the hardware, and it is why BF16 matmul does not visibly degrade quality. On NVIDIA tensor cores (and TPU MXUs), a BF16 matmul reads BF16 inputs but runs the multiply-add chain in an FP32 accumulator: each product of two BF16 numbers is widened to FP32 and summed there, and only the final result is cast back down.
This is the best of both worlds: the operands are cheap — half the memory bandwidth and double the throughput of FP32 — while the long reduction that would otherwise swamp small terms runs at full FP32 resolution. The 8-bit significand only rounds the inputs to each product, a bounded ~0.4% per operand; it never compounds across the thousands of adds. When people say ‘BF16 just works,’ FP32 accumulation is half the reason.
BF16 vs FP16, and what to reach for
Both are 16 bits; they differ only in how the bits are split, and that split decides their character.
| BF16 | FP16 | |
|---|---|---|
| Layout (s/e/m) | 1 / 8 / 7 | 1 / 5 / 10 |
| Max normal | ≈ 3.4 × 10^38 | 65504 |
| Min normal | ≈ 1.2 × 10^-38 | ≈ 6.1 × 10^-5 |
| Epsilon (2^-m) | 2^-7 ≈ 0.0078 | 2^-10 ≈ 0.00098 |
| Sig. digits | ~2–3 | ~3–4 |
| Loss scaling | not needed | required |
FP16 has the finer mantissa, so inside its narrow range it is the more precise of the two — which is why it remains common for inference, where activations are known-scale. For training, range beats precision: the danger is vanishing gradients, not a coarse mantissa, and BF16 handles range for free while master weights and FP32 accumulation recover the precision that matters — the trade the ecosystem settled on.
Practical notes for small models and CPUs
The same numerics scale down to a laptop-sized SLM. BF16’s memory win is real — parameters, gradients, and activations at 2 bytes instead of 4 — and it matters most when you are RAM-bound rather than FLOP-bound, the usual story on a CPU. But watch the hardware: many CPUs lack native BF16 arithmetic and emulate it, so a BF16 tensor may be up-converted to FP32 for the math — the memory saving without the speed saving.
Two rules carry over. First, never keep optimizer state in BF16 without stochastic rounding — the small-update trap does not care how big your model is. Second, if a reduction (a layer-norm statistic, a loss sum, a softmax denominator) looks off, check the accumulator width first; the fix is almost always ‘accumulate in FP32,’ not ‘abandon BF16.’ Range for storage, FP32 for the sums — that recipe holds from a cluster down to a model on your desk.