Advertisement
BF16: 8 exp bits (FP32 range), 7 mantissa. FP16: 5 exp bits (narrow), 10 mantissa.
What you're seeing
FP16 range: ±65504. Easily overflows in attention. BF16 has FP32's range. Slightly less precise but training is robust to it.
★ KEY TAKEAWAY
BF16 = FP32 range + half mantissa. Faster, half memory, training-safe. The right choice for modern CPU/GPU training.
▶ WHAT TO TRY
- Compare the bit layouts visually.
- FP16 has only 5 exponent bits — overflows easily in attention. BF16 has FP32's 8 — won't.