Mixed Precision Training — FP16/BF16 with FP32 Master

Why it matters

Mixed precision is training standard. Understanding shapes efficient training.

The architecture

FP16/BF16 for forward + backward.

FP32 for optimizer state + master weights.

How it works end to end

FP16: loss scaling to prevent underflow.

BF16: no loss scaling needed; wider range.

FP8: newer, aggressive; requires transformer engine.

Frameworks: torch.cuda.amp, DeepSpeed handle automatically.