Why it matters

Checkpointing prevents lost training time. Optimizer state huge; storage matters.

Advertisement

The architecture

Save model + optimizer state.

Parallel writes to storage.

Restore on restart.

Training checkpointingSave model+ optimizerParallel writessharded I/ORestoreon restartTB-scale checkpoints; async writes; parallel FS common
Training checkpointing.
Advertisement

How it works end to end

TB-scale checkpoints.

Async writes.

Sharded across GPUs.