Why it matters

DeepSpeed enables massive-scale training. Understanding shapes large-model training.

Advertisement

The architecture

ZeRO-1: optimizer state sharded.

ZeRO-2: + gradients.

ZeRO-3: + weights.

ZeRO levelsZeRO-1optimizer shardedZeRO-2+ gradientsZeRO-3+ weights (like FSDP)ZeRO-3 similar to PyTorch FSDP; each shard training memory across all GPUs
ZeRO memory savings.
Advertisement

How it works end to end

Pipeline parallel: split model layers across GPUs. Micro-batches flow through.

Sequence parallel: split within sequence dim.

CPU offloading: swap optimizer state to CPU.

Integration: works with PyTorch models via wrapper.