Core concept
Equivalent to DeepSpeed ZeRO-3. PyTorch-native.
Advertisement
How it works
Communication overhead. Careful config for overlap.
Advertisement
Trade-offs + gotchas
Very large models with data parallelism. Enables 70B+ training on smaller setups.