Core concept
DeepSpeed ZeRO-Offload + ZeRO-Infinity. Bandwidth to CPU/NVMe becomes bottleneck.
Advertisement
How it works
Very memory-limited setups. Extreme scale training on fewer GPUs.
Advertisement
Trade-offs + gotchas
Training 100B+ params on limited GPUs. Consumer-hardware fine-tuning.