Core concept
Simple. Doesn't help with memory (model must fit on one GPU).
Advertisement
How it works
Not enough for large models. Combine with other parallelism.
Advertisement
Trade-offs + gotchas
Small-medium models. Baseline distributed training.