Core concept

~2x compute overhead. But 5-10x memory saving on transformer.

Advertisement

How it works

Complexity in framework. Some ops don't fit.

Advertisement

Trade-offs + gotchas

Memory-constrained training. Common with FSDP.