ZeRO-Offload is a simple idea with a precise payoff: keep the forward and backward passes on the GPU where they belong, but push the optimizer — the fp32 master weights, the Adam momentum and variance, and the parameter-update step itself — down onto the CPU and its far larger pool of DRAM. Those optimizer states are the biggest single consumer of GPU memory in mixed-precision training, so evicting them lets a single commodity GPU train a model many times larger than its own VRAM would otherwise allow. Nothing is free: every step you now ship gradients across PCIe to the CPU and copy freshly updated weights back. This article works the memory arithmetic and the PCIe arithmetic side by side, shows why the Adam update is cheap enough to survive on a CPU while the matrix-heavy passes stay on the GPU, and where the throughput trade turns against you.
The memory the GPU cannot afford
Start with why offloading is worth the trouble. In mixed-precision Adam training, the memory that scales with the parameter count Ψ is not dominated by the model itself — it is dominated by the optimizer. For every parameter you pay for an fp16 weight, an fp16 gradient, and then a full fp32 triple: a master copy of the weight, a first-moment (momentum) estimate, and a second-moment (variance) estimate. That fp32 triple is the expensive part, and it lives on the GPU purely so the optimizer can touch it once per step.
The tragedy is that this memory is almost idle. The forward and backward passes — the compute-hungry work — never read momentum or variance. They only need the fp16 weights and produce fp16 gradients. So the largest block of GPU memory is reserved for a step that runs once per iteration and is arithmetically light. ZeRO-Offload’s whole thesis is that this idle-but-huge state is exactly what you should relocate to cheaper, more plentiful CPU RAM.
The 16-bytes-per-parameter budget
Make it concrete. For each parameter, standard mixed-precision Adam holds:
| State | Precision | Bytes / param | Where it must live |
|---|---|---|---|
| Weight | fp16 | 2 | GPU (used in fwd/bwd) |
| Gradient | fp16 | 2 | GPU (produced in bwd) |
| Master weight | fp32 | 4 | optimizer only |
| Momentum (m) | fp32 | 4 | optimizer only |
| Variance (v) | fp32 | 4 | optimizer only |
That sums to 16Ψ bytes, and the split is stark: 4Ψ for the fp16 weight-and-gradient pair that the GPU genuinely needs during compute, and 12Ψ — three quarters of the total — for the fp32 optimizer states that only the update step reads. A 10-billion-parameter model therefore demands 16 × 10×10^9 = 160 GB of state before a single activation is stored. No single commodity GPU has that. But 120 GB of it is optimizer state you can send elsewhere.
The split: what stays, what moves
ZeRO-Offload draws the line exactly along that 4Ψ / 12Ψ seam. The GPU keeps the fp16 weights (it needs them to run the forward pass) and computes gradients in the backward pass. Everything the optimizer owns — the fp32 master weights, momentum m, and variance v — is pinned in CPU memory, and the Adam update runs on the CPU.
The per-step data flow follows naturally. As the backward pass finishes each gradient, that fp16 gradient is streamed off the GPU to the CPU. The CPU then runs Adam entirely in fp32, mutating m, v, and the master weights in place. It casts the updated master weights back to fp16 and copies them onto the GPU, overwriting the stale weights just in time for the next forward pass. The GPU never holds the fp32 triple at all: its model-state footprint drops from 16Ψ to roughly 2Ψ (the fp16 weights), an 8× reduction — leaving VRAM free for activations and a bigger batch.
The CPU-side Adam step
Why is it acceptable to run the update on a CPU that is far slower than the GPU at dense linear algebra? Because the Adam update is not linear algebra — it is a cheap, fully element-wise recipe applied once per parameter:
m ← β1·m + (1-β1)·g
v ← β2·v + (1-β2)·g^2
m̂ = m / (1-β1^t) v̂ = v / (1-β2^t)
w ← w - lr · m̂ / (√v̂ + ε)That is a handful of multiplies, adds, a square, and a square root per parameter — on the order of a dozen FLOPs. Crucially there are no matrix products, so the total optimizer compute is O(Ψ): it scales with the parameter count and, unlike the forward and backward passes, is completely independent of the batch size. DeepSpeed ships a hand-tuned SIMD “CPU-Adam” kernel for exactly this loop. The bottleneck on the CPU is not arithmetic but memory bandwidth — the step must stream the 12Ψ bytes of fp32 state through the cores once — which is precisely what CPU DRAM is good at.
Why the update is cheap enough for the CPU
The trade only works because of an asymmetry in how the two halves scale. The GPU’s forward and backward passes cost about 6Ψ FLOPs per token, so a step that processes a batch of B tokens burns roughly 6Ψ·B FLOPs. The CPU’s Adam update costs a fixed ~12Ψ FLOPs regardless of B.
The ratio of GPU work to CPU work is therefore about 6B / 12 = B/2 — it grows linearly with the batch. Process a handful of tokens per step and the puny CPU update dominates wall-clock time; process tens of thousands of tokens per step and the GPU compute towers over it, so the optimizer step becomes a rounding error you can hide behind the next backward pass. This is the design’s central insight: the one part of training whose cost does not grow with batch size is the one part you offload, because a large batch amortizes it into invisibility. Offload the forward pass instead and you would be shipping O(B) work to the wrong processor.
The PCIe transfer cost
The second cost is communication, and it obeys the same O(Ψ) law. Each step moves two things across the PCIe bus: the gradients out to the CPU (2Ψ bytes in fp16) and the freshly updated weights back to the GPU (2Ψ bytes in fp16). That is 4Ψ bytes of PCIe traffic per step — and ZeRO-Offload is built so this is provably the minimum: you cannot update on the CPU without sending gradients down and results back up.
Like the CPU compute, this volume is fixed per step and independent of batch size. Bandwidth is the constraint: PCIe Gen3 ×16 delivers roughly 12 GB/s per direction and Gen4 about 25 GB/s. Divide 4Ψ bytes by that rate to get the transfer time. Because it is a fixed cost per step, the same lever that hides the CPU update — a large batch, which stretches the GPU compute per step — is what buys you the time to overlap these copies with computation rather than stalling on them.
A worked example: a 10B model on one GPU
Take Ψ = 10×10^9 parameters. Full Adam state is 16Ψ = 160 GB — impossible on a 32 GB or even 80 GB GPU. ZeRO-Offload leaves 2Ψ = 20 GB of fp16 weights on the GPU and relocates 12Ψ = 120 GB of optimizer state (plus the streamed gradients) into CPU RAM, where 120 GB is ordinary. The GPU footprint for model state collapses from 160 GB to ~20 GB, and the freed VRAM absorbs activations — a 10B model now trains on a single 32 GB card.
Now the per-step overhead. PCIe traffic is 4Ψ = 40 GB; at Gen4’s 25 GB/s that is about 1.6 s of copying. The CPU-Adam step streams 12Ψ = 120 GB of state at, say, 80 GB/s DRAM bandwidth — roughly 1.5 s. If the GPU forward and backward take 6 s for a large batch, both the ~1.5 s update and the ~1.6 s of copies overlap behind it and cost almost nothing. Shrink the batch until the GPU pass is under 3 s, and that same fixed overhead becomes the thing you wait on.
The throughput trade
So the honest summary is that ZeRO-Offload buys capacity with throughput. You are almost always slower per step than an imaginary GPU big enough to hold everything, because you have inserted a CPU update and two PCIe crossings into the critical path. What you get in return is the ability to train a model that otherwise simply would not fit — and often at a fraction of the hardware cost of the multi-GPU cluster that would be the alternative.
The verdict hinges entirely on batch size. Because both overheads (the update and the transfer) are fixed per step while GPU compute grows with tokens, a large batch amortizes the offload cost toward zero and the throughput penalty can shrink to a modest percentage. A small batch — forced by short sequences, a memory-tight activation budget, or latency-sensitive fine-tuning — leaves the fixed cost exposed and the penalty grows. If you are offloading, push the batch as large as your freed VRAM allows; that is the mechanism that makes the scheme economical.
Practical implications and pitfalls
Several details decide whether offload feels smooth or painful. Use pinned (page-locked) host memory for the CPU-side buffers — pageable memory forces an extra staging copy and can halve effective PCIe bandwidth. Ensure your CPU actually has the DRAM: 12Ψ bytes of fp32 state is a hard requirement, and undersized host RAM turns the scheme into swapping and ruin. Watch that the update and the copies genuinely overlap compute; a naive implementation that runs them synchronously pays the full serial cost.
The most common disappointment is running offload with a tiny batch and concluding it is slow — the tool was mis-used, not defective. Offload also composes: pair it with gradient accumulation to enlarge the effective batch, or with activation checkpointing to free still more VRAM. And remember what it does not solve — the fp16 weights and activations still sit on the GPU, so past a point even an empty optimizer will not make an enormous model fit.
Sibling: when even CPU RAM runs out (NVMe)
ZeRO-Offload’s ceiling is the size of host DRAM: it can only relocate as much optimizer state as the CPU can hold. For a 10B model that is a comfortable 120 GB, but scale to hundreds of billions of parameters and 12Ψ outgrows any reasonable RAM budget too. That is where its sibling picks up.
ZeRO-Infinity extends the same idea one storage tier further, spilling optimizer states (and even parameters and gradients) onto NVMe SSDs — terabytes of cheap capacity in exchange for far lower bandwidth and higher latency than DRAM. The memory-freed arithmetic is identical; only the transfer-cost side of the ledger changes, because an NVMe hop is much slower than a PCIe copy to pinned RAM, so overlap and prefetching matter even more. The companion article NVMe offload & ZeRO-Infinity works that bandwidth-versus-capacity trade in full. Think of the two as one continuum: GPU → CPU RAM → NVMe, each tier trading speed for room to grow.