When people size a training run they instinctively count parameters, but the parameters are rarely the biggest line item. The optimizer quietly keeps its own running memory for every single weight — a first moment, a second moment, sometimes a high-precision master copy — and for Adam-family optimizers that bookkeeping is larger than the model itself. This article works the arithmetic from first principles: how many bytes per parameter each optimizer costs, why the fp32 master weights plus the two Adam moments come to 12 bytes/param, how that dominates the training footprint, and the two standard escapes — 8-bit optimizers and factored state — that buy the memory back.
The four buckets of training memory
Training memory splits into four buckets, and confusing them is the root of most out-of-memory surprises. First, the parameters themselves: P weights at some precision. Second, the gradients, one per parameter, from the backward pass. Third, the optimizer state — the per-parameter buffers the update rule maintains between steps. Fourth, the activations stashed for backprop, which scale with batch size and sequence length rather than with P.
The first three all scale with P and are fixed the moment you choose a model and an optimizer; activations are the elastic bucket you tune with batch size. This piece is about the third bucket, because for the standard recipe it is the single largest of the fixed three, and the one most people forget to count until the allocator refuses.
SGD: the free baseline
Plain stochastic gradient descent is the reference point because it carries no per-parameter state at all. The update is memoryless: θ ← θ − η · g, where g is this step’s gradient and η is the learning rate. Nothing from previous steps is retained, so the optimizer’s footprint is exactly zero bytes per parameter.
That makes SGD the cheapest optimizer in memory terms: the training footprint is just parameters, gradients, and activations. The catch is that pure SGD converges slowly on the sharp, ill-conditioned loss surfaces of transformers, which is why almost nobody trains large language models with it. Every memory-hungry optimizer below is buying faster, more stable convergence with bytes — and the question of this article is how many bytes, and whether the trade is worth it on a constrained box.
Momentum: one buffer, four bytes
The first upgrade is momentum, which smooths the gradient into a running average of recent directions. It keeps one extra buffer, the velocity v, the same shape as the parameters:
v ← β · v + g
θ ← θ − η · vBecause v has one entry per parameter, momentum costs one buffer per parameter. In fp32 that is 4 bytes/param of optimizer state — the model’s memory grows by roughly a third on top of parameters and gradients. Momentum (and its Nesterov variant) is the sweet spot for many vision and convolutional models: much better convergence than plain SGD for a modest, single-buffer memory cost. It is when we move to the adaptive optimizers that the state budget doubles again, because they track a second statistic per weight.
Adam: two moments per parameter
Adam and its weight-decay-corrected cousin AdamW are the default for transformers because they adapt the effective learning rate per parameter. To do that they track two running statistics for every weight: the first moment m (a momentum-like mean of the gradient) and the second moment v (a mean of the squared gradient).
m ← β_1 · m + (1 − β_1) · g
v ← β_2 · v + (1 − β_2) · g^2
θ ← θ − η · m̂ / (√v̂ + ε)Both m and v are full-size, one entry per parameter. Stored in fp32 that is 4 + 4 = 8 bytes/param of optimizer state before counting weights or gradients — already twice the size of a fp32 copy of the model. Under mixed precision a third piece joins them.
The fp32 master copy, and why it exists
Modern training runs the forward and backward pass in a low-precision format — fp16 or bf16 — for speed, but low precision cannot safely accumulate the tiny updates that fill a long training run. The update η · m̂ / √v̂ is often orders of magnitude smaller than the weight it modifies, and in fp16 it rounds away to nothing.
The fix is a master copy of the weights kept in fp32. The optimizer applies updates to this high-precision copy, then casts it down to the fp16/bf16 weights the model computes with. That master copy is 4 bytes/param, counted as part of the optimizer state because it exists only to serve the update rule. Add it to Adam’s two moments and the total optimizer-state cost lands at 4 (master) + 4 (m) + 4 (v) = 12 bytes/param — the number worth memorizing for any mixed-precision Adam run.
The full 16-bytes-per-parameter picture
Put the fixed buckets together for a standard mixed-precision AdamW recipe and the per-parameter accounting is:
| Item | Precision | Bytes/param |
|---|---|---|
| Weights (compute copy) | fp16 / bf16 | 2 |
| Gradients | fp16 / bf16 | 2 |
| Master weights | fp32 | 4 |
| Adam m (first moment) | fp32 | 4 |
| Adam v (second moment) | fp32 | 4 |
| Optimizer state subtotal | 12 | |
| Total (excl. activations) | 16 |
This is the well-known ‘16 bytes per parameter’ figure from the ZeRO analysis. Of those 16 bytes, 12 belong to the optimizer — three-quarters of the fixed footprint is bookkeeping, not the model. The 2-byte compute weights everyone quotes are the smallest slice of all.
A worked example: 7 billion parameters
Make it concrete with a 7B-parameter model, P = 7 × 10^9, trained with mixed-precision AdamW. Multiply each row by P:
weights (fp16) : 7e9 × 2 = 14 GB
gradients (fp16) : 7e9 × 2 = 14 GB
master (fp32) : 7e9 × 4 = 28 GB
m (fp32) : 7e9 × 4 = 28 GB
v (fp32) : 7e9 × 4 = 28 GB
-------------------------------------------
optimizer state : 28+28+28 = 84 GB
total (fixed) : = 112 GBThe optimizer state alone is 84 GB — six times the 14 GB of compute weights, and 75% of the 112 GB fixed total. A single 80 GB accelerator cannot even hold the fixed buckets, let alone activations, which is precisely why 7B full-fine-tuning demands multiple devices or an optimizer-state reduction.
Why it dominates the footprint
The reason optimizer state dominates is structural, not incidental. The compute weights and gradients are stored in 2-byte low precision, but every piece of optimizer state is stored in 4-byte fp32, because the whole point of the master copy and the moments is numerical stability that low precision would destroy. So each optimizer buffer is already twice the size of the corresponding compute tensor, and Adam keeps three such buffers.
Three fp32 buffers against one fp16 weight is a 6-to-1 ratio, fixed regardless of model size — it does not amortize away as you scale up. This is why memory-reduction research targets the optimizer first: shaving activations helps the elastic bucket, but the optimizer is the largest of the inelastic buckets, the one you cannot escape by lowering the batch size. The next sections cover the mainstream ways to shrink it.
8-bit optimizers and bitsandbytes
The first escape is to stop storing the moments in fp32. An 8-bit optimizer — popularized by the bitsandbytes library’s Adam8bit — quantizes m and v to a single byte each using block-wise dynamic quantization: the tensor is split into small blocks, each block normalized by its own maximum, and the values mapped onto a nonlinear 8-bit grid. Because normalization is per block, a few large values in one block do not crush the precision of another.
The effect on the budget is direct. The two moments fall from 4 + 4 = 8 bytes to 1 + 1 = 2 bytes. Keeping the fp32 master copy, optimizer state drops from 12 to 4 + 1 + 1 = 6 bytes/param — roughly half. For the 7B example that is 84 GB down to about 42 GB, and convergence stays close to fp32 Adam in practice, which is why 8-bit optimizers are a near-free win on memory-constrained hardware.
Adafactor: factored second moment
The second escape attacks the shape of the state. Adam’s second moment v is a full matrix for every weight matrix W of shape [n, m], costing n × m numbers. Adafactor observes that this matrix can be well approximated by an outer product of two vectors: a row statistic R of length n and a column statistic C of length m, with v[i,j] ≈ R[i] · C[j] / sum(R).
Storing R and C costs n + m numbers instead of n × m — sub-linear in the matrix, a dramatic reduction for large layers. Adafactor also, by default, drops the first moment m entirely, leaving optimizer state that is a small fraction of Adam’s: essentially the master copy plus a thin factored second moment. The cost is slightly noisier updates and extra hyperparameter care, but for very large models the memory saving is decisive.
Pitfalls when you count
A few mistakes recur. First, forgetting the master copy: people count 8 bytes for Adam’s two moments and stop, missing the 4-byte fp32 weights, which undercounts the fixed footprint by a quarter. Second, assuming bf16 removes the master copy — bf16 has a wide exponent but only 8 bits of mantissa, so it still cannot accumulate tiny updates, and a fp32 master copy is still standard.
Third, miscounting gradients: they are a separate 2-byte bucket, not part of optimizer state. Finally, remember these reductions trade memory for something: 8-bit optimizers add quantization noise and Adafactor drops the first moment. On a CPU or a single small accelerator, an 8-bit optimizer or Adafactor is often the difference between a run that fits and one that never starts.