The batch size that matters for optimization is almost never the number you can fit on one device. Effective batch size — also called the global batch — is the count of examples that contribute to a single optimizer step, and it factors cleanly into three independent levers: the micro-batch on each device, the number of gradient-accumulation steps, and the data-parallel world size. That factoring is the whole point: it lets you hold the batch your model actually trains on fixed while you slide the hardware footprint up or down. This piece derives the identity, works a concrete example, shows why the many hardware configurations that reach the same global batch are mathematically equivalent, connects batch size to learning rate, and is honest about the handful of places where the equivalence quietly stops holding.
The identity
Effective batch size is a product of three factors:
B_eff = b_micro × n_accum × W
b_micro : examples per device per forward pass
n_accum : forward/backward passes accumulated before one update
W : data-parallel world size (number of devices)
Each factor answers a different question. The micro-batch b_micro is what a single device holds in memory for one forward/backward pass — it is bounded by activation memory. Accumulation n_accum is how many of those passes you sum before touching the weights. The world size W is how many devices run the same model on different data in parallel. Multiply them and you get the number of examples whose gradients are averaged into one step. The optimizer never sees b_micro, n_accum, or W individually; it only ever sees B_eff. That single fact is what makes the decomposition useful.
Decoupling the batch from the hardware
The reason to write the batch as a product is that it separates a statistical choice from an engineering constraint. The value of B_eff is a hyperparameter of your optimization — it controls gradient-noise and interacts with the learning rate. The micro-batch, by contrast, is dictated by how much activation memory one device has. These two concerns have nothing to do with each other, and the identity lets you satisfy both at once.
Concretely: pick the B_eff your recipe wants, then measure the largest b_micro that fits, then solve for the remaining levers. If you have many devices, W does the work and each step is fast. If you have one small CPU box, W = 1 and accumulation carries the entire load. Either way the model trains on the same global batch. The batch your optimizer sees is deliberately decoupled from the memory on any one piece of hardware — that is the property the whole scheme exists to provide.
A worked example
Suppose your training recipe calls for a global batch of 1024 sequences, and you have 8 accelerators. You measure that each device can hold 16 sequences of your chosen length before it runs out of activation memory. Solve for accumulation:
B_eff = b_micro × n_accum × W
1024 = 16 × n_accum × 8
n_accum = 1024 / (16 × 8) = 1024 / 128 = 8
So each device runs 8 forward/backward passes of 16 sequences, accumulating gradients, and all 8 devices do this in parallel; after the 8th pass they all-reduce and take one step. The same 1024 is reachable many ways: on a single device you would need n_accum = 64; on 64 devices you could set n_accum = 1 and b_micro = 16. Every one of these configurations produces the identical update — they differ only in wall-clock time and how much hardware you rent, not in what the model learns.
Why the assemblies are equivalent
The equivalence is not a coincidence; it falls straight out of two facts about gradients. First, differentiation is linear, so the gradient of a sum is the sum of gradients. Second, the loss we optimize is a mean over examples. Put together, the gradient of the mean loss over a batch is exactly the mean of the per-example gradients:
L(θ) = (1/B) Σ_i 𝓁_i(θ)
∇L = (1/B) Σ_i ∇𝓁_i
A mean does not care how you partition the terms before adding them. Splitting the B examples across devices (data parallelism) and across time (accumulation) just regroups the summation; the all-reduce averages the device partials and the accumulator averages the temporal partials, and both operations commute with the outer mean. As long as every partial is scaled consistently and the final result is divided by the total B_eff, the reassembled gradient is bit-for-bit the average you would have gotten from one giant batch on one impossible device. That is the theorem the whole technique rests on.
The normalization that makes it exact
The equivalence above assumed ‘divided by the total.’ That assumption is precisely where implementations go wrong. If each accumulation step computes a mean loss over its own micro-batch and you simply add those gradients across n_accum steps without dividing by n_accum, your effective gradient is n_accum times too large — you have silently multiplied your learning rate by 8 in the example above.
The fix is to scale each micro-batch loss by 1 / n_accum before calling backward, so the accumulated sum lands as the correct global mean. Data parallelism has the mirror-image requirement: the gradient all-reduce must average (not sum) across the W devices, which standard frameworks do by default. Get either normalization wrong and the run still trains, still prints a falling loss, and quietly uses the wrong batch size and the wrong effective learning rate — no error message, just a recipe that no longer matches the one you meant to reproduce.
Coupling to the learning rate
Changing B_eff is not free: the batch size sets the variance of the gradient estimate, and that variance is coupled to the learning rate. A larger batch is a lower-noise estimate of the true gradient, so you can safely take a bigger step. The most common heuristic is the linear scaling rule: when you multiply the batch by k, multiply the learning rate by k as well.
lr(B) ≈ lr_base × (B / B_base)
The intuition: over a fixed number of examples, a k-times-larger batch takes k times fewer steps, so each step must move k times as far to cover the same ground. The rule holds well up to a point and needs a learning-rate warmup to survive the unstable early steps. Some recipes prefer square-root scaling (lr ∝ √B), which matches the way gradient-noise standard deviation actually falls. Both are approximations, and knowing which one your recipe assumed matters more than which is theoretically purest.
Diminishing returns and the critical batch size
Scaling the batch buys you parallelism — fewer, larger steps that finish a training run in less wall-clock time — but only up to a point. Below a certain size, gradient noise dominates and doubling the batch roughly halves the number of steps you need. Above it, the gradient is already a near-perfect estimate of the true direction, and adding more examples per step barely reduces the steps required. That crossover is the critical batch size.
Past the critical batch size you are spending compute for almost no reduction in training time: the run becomes compute-bound rather than noise-bound. This is why you cannot simply crank B_eff to the sky and expect proportional speedups — the linear scaling rule for the learning rate also breaks down in the same regime, and training can become unstable or waste enormous compute for a marginal step-count win. The critical batch size grows over training and is larger for harder tasks, which is why large-model recipes ramp the batch size up.
Where the equivalence holds exactly
For the core of a transformer, the reassembly is genuinely exact up to floating-point rounding. The linear layers, attention, and the loss are all per-example computations: each sequence produces its own gradient independent of the others in the batch, so partitioning the batch across devices and time changes nothing about any individual gradient.
Crucially, transformers normalize with LayerNorm, which computes its statistics across the feature dimension of a single token — it never looks at other examples in the batch. That means the normalization is invariant to how the batch is split, and the whole family of micro-batch and accumulation configurations stays equivalent. This is a happy accident of the architecture: it is exactly the property that makes gradient accumulation a safe, transparent tool for transformers, and the reason the batch-decoupling trick is leaned on so heavily in large-language-model training.
Where it quietly breaks
The equivalence assumed every example’s contribution depends only on that example. Any layer whose output for one example depends on the rest of its micro-batch breaks that assumption. The classic offender is BatchNorm: its mean and variance are computed over the current micro-batch, so a batch of 128 split into 8 micro-batches of 16 normalizes with statistics from 16 examples, not 128 — a different function entirely. SyncBatchNorm exists to paper over the data-parallel half of this, but accumulation still changes the per-step statistics. (Transformers dodge this by using LayerNorm, which is why the problem rarely surfaces here.)
Two subtler breaks matter even with LayerNorm. Dropout and other stochastic layers draw fresh random masks on every forward pass, so splitting a batch into more passes changes the realized noise — the expected gradient is unchanged, but the specific update differs, and reproducibility across configurations is lost. And with variable-length sequences, if each micro-batch normalizes the loss by its own token count and you then average across accumulation steps, you get an unweighted average of per-batch means, not the true token-weighted global mean — long and short batches get equal say when they should not.
The CPU-SLM angle
On a single CPU or a small box — the home turf of small language models — the identity collapses to B_eff = b_micro × n_accum, with W = 1. There is no data parallelism to lean on, so accumulation is the only lever that reaches a respectable global batch. If a recipe wants B_eff = 256 and you can fit a micro-batch of 4, you set n_accum = 64 and run 64 sequential passes per update.
The cost is pure wall-clock: those 64 passes run one after another, so a step takes 64 times as long as a single micro-batch pass. Accumulation buys statistical batch size with time rather than hardware, which is exactly the right trade when hardware is what you lack. It also keeps a small-scale training run faithful to a recipe designed for a GPU cluster: you match the B_eff and the learning rate exactly, accept the slower steps, and get the same optimization trajectory.