The identity
Effective batch size is a product of three factors:
B_eff = b_micro × n_accum × W
b_micro : examples per device per forward pass
n_accum : forward/backward passes accumulated before one update
W : data-parallel world size (number of devices)
Each factor answers a different question. The micro-batch b_micro is what a single device holds in memory for one forward/backward pass — it is bounded by activation memory. Accumulation n_accum is how many of those passes you sum before touching the weights. The world size W is how many devices run the same model on different data in parallel. Multiply them and you get the number of examples whose gradients are averaged into one step. The optimizer never sees b_micro, n_accum, or W individually; it only ever sees B_eff. That single fact is what makes the decomposition useful.
Decoupling the batch from the hardware
The reason to write the batch as a product is that it separates a statistical choice from an engineering constraint. The value of B_eff is a hyperparameter of your optimization — it controls gradient-noise and interacts with the learning rate. The micro-batch, by contrast, is dictated by how much activation memory one device has. These two concerns have nothing to do with each other, and the identity lets you satisfy both at once.
Concretely: pick the B_eff your recipe wants, then measure the largest b_micro that fits, then solve for the remaining levers. If you have many devices, W does the work and each step is fast. If you have one small CPU box, W = 1 and accumulation carries the entire load. Either way the model trains on the same global batch. The batch your optimizer sees is deliberately decoupled from the memory on any one piece of hardware — that is the property the whole scheme exists to provide.
A worked example
Suppose your training recipe calls for a global batch of 1024 sequences, and you have 8 accelerators. You measure that each device can hold 16 sequences of your chosen length before it runs out of activation memory. Solve for accumulation:
B_eff = b_micro × n_accum × W
1024 = 16 × n_accum × 8
n_accum = 1024 / (16 × 8) = 1024 / 128 = 8
So each device runs 8 forward/backward passes of 16 sequences, accumulating gradients, and all 8 devices do this in parallel; after the 8th pass they all-reduce and take one step. The same 1024 is reachable many ways: on a single device you would need n_accum = 64; on 64 devices you could set n_accum = 1 and b_micro = 16. Every one of these configurations produces the identical update — they differ only in wall-clock time and how much hardware you rent, not in what the model learns.
Why the assemblies are equivalent
The equivalence is not a coincidence; it falls straight out of two facts about gradients. First, differentiation is linear, so the gradient of a sum is the sum of gradients. Second, the loss we optimize is a mean over examples. Put together, the gradient of the mean loss over a batch is exactly the mean of the per-example gradients:
L(θ) = (1/B) Σ_i 𝓁_i(θ)
∇L = (1/B) Σ_i ∇𝓁_i
A mean does not care how you partition the terms before adding them. Splitting the B examples across devices (data parallelism) and across time (accumulation) just regroups the summation; the all-reduce averages the device partials and the accumulator averages the temporal partials, and both operations commute with the outer mean. As long as every partial is scaled consistently and the final result is divided by the total B_eff, the reassembled gradient is bit-for-bit the average you would have gotten from one giant batch on one impossible device. That is the theorem the whole technique rests on.
The normalization that makes it exact
The equivalence above assumed ‘divided by the total.’ That assumption is precisely where implementations go wrong. If each accumulation step computes a mean loss over its own micro-batch and you simply add those gradients across n_accum steps without dividing by n_accum, your effective gradient is n_accum times too large — you have silently multiplied your learning rate by 8 in the example above.
The fix is to scale each micro-batch loss by 1 / n_accum before calling backward, so the accumulated sum lands as the correct global mean. Data parallelism has the mirror-image requirement: the gradient all-reduce must average (not sum) across the W devices, which standard frameworks do by default. Get either normalization wrong and the run still trains, still prints a falling loss, and quietly uses the wrong batch size and the wrong effective learning rate — no error message, just a recipe that no longer matches the one you meant to reproduce.
Coupling to the learning rate
Changing B_eff is not free: the batch size sets the variance of the gradient estimate, and that variance is coupled to the learning rate. A larger batch is a lower-noise estimate of the true gradient, so you can safely take a bigger step. The most common heuristic is the linear scaling rule: when you multiply the batch by k, multiply the learning rate by k as well.
lr(B) ≈ lr_base × (B / B_base)
The intuition: over a fixed number of examples, a k-times-larger batch takes k times fewer steps, so each step must move k times as far to cover the same ground. The rule holds well up to a point and needs a learning-rate warmup to survive the unstable early steps. Some recipes prefer square-root scaling (lr ∝ √B), which matches the way gradient-noise standard deviation actually falls. Both are approximations, and knowing which one your recipe assumed matters more than which is theoretically purest.
Diminishing returns and the critical batch size
Scaling the batch buys you parallelism — fewer, larger steps that finish a training run in less wall-clock time — but only up to a point. Below a certain size, gradient noise dominates and doubling the batch roughly halves the number of steps you need. Above it, the gradient is already a near-perfect estimate of the true direction, and adding more examples per step barely reduces the steps required. That crossover is the critical batch size.
Past the critical batch size you are spending compute for almost no reduction in training time: the run becomes compute-bound rather than noise-bound. This is why you cannot simply crank B_eff to the sky and expect proportional speedups — the linear scaling rule for the learning rate also breaks down in the same regime, and training can become unstable or waste enormous compute for a marginal step-count win. The critical batch size grows over training and is larger for harder tasks, which is why large-model recipes ramp the batch size up.