Splitting a model by depth
A transformer is a deep stack: an embedding, L identical decoder blocks, and an output projection. Pipeline parallelism partitions that stack into p contiguous groups of layers — the stages — and assigns stage i to device i. If the model has 48 blocks and p = 4, each stage owns 12 consecutive blocks. During a forward pass, device 0 runs its 12 blocks on the input, ships the resulting activations to device 1, which runs the next 12, and so on until the last stage produces logits and the loss.
The appeal is memory. Each device stores only 1/p of the parameters, plus its slice of optimizer state and gradients — so a model whose weights and states dwarf a single GPU’s memory becomes trainable by adding stages. Communication is modest too: a stage only ever talks to its two neighbours, and only sends the activation tensor at the cut point. That locality is what makes pipelining scale across nodes where bandwidth between far-apart GPUs is scarce.
The naive schedule and the bubble
Run a single batch through a p-stage pipeline the obvious way and the inefficiency is stark. Stage 0 computes its forward pass; only then can stage 1 start; only then stage 2, and so on. The forward wavefront takes p steps just to reach the last stage. Then the backward pass unwinds in reverse, another p steps. At any given moment, exactly one stage is doing useful work and the other p-1 are idle — waiting for data that has not arrived yet, or waiting for gradients coming back.
That idle time is the pipeline bubble. Picture a spacetime diagram with devices on the vertical axis and time on the horizontal: the busy cells form a diagonal band of forward work, then a diagonal band of backward work, and everything outside those bands is wasted. With one batch and p stages, utilisation is roughly 1/p — a 4-stage pipeline running at 25% efficiency, an 8-stage one at 12.5%. The devices you added to fit the model are mostly stalled. Something has to keep them fed.
Microbatching: GPipe fills the pipe
The insight behind GPipe is that the bubble comes from having only one unit of work in flight. Split the minibatch into m smaller microbatches and feed them into stage 0 one after another. As soon as microbatch 1 clears stage 0, stage 0 starts microbatch 2 while stage 1 works on microbatch 1. After a short fill period, every stage is processing a different microbatch simultaneously — the pipeline is full and all p devices are busy.
The bubble does not vanish; it is amortised. There is still a fill phase at the start (stages downstream of 0 wait for the wavefront to reach them) and a drain phase at the end (upstream stages finish while the tail empties). But those fixed p-1 steps of ramp are now shared across m microbatches of steady-state work instead of a single batch, so the overhead shrinks as a fraction of the whole. GPipe accumulates gradients across all m microbatches, then applies one optimizer step — so the math is identical to a single large batch; only the scheduling changes.
Deriving the bubble fraction
Let each microbatch cost one unit of time per stage for the forward pass and, by convention, we count forward and backward in the same accounting. Ignore communication for the moment and measure everything in these unit steps.
useful work per device = m (all m microbatches)
bubble (fill + drain) = p - 1 (the (p-1) ramp)
total length T = m + (p - 1)
bubble (p - 1)
bubble fraction = ------ = -----------
total m + p - 1So the fraction of time lost to the bubble is (p-1) / (m + p-1). Read it two ways. Fix p and grow m: the denominator grows while the numerator stays put, so the bubble fraction falls toward 0 — more microbatches, less waste. Fix m and grow p: both numerator and denominator grow, but the numerator grows faster relative to the useful m, so deeper pipelines bubble more. The efficiency is simply 1 − (p-1)/(m+p-1) = m/(m+p-1). The practical rule of thumb that falls straight out: keep m well above p — a common target is m ≥ 4p.
A worked example
Take p = 4 stages and m = 8 microbatches. The bubble fraction is (4-1)/(8+4-1) = 3/11 ≈ 0.27 — about 27% of the time wasted, 73% efficiency. Not great. Now raise the microbatch count to m = 32 without touching the stage count: (4-1)/(32+4-1) = 3/35 ≈ 0.086 — under 9% bubble, over 91% efficiency. Same hardware, same model split, four times the microbatches, and the idle time collapsed.
Contrast that with making the pipeline deeper. Hold m = 8 and go to p = 8 stages: (8-1)/(8+8-1) = 7/15 ≈ 0.47 — nearly half the time is bubble. This is the fundamental tension of pipelining: adding stages buys memory headroom (each holds 1/p of the model) but costs utilisation unless you also raise m. The lever you control cheaply is m; its ceiling is memory, because more in-flight microbatches means more stored activations awaiting their backward pass — exactly the problem the next schedule attacks.
1F1B: cutting activation memory
GPipe’s simple schedule runs all m forward passes first, then all m backward passes. That maximises how many activation sets a stage must keep alive at once — up to m of them on the first stage, since its outputs are needed until their backward pass finally comes around at the very end. Activation memory scales with m, which fights directly against the desire to push m high.
1F1B (‘one-forward-one-backward’), used by PipeDream and Megatron-LM, reorders the work. Once the pipeline is full, each stage alternates: do one microbatch’s forward, then immediately do one (earlier) microbatch’s backward, freeing that activation before starting the next forward. The number of activations a stage holds in flight is now bounded by the pipeline depth — roughly p — not by m. The bubble fraction is unchanged (1F1B reorders the same operations, it does not remove the fill/drain), but activation memory is decoupled from the microbatch count, so you can raise m to shrink the bubble without running out of memory. An interleaved variant assigns each device several non-contiguous stage chunks, shortening each wavefront hop to further shave the bubble at the cost of more communication.