Pipeline parallelism splits a model the long way — by depth. You cut the stack of layers into contiguous stages, put each stage on its own device, and pass activations forward from one stage to the next like parts moving down an assembly line. It is what lets a model too tall to fit on a single GPU train at all: stage 0 holds the embedding and the first few blocks, stage 1 the next few, and so on to the final stage with the output head. The whole subtlety is timing. Run one batch naively and most of your devices sit idle most of the time — the notorious pipeline bubble. The fix is to chop the batch into microbatches and stream them so every stage stays busy, and the arithmetic of how well that works reduces to one clean formula. This piece derives that formula, works a numeric example, explains the 1F1B schedule that tames activation memory, and draws the line between pipeline and tensor parallelism.
Splitting a model by depth
A transformer is a deep stack: an embedding, L identical decoder blocks, and an output projection. Pipeline parallelism partitions that stack into p contiguous groups of layers — the stages — and assigns stage i to device i. If the model has 48 blocks and p = 4, each stage owns 12 consecutive blocks. During a forward pass, device 0 runs its 12 blocks on the input, ships the resulting activations to device 1, which runs the next 12, and so on until the last stage produces logits and the loss.
The appeal is memory. Each device stores only 1/p of the parameters, plus its slice of optimizer state and gradients — so a model whose weights and states dwarf a single GPU’s memory becomes trainable by adding stages. Communication is modest too: a stage only ever talks to its two neighbours, and only sends the activation tensor at the cut point. That locality is what makes pipelining scale across nodes where bandwidth between far-apart GPUs is scarce.
The naive schedule and the bubble
Run a single batch through a p-stage pipeline the obvious way and the inefficiency is stark. Stage 0 computes its forward pass; only then can stage 1 start; only then stage 2, and so on. The forward wavefront takes p steps just to reach the last stage. Then the backward pass unwinds in reverse, another p steps. At any given moment, exactly one stage is doing useful work and the other p-1 are idle — waiting for data that has not arrived yet, or waiting for gradients coming back.
That idle time is the pipeline bubble. Picture a spacetime diagram with devices on the vertical axis and time on the horizontal: the busy cells form a diagonal band of forward work, then a diagonal band of backward work, and everything outside those bands is wasted. With one batch and p stages, utilisation is roughly 1/p — a 4-stage pipeline running at 25% efficiency, an 8-stage one at 12.5%. The devices you added to fit the model are mostly stalled. Something has to keep them fed.
Microbatching: GPipe fills the pipe
The insight behind GPipe is that the bubble comes from having only one unit of work in flight. Split the minibatch into m smaller microbatches and feed them into stage 0 one after another. As soon as microbatch 1 clears stage 0, stage 0 starts microbatch 2 while stage 1 works on microbatch 1. After a short fill period, every stage is processing a different microbatch simultaneously — the pipeline is full and all p devices are busy.
The bubble does not vanish; it is amortised. There is still a fill phase at the start (stages downstream of 0 wait for the wavefront to reach them) and a drain phase at the end (upstream stages finish while the tail empties). But those fixed p-1 steps of ramp are now shared across m microbatches of steady-state work instead of a single batch, so the overhead shrinks as a fraction of the whole. GPipe accumulates gradients across all m microbatches, then applies one optimizer step — so the math is identical to a single large batch; only the scheduling changes.
Deriving the bubble fraction
Let each microbatch cost one unit of time per stage for the forward pass and, by convention, we count forward and backward in the same accounting. Ignore communication for the moment and measure everything in these unit steps.
useful work per device = m (all m microbatches)
bubble (fill + drain) = p - 1 (the (p-1) ramp)
total length T = m + (p - 1)
bubble (p - 1)
bubble fraction = ------ = -----------
total m + p - 1So the fraction of time lost to the bubble is (p-1) / (m + p-1). Read it two ways. Fix p and grow m: the denominator grows while the numerator stays put, so the bubble fraction falls toward 0 — more microbatches, less waste. Fix m and grow p: both numerator and denominator grow, but the numerator grows faster relative to the useful m, so deeper pipelines bubble more. The efficiency is simply 1 − (p-1)/(m+p-1) = m/(m+p-1). The practical rule of thumb that falls straight out: keep m well above p — a common target is m ≥ 4p.
A worked example
Take p = 4 stages and m = 8 microbatches. The bubble fraction is (4-1)/(8+4-1) = 3/11 ≈ 0.27 — about 27% of the time wasted, 73% efficiency. Not great. Now raise the microbatch count to m = 32 without touching the stage count: (4-1)/(32+4-1) = 3/35 ≈ 0.086 — under 9% bubble, over 91% efficiency. Same hardware, same model split, four times the microbatches, and the idle time collapsed.
Contrast that with making the pipeline deeper. Hold m = 8 and go to p = 8 stages: (8-1)/(8+8-1) = 7/15 ≈ 0.47 — nearly half the time is bubble. This is the fundamental tension of pipelining: adding stages buys memory headroom (each holds 1/p of the model) but costs utilisation unless you also raise m. The lever you control cheaply is m; its ceiling is memory, because more in-flight microbatches means more stored activations awaiting their backward pass — exactly the problem the next schedule attacks.
1F1B: cutting activation memory
GPipe’s simple schedule runs all m forward passes first, then all m backward passes. That maximises how many activation sets a stage must keep alive at once — up to m of them on the first stage, since its outputs are needed until their backward pass finally comes around at the very end. Activation memory scales with m, which fights directly against the desire to push m high.
1F1B (‘one-forward-one-backward’), used by PipeDream and Megatron-LM, reorders the work. Once the pipeline is full, each stage alternates: do one microbatch’s forward, then immediately do one (earlier) microbatch’s backward, freeing that activation before starting the next forward. The number of activations a stage holds in flight is now bounded by the pipeline depth — roughly p — not by m. The bubble fraction is unchanged (1F1B reorders the same operations, it does not remove the fill/drain), but activation memory is decoupled from the microbatch count, so you can raise m to shrink the bubble without running out of memory. An interleaved variant assigns each device several non-contiguous stage chunks, shortening each wavefront hop to further shave the bubble at the cost of more communication.
Point-to-point communication
Pipeline stages exchange data through point-to-point primitives — a send on one device paired with a matching recv on its neighbour — not the collective all-reduce that data and tensor parallelism lean on. In the forward direction, stage i sends its output activation tensor to stage i+1. In the backward direction, stage i+1 sends the gradient of its input back to stage i. Each message is a single activation-shaped tensor of size roughly [microbatch × sequence × hidden].
This makes pipelining bandwidth-friendly. Only adjacent stages talk, only at the cut boundaries, and the payload is one tensor per microbatch per direction — independent of how many layers sit inside the stage. That locality is why pipeline parallelism is the natural choice for spanning across nodes, where inter-node links are slow: you place the high-bandwidth splits (tensor parallelism) inside a node and let the thin pipeline links cross between nodes. Well-implemented pipelines also overlap these transfers with computation — a stage sends microbatch k’s activations while already computing microbatch k+1 — so communication hides behind work rather than adding to the bubble.
Pipeline vs tensor parallelism
The two split a model along perpendicular axes, and confusing them is a common error. Pipeline parallelism splits between layers — it cuts the depth, giving each device a contiguous block of whole layers. Tensor parallelism splits within a layer — it shards the individual weight matrices of a single layer (the attention projections, the MLP’s W_1 and W_2) across devices, so every device computes a slice of the same layer.
The consequences differ sharply. Tensor parallelism communicates inside every layer — an all-reduce per attention block and per MLP block, on the critical path of every forward and backward — so it demands fat, low-latency links and is kept within a single node (NVLink-class). It has no bubble but heavy, frequent traffic. Pipeline parallelism communicates only at p-1 stage boundaries with cheap point-to-point sends, tolerates slow links, and scales across nodes — but pays the bubble tax. They are complementary: large-scale training composes tensor parallelism inside a node, pipeline parallelism across nodes, and data parallelism across replicas — the ‘3D parallelism’ that trains the biggest models.
Practical tuning and pitfalls
Getting a pipeline efficient is mostly about three balances. First, balance the stages: the pipeline runs at the speed of its slowest stage, so an uneven layer split — or forgetting that the first stage carries the embedding and the last carries the loss and output head — creates a straggler that stalls everyone. Aim for equal per-stage compute, not equal layer counts. Second, choose m deliberately: the (p-1)/(m+p-1) formula tells you the bubble, so pick m to hit an efficiency target, bounded by activation memory (which 1F1B relaxes).
The pitfalls are predictable once you know the formula. Too few microbatches and the bubble eats your throughput — the most common mistake. Too many stages relative to m does the same from the other direction. Ignoring the load imbalance from the embedding and head stages silently caps utilisation, and treating pipeline communication as free ignores that unoverlapped activation transfers widen the bubble. On modest hardware — the CPU-SLM regime this series cares about — pipelining is usually overkill for a small model that fits in memory; its payoff is precisely when depth, not width, overflows the device.
p-1 ramp across m microbatches, and the bubble fraction is exactly (p-1)/(m+p-1) — so keep m well above p (roughly m ≥ 4p) and efficiency climbs toward m/(m+p-1). 1F1B reorders forward and backward passes to cap activation memory at about p rather than m, letting you raise m freely. Stages talk only to neighbours via cheap point-to-point send/recv, which is why pipelines span slow inter-node links. Keep it distinct from tensor parallelism, which splits within a layer and all-reduces on every block — the two are perpendicular, and the largest training runs use both together.