Pipeline parallelism (PP) takes a model too tall to fit on one device and cuts it horizontally — layer 1–8 on GPU 0, layer 9–16 on GPU 1, and so on — so each device holds only a slice of the network’s depth. That solves the memory problem, but it introduces a scheduling problem: the stages form an assembly line, and an assembly line is only efficient if every station is busy. When a batch first enters, most stages sit idle; when it drains, most sit idle again. That idle time is the pipeline bubble, and almost all of the math of PP is the math of making it small. This piece derives the bubble fraction from first principles, shows why more micro-batches shrink it, compares the GPipe and 1F1B schedules, and works through the memory-versus-bubble tradeoff.
Splitting a model across stages
Start with the shape of the cut. A transformer is a stack of L identical layers. Pipeline parallelism partitions that stack into p contiguous groups called stages, one per device, each holding roughly L / p layers. Device i runs a forward pass on its layers, sends the activations to device i+1, and on the way back receives gradients from i+1 and sends its own to i-1.
The appeal is that per-device parameter memory drops by a factor of p: a 70B model that needs ~140 GB in fp16 for weights alone splits into ~20 GB slices across 8 stages. Only the activations at stage boundaries cross the wire, and that volume — micro_batch × seq_len × d_model per hand-off — is tiny next to the all-reduces tensor parallelism demands. That cheap, point-to-point communication is why PP scales even across nodes where tensor parallelism would choke.
The problem: a strict data dependency
The catch is that the stages are not independent — they are a chain. Stage i+1 cannot begin until stage i has produced its output, and the backward pass cannot begin on any stage until the forward pass has reached the very end and the loss is computed. If you feed the whole batch through as one unit, the timeline is brutal: stage 0 works while stages 1 through p-1 sit idle, then stage 1 works while the rest wait, and so on.
With naive whole-batch execution, only one of p devices is ever active. Utilization is 1/p — an 8-way pipeline running at 12.5 percent efficiency, worse than not splitting at all. Recovering throughput requires breaking the batch into smaller pieces so different stages work on different pieces at once. Those pieces are micro-batches, the lever the rest of the math turns on.
Micro-batches fill the pipe
Split the global batch into m micro-batches. Now stage 0 processes micro-batch 1 and immediately hands it to stage 1; while stage 1 works on micro-batch 1, stage 0 starts micro-batch 2. After a short fill period every stage is working on a different micro-batch simultaneously — the assembly line is full and all p devices run in parallel.
It is exactly CPU instruction pipelining: the first instruction takes several cycles to traverse, but once the pipe is full a result retires every cycle. Here the ‘instruction’ is a micro-batch and the ‘stages’ are devices. The steady state is efficient; the inefficiency lives entirely at the two ends — the fill, while the pipe fills toward full occupancy, and the drain, while the last micro-batches trickle out and early stages run dry. Those two triangles of idle time are the bubble, and their size relative to useful work is what we now compute.
Deriving the bubble fraction
Measure time in units of one stage processing one micro-batch (take forward and backward as one combined unit for now). The last stage cannot start its first micro-batch until that micro-batch has traversed the previous p-1 stages — that is the fill cost, p-1 units. Symmetrically, after the last stage finishes, the pipeline drains for another p-1 units as work clears the remaining stages.
useful work per device = m
fill + drain idle (bubble) = p - 1
total wall-clock time = m + (p - 1)
bubble fraction = (p - 1) / (m + p - 1)This single ratio governs pipeline efficiency; utilization is its complement, m / (m + p - 1). The structure is intuitive: the bubble grows with the stage count p (a longer pipe takes longer to fill and drain) and shrinks as micro-batches m grow (more work to amortize the fixed fill/drain cost against). Everything else in PP scheduling makes one of those two terms more favorable.
A worked example
Take an 8-stage pipeline, p = 8, and run it with m = 8 micro-batches — a common first guess of one micro-batch per stage. The bubble fraction is (8-1)/(8+8-1) = 7/15 ≈ 0.47. Nearly half the pipeline’s wall-clock time is idle. That is a disaster hiding behind a plausible-looking configuration.
Now raise m to 32: 7/39 ≈ 0.18. At m = 64 it is 7/71 ≈ 0.10. To push under 5 percent you need m ≈ 133. The rule of thumb: you want m to be several times p — often m ≥ 4p to 8p — before the bubble stops dominating. Below that, adding pipeline stages can actively slow you down, because each new stage lengthens the fill and drain while the micro-batch count stays put. The number of micro-batches, not the number of GPUs, decides whether PP is worth using at all.
GPipe: all forward, then all backward
The original GPipe schedule is the simplest to reason about: push all m micro-batches forward through the pipeline, then run all m backward passes. Its bubble fraction is exactly the (p-1)/(m+p-1) derived above, and it is easy to implement because forward and backward phases are cleanly separated.
The cost is memory. Because backward for micro-batch 1 does not start until every forward pass is done, GPipe must keep the activations of all m in-flight micro-batches stashed for the backward pass. Peak activation memory therefore scales with m — and m is exactly the quantity you wanted to make large to shrink the bubble. This is the central tension of pipeline parallelism: the bubble pulls m up, and memory pulls it down. GPipe blunts the memory side with activation re-computation (checkpointing), but the fundamental O(m) growth remains.
1F1B: one forward, one backward
The 1F1B schedule (one-forward-one-backward, from PipeDream and adopted by Megatron-LM) attacks the memory problem without changing the bubble fraction. After the fill phase, each stage alternates: do one forward micro-batch, then immediately do one backward micro-batch, in steady lockstep. Because a backward pass runs as soon as possible, the activations it needs are freed early instead of piling up.
The result is that the number of in-flight micro-batches whose activations must be retained is capped at roughly the pipeline depth p, not the micro-batch count m. Peak activation memory becomes O(p) — independent of m. This is the quiet workhorse result of modern PP: since memory no longer grows with m, you are free to crank m as high as you like. 1F1B delivers the same (p-1)/(m+p-1) efficiency as GPipe while removing the memory ceiling that stopped you from reaching it — which is why 1F1B, not GPipe, is the default in production frameworks.
The activation-memory versus bubble tradeoff
Put the two forces side by side. Shrinking the bubble wants large m: efficiency is m/(m+p-1), monotonically increasing in m. Fitting in device memory wants small in-flight activation storage. The schedule you choose decides how these collide.
| Schedule | Bubble fraction | Peak activation memory |
|---|---|---|
| Naive (whole batch) | (p-1)/p | O(1) |
| GPipe | (p-1)/(m+p-1) | O(m) |
| 1F1B | (p-1)/(m+p-1) | O(p) |
| Interleaved 1F1B | (p-1)/(v·m) approx. | O(p)–O(vp) |
The table makes the design path clear. GPipe and 1F1B share a bubble, but 1F1B decouples memory from m, so it is strictly the better place to stand. From there, the only way to shrink the bubble further — without an impractically huge m or more stages — is to change the schedule geometry itself, which is what interleaving does.
Interleaved schedules: virtual stages
Interleaved 1F1B attacks the p-1 term in the numerator. Instead of giving each device one contiguous block of layers, give it v smaller, non-adjacent chunks — called virtual stages or model chunks. Device 0 might own layers 1–2 and 17–18. The pipeline now has v·p virtual stages but still only p physical devices.
Smaller chunks mean each micro-batch traverses a stage faster, so the fill and drain triangles shrink. The bubble fraction falls by roughly a factor of v, to approximately (p-1)/(v·m). The price is communication: each micro-batch now crosses device boundaries v times as often, so point-to-point traffic scales up by v. Interleaving therefore trades bandwidth for idle time — a good deal when the interconnect is fast (NVLink within a node) and a poor one when stages span slow links.
Choosing p, m, and the schedule in practice
The recipe falls straight out of the math. First, choose the smallest p that makes the model fit — every extra stage adds fill/drain cost and slow hand-offs, so PP is a memory tool, not a throughput tool. Second, with p fixed, choose m as large as the schedule’s memory budget and your global batch size allow, aiming for m at least several times p so the bubble drops into single digits.
Use 1F1B as the baseline so memory stays O(p) and does not fight your micro-batch count. Reach for interleaving only when the bubble is still costly and you have bandwidth to spare. The common failure mode is treating PP like data parallelism — adding stages expecting linear speedup — and instead paying a fat bubble for a model that, with a larger m or a smarter schedule, could have run near its compute roofline. The formulas tell you exactly where that line is.
(p-1)/(m+p-1) — it grows with the number of stages p and shrinks with the number of micro-batches m, which is why you want m to be several times p before PP earns its keep. GPipe and 1F1B share that bubble, but 1F1B keeps activation memory at O(p) instead of O(m), freeing you to raise m and actually reach a small bubble. Interleaving cuts the bubble by a further factor of v at the cost of v times the communication. The rule of thumb: split as little as memory forces, feed many micro-batches, default to 1F1B, and interleave only when bandwidth is cheap.