3D parallelism is how a model too big for a single accelerator, and too big for a single machine, gets trained anyway — by splitting it along three independent axes at once. Data parallelism (DP) replicates the model and feeds each copy different data; tensor parallelism (TP) slices each layer’s matrices across GPUs; pipeline parallelism (PP) assigns different layers to different GPUs. Used together they multiply, and the product is the whole cluster. The art is not any single technique — each has its own article in this series — but how you compose them: which axis goes on the fast links, which tolerates the slow ones, and how to size the three so communication and idle time stay small. This piece covers the factorization math, the placement heuristic every large-scale trainer converges on, and a worked mapping of a trillion-parameter model onto real hardware.
One GPU is never enough
Start with the wall. A trillion-parameter model, trained with the Adam optimizer in mixed precision, needs roughly 16 bytes per parameter: 2 for the fp16 weight, 2 for its gradient, and 12 for the fp32 master copy plus Adam’s two moments. That is 16 × 10^12 = 16 TB of state — before a single activation. An 80 GB GPU holds none of it. Even a 10B model’s ~160 GB overflows one card.
So you must partition. But partitioning is not free: every split introduces communication, and communication is bounded by whichever link the data crosses. The central fact that shapes everything below is that links are wildly unequal. Inside a node, NVLink moves data at ~900 GB/s; between nodes, even fast InfiniBand tops out near 50 GB/s per direction — an order of magnitude slower. 3D parallelism is, at heart, the discipline of matching each kind of communication to a link that can afford it.
Three axes, one recap
Each axis is a separate article; here is the one-line version of each, framed by what it costs to communicate. Tensor parallelism splits an individual matrix multiply — the attention projections and the FFN — across GPUs, so every layer’s forward and backward pass needs an all-reduce to stitch partial results back together. That is heavy, frequent, on the critical path.
Pipeline parallelism assigns contiguous layers to different GPUs (stages); the only thing crossing a boundary is the activation tensor for a microbatch — a small point-to-point send. Data parallelism replicates the entire model and synchronizes gradients with one all-reduce per step, which can overlap with the backward pass. Three axes, three utterly different communication profiles: TP chatty and latency-critical, PP cheap and occasional, DP bulky but hideable. That asymmetry is what the placement heuristic exploits.
The factorization: N = DP × TP × PP
The axes are orthogonal, so their sizes multiply. If you dedicate TP GPUs to slicing tensors, PP GPUs to holding pipeline stages, and replicate that whole arrangement DP times, the total GPU count is simply:
N = DP × TP × PP
TP × PP = the "model-parallel group": one full copy of the model
DP = how many copies run in parallelRead it two ways. Bottom-up: TP × PP is the number of GPUs it takes to hold one model, and it must be large enough that 16 TB / (TP × PP) fits in a GPU’s memory. Top-down: once one copy fits, DP = N / (TP × PP) spends the remaining GPUs on throughput. Every valid configuration is an integer factorization of N into three factors — and the whole design problem is choosing which factorization, because they are not equally fast.
Tensor parallelism belongs inside the node
TP’s all-reduce fires twice per transformer layer in the forward pass and twice again in the backward — for a 100-layer model that is hundreds of collective operations per step, each blocking the next matrix multiply. This traffic is enormous and sits squarely on the critical path, so it must ride the fastest link you own.
That link is NVLink, and it exists only within a node. The consequence is a near-universal rule: set TP equal to the number of GPUs in one node — typically TP = 8. All eight GPUs sharing a tensor split talk over 900 GB/s NVLink and never touch the slow inter-node fabric. Push TP to 16 and you straddle two nodes; the all-reduce now crosses InfiniBand and the layer stalls waiting on a link ~18× slower, wrecking utilization. TP is powerful but bandwidth-hungry, so you use exactly as much of it as one NVLink domain provides — and not one GPU more.
Pipeline parallelism spans the nodes
Once one node is saturated with tensor parallelism, the next unit of scale is the pipeline. PP cuts the layer stack into stages and places each stage on a different node. Because the only thing crossing a stage boundary is one microbatch’s activation tensor — a modest point-to-point send, not a collective — PP tolerates the slower inter-node link cheerfully. This is exactly the traffic profile you want on InfiniBand.
PP’s tax is different: the pipeline bubble. While the first microbatch works through stage 1, stages 2..p sit idle; the pipeline only fills after p - 1 steps and drains at the end. Feed it m microbatches and the wasted fraction is:
bubble fraction ≈ (p - 1) / mWith p = 8 stages and m = 8 microbatches you idle ~47%; raise m to 128 and the bubble shrinks to ~5%. More microbatches is the cure.
Data parallelism wraps the outside
TP fills a node, PP chains nodes into one model copy, and data parallelism spends everything left over on more copies. Each DP replica processes a different slice of the global batch, computes gradients locally, and the replicas average them with a single all-reduce per optimizer step.
DP earns the outermost position for two reasons. First, its communication happens once per step, not once per layer, and modern frameworks overlap that gradient all-reduce with the still-running backward pass — so much of its cost hides for free. Second, it is the axis that scales throughput most gracefully: doubling DP roughly doubles tokens per second with no change to the model layout. That is why you size the model-parallel group (TP × PP) as small as memory permits, then let DP absorb the rest of the cluster. The one limit is the global batch size: very large DP inflates the batch, and past a point that hurts convergence.
The placement heuristic and rank ordering
Put the three rules together and a single canonical mapping falls out, expressed as how you assign global GPU ranks. You order the axes from fastest-communication (inner) to slowest (outer):
innermost -> TP : contiguous ranks within a node (NVLink)
middle -> PP : consecutive nodes form a pipeline (InfiniBand)
outermost -> DP : replicas across pipeline groupsConcretely, ranks 0..7 are one TP group on node 0; ranks 8..15 are the next stage on node 1; after PP stages are laid down, the pattern repeats as the next DP replica. The principle is invariant: the axis with the heaviest, most frequent traffic gets the fastest wires. TP’s per-layer all-reduce sits on NVLink, PP’s occasional activation send sits on InfiniBand, and DP’s once-per-step gradient sync — the most latency-tolerant of all — can stretch across the whole cluster. Nearly every large-scale training framework encodes exactly this ordering.
Balancing the three to minimize bubble and communication
The factorization gives you freedom; the constraints tell you how to spend it. Work the priorities in order. TP is pinned to the NVLink domain size (usually 8) — that choice is essentially made for you. TP × PP must clear the memory bar: it has to be large enough that one model shard plus its activations fit per GPU. PP then wants to stay small, because bubble grows with the stage count — but it must be big enough, combined with TP, to satisfy memory.
The tension is real: memory pushes PP up, the bubble pushes it down. You resolve it by raising the microbatch count m until (p-1)/m is tolerable, and by using an interleaved 1F1B schedule, which assigns each GPU several non-contiguous virtual stages to shrink the bubble further (plain 1F1B only trims activation memory, not the bubble). Whatever survives goes to DP, the safe, overlappable axis — capped only by how large a global batch your optimizer can still learn from.
A worked example: a 1T model on 4096 GPUs
Map a 1-trillion-parameter model onto 4096 H100s (512 nodes of 8). Training state is ~16 TB. A GPU has 80 GB, of which ~60 GB is usable for model state after activations and framework overhead. So the model-parallel group must satisfy 16 TB / (TP × PP) ≤ 60 GB, i.e. TP × PP ≥ 267.
TP = 8 (one NVLink node)
PP = 32 -> TP x PP = 256 GPUs per model copy
-> state per GPU = 16 TB / 256 = 62.5 GB (fits, tight)
DP = N / (TP x PP) = 4096 / 256 = 16
check: DP x TP x PP = 16 x 8 x 32 = 4096 ✓PP’s 32 stages threaten a big bubble: at m = 32 microbatches you waste 31/32 ≈ 3%... only if m >> 32. Run m = 256 and the bubble is 31/256 ≈ 12% — the price of splitting one model across 32 nodes. This is why PP=32 is uncomfortable, and why the next section’s memory tricks matter: shrink TP × PP and the bubble eases too.
Beyond 3D: ZeRO and expert parallelism
Two extensions bend these constraints. ZeRO is not a fourth factor — it is sharded data parallelism. Instead of every DP replica holding a full copy of the optimizer state, ZeRO partitions that state (and optionally gradients and parameters) across the DP group, so N = DP × TP × PP still holds. Its payoff is memory: shard the 12 bytes of Adam state across DP = 16 and per-GPU optimizer memory drops ~16×, which can let you lower TP × PP — fewer pipeline stages, a smaller bubble.
Expert parallelism (EP) is a genuine new axis, for mixture-of-experts models: different feed-forward experts live on different GPUs, and tokens are routed to them, giving N = DP × TP × PP × EP — true 4D parallelism. The composition principle is unchanged: EP’s all-to-all token routing is placed on whichever link its traffic can afford, and the total is still just the product of the axes.