3D Parallelism — DP + TP + PP

One GPU is never enough

Start with the wall. A trillion-parameter model, trained with the Adam optimizer in mixed precision, needs roughly 16 bytes per parameter: 2 for the fp16 weight, 2 for its gradient, and 12 for the fp32 master copy plus Adam’s two moments. That is 16 × 10^12 = 16 TB of state — before a single activation. An 80 GB GPU holds none of it. Even a 10B model’s ~160 GB overflows one card.

So you must partition. But partitioning is not free: every split introduces communication, and communication is bounded by whichever link the data crosses. The central fact that shapes everything below is that links are wildly unequal. Inside a node, NVLink moves data at ~900 GB/s; between nodes, even fast InfiniBand tops out near 50 GB/s per direction — an order of magnitude slower. 3D parallelism is, at heart, the discipline of matching each kind of communication to a link that can afford it.

Three axes, one recap

Each axis is a separate article; here is the one-line version of each, framed by what it costs to communicate. Tensor parallelism splits an individual matrix multiply — the attention projections and the FFN — across GPUs, so every layer’s forward and backward pass needs an all-reduce to stitch partial results back together. That is heavy, frequent, on the critical path.

Pipeline parallelism assigns contiguous layers to different GPUs (stages); the only thing crossing a boundary is the activation tensor for a microbatch — a small point-to-point send. Data parallelism replicates the entire model and synchronizes gradients with one all-reduce per step, which can overlap with the backward pass. Three axes, three utterly different communication profiles: TP chatty and latency-critical, PP cheap and occasional, DP bulky but hideable. That asymmetry is what the placement heuristic exploits.

The factorization: N = DP × TP × PP

The axes are orthogonal, so their sizes multiply. If you dedicate TP GPUs to slicing tensors, PP GPUs to holding pipeline stages, and replicate that whole arrangement DP times, the total GPU count is simply:

N = DP × TP × PP

  TP × PP  = the "model-parallel group": one full copy of the model
  DP          = how many copies run in parallel

Read it two ways. Bottom-up: TP × PP is the number of GPUs it takes to hold one model, and it must be large enough that 16 TB / (TP × PP) fits in a GPU’s memory. Top-down: once one copy fits, DP = N / (TP × PP) spends the remaining GPUs on throughput. Every valid configuration is an integer factorization of N into three factors — and the whole design problem is choosing which factorization, because they are not equally fast.

Tensor parallelism belongs inside the node

TP’s all-reduce fires twice per transformer layer in the forward pass and twice again in the backward — for a 100-layer model that is hundreds of collective operations per step, each blocking the next matrix multiply. This traffic is enormous and sits squarely on the critical path, so it must ride the fastest link you own.

That link is NVLink, and it exists only within a node. The consequence is a near-universal rule: set TP equal to the number of GPUs in one node — typically TP = 8. All eight GPUs sharing a tensor split talk over 900 GB/s NVLink and never touch the slow inter-node fabric. Push TP to 16 and you straddle two nodes; the all-reduce now crosses InfiniBand and the layer stalls waiting on a link ~18× slower, wrecking utilization. TP is powerful but bandwidth-hungry, so you use exactly as much of it as one NVLink domain provides — and not one GPU more.

Pipeline parallelism spans the nodes

Once one node is saturated with tensor parallelism, the next unit of scale is the pipeline. PP cuts the layer stack into stages and places each stage on a different node. Because the only thing crossing a stage boundary is one microbatch’s activation tensor — a modest point-to-point send, not a collective — PP tolerates the slower inter-node link cheerfully. This is exactly the traffic profile you want on InfiniBand.

PP’s tax is different: the pipeline bubble. While the first microbatch works through stage 1, stages 2..p sit idle; the pipeline only fills after p - 1 steps and drains at the end. Feed it m microbatches and the wasted fraction is:

bubble fraction ≈ (p - 1) / m

With p = 8 stages and m = 8 microbatches you idle ~47%; raise m to 128 and the bubble shrinks to ~5%. More microbatches is the cure.

Data parallelism wraps the outside

TP fills a node, PP chains nodes into one model copy, and data parallelism spends everything left over on more copies. Each DP replica processes a different slice of the global batch, computes gradients locally, and the replicas average them with a single all-reduce per optimizer step.

DP earns the outermost position for two reasons. First, its communication happens once per step, not once per layer, and modern frameworks overlap that gradient all-reduce with the still-running backward pass — so much of its cost hides for free. Second, it is the axis that scales throughput most gracefully: doubling DP roughly doubles tokens per second with no change to the model layout. That is why you size the model-parallel group (TP × PP) as small as memory permits, then let DP absorb the rest of the cluster. The one limit is the global batch size: very large DP inflates the batch, and past a point that hurts convergence.

The placement heuristic and rank ordering

Put the three rules together and a single canonical mapping falls out, expressed as how you assign global GPU ranks. You order the axes from fastest-communication (inner) to slowest (outer):

innermost  ->  TP   : contiguous ranks within a node (NVLink)
middle     ->  PP   : consecutive nodes form a pipeline (InfiniBand)
outermost  ->  DP   : replicas across pipeline groups

Concretely, ranks 0..7 are one TP group on node 0; ranks 8..15 are the next stage on node 1; after PP stages are laid down, the pattern repeats as the next DP replica. The principle is invariant: the axis with the heaviest, most frequent traffic gets the fastest wires. TP’s per-layer all-reduce sits on NVLink, PP’s occasional activation send sits on InfiniBand, and DP’s once-per-step gradient sync — the most latency-tolerant of all — can stretch across the whole cluster. Nearly every large-scale training framework encodes exactly this ordering.