Sequence parallelism (SP) is a memory optimization, not a compute one. Tensor parallelism already splits the heavy matmuls of attention and the MLP across devices, but it leaves a stubborn residue — the LayerNorm and dropout regions between those blocks — fully replicated on every rank, and their activations dominate memory once you train at scale. Megatron-style sequence parallelism removes that residue by partitioning those regions along the sequence dimension, so each device holds only s/t of the tokens. The elegant part: this is nearly free. The single all-reduce that tensor parallelism uses at each block boundary decomposes exactly into an all-gather plus a reduce-scatter — the same total bytes on the wire. This piece works through the activation-memory math, the collective substitution, and ring attention for very long sequences.
The one-line idea
A transformer layer alternates between two kinds of region. The tensor-parallel regions — the QKV projection, the attention core, the output projection, and the two MLP linears — hold the big matmuls, and tensor parallelism (TP) shards them across t devices by splitting weight matrices. Between them sit the non-matmul regions: the LayerNorms and residual dropouts — cheap in FLOPs but still producing activations of shape [s, b, h] that must be stored for the backward pass.
In plain TP those non-matmul regions run identically on every rank — each device redundantly computes the same LayerNorm over the same full tensor and keeps the same full activation. Sequence parallelism’s one idea is to stop replicating them: split those tensors along the sequence axis s, so rank i owns only tokens [i·s/t : (i+1)·s/t]. The work and memory of the LayerNorm/dropout band then drop by a factor of t, matching the reduction TP already achieved inside the matmuls.
The problem: activation memory, not weight memory
When people size a model they think of weights, but during training the activations — every intermediate tensor kept for backprop — often dominate. For one transformer layer, with sequence length s, batch b, hidden size h, and a attention heads, the stored activation count (following Korthikanti et al., 2022) is:
A_layer = s·b·h · (34 + 5·a·s/h) [elements]The 34·sbh term is the pile of linear-layer and normalization activations; the 5·a·s·b·s term (rewritten as 5as/h · sbh) is the quadratic attention scores and their dropout. Multiply by layer count and by 2 bytes for fp16 and the number is enormous — which is why gradient checkpointing and model parallelism exist. Sequence parallelism attacks a specific, otherwise-un-parallelized slice of this total, easiest to see by first asking what tensor parallelism leaves behind.
What tensor parallelism leaves replicated
Tensor parallelism divides the matmul-region activations by t, but it cannot touch the LayerNorm and dropout bands, because those operate on the full hidden vector and TP has no weight to split there. The per-layer activation memory with TP alone is:
A_TP = sbh · (10 + 24/t + 5as/(h·t))Read the three terms. The 24/t and 5as/(ht) pieces are the matmul and attention activations, dutifully shrinking as you add devices. But the 10·sbh term does not depend on t at all — it is the two LayerNorm inputs, the two dropout masks, and the block-input tensors, replicated on every rank. Push t to 8 or 16 and the parallelizable terms melt away while that constant 10·sbh just sits there, eventually the majority of your activation memory. That is the target.
The move: split the LayerNorm/dropout band along the sequence
Here is the observation that makes SP work: LayerNorm normalizes each token independently across the hidden dimension, and residual dropout is applied elementwise. Neither operation mixes information between sequence positions. So there is no correctness reason to have every position on every device — hand each rank a disjoint slice of tokens and it computes exactly the same result for its slice.
Sequence parallelism therefore shards these regions as [s/t, b, h]: rank i stores and processes only its s/t tokens through the LayerNorm and dropout. The catch is the boundary. The tensor-parallel attention and MLP that follow need the full sequence assembled the way TP expects it (activations split across the hidden/head dimension, not the sequence). So SP has to reshard the tensor at every transition — sequence-split to hidden-split going into a matmul region, and back coming out. Done naively that would add communication; it does not, and that is the crux of the design.
The collective substitution: all-reduce becomes all-gather + reduce-scatter
In pure tensor parallelism, each block boundary uses a pair of conjugate operators. The Megatron notation calls them g and g̅: g̅ is an all-reduce in the forward pass (summing each rank’s partial output into the full activation) and identity in backward; g is identity forward and all-reduce backward. Every layer runs two forward all-reduces — one closing attention, one closing the MLP.
Sequence parallelism replaces each operator with a sequence-aware collective. Entering a matmul region, g becomes an all-gather along the sequence (assembling the full s from the s/t shards). Leaving it, g̅ becomes a reduce-scatter (summing the partial results and scattering them back into sequence shards in one step). The backward pass swaps the two, as their conjugates. Crucially, no operation was added — each single all-reduce turned into exactly one all-gather plus one reduce-scatter.
Why the byte count is identical
The substitution is free because of how a ring all-reduce is already implemented under the hood: a reduce-scatter followed by an all-gather. Each phase moves (t−1)/t of the tensor per device, so an all-reduce of V elements costs 2(t−1)/t · V per device. Sequence parallelism simply uses the two halves separately — one all-gather at entry, one reduce-scatter at exit — for the same total.
Worked example. Take t = 8 and an activation tensor of V elements crossing one boundary. A ring all-reduce moves 2 × (7/8) × V = 1.75V per device. Under SP, the all-gather moves (7/8)V = 0.875V and the reduce-scatter another 0.875V — total 1.75V. Per layer, TP does two forward all-reduces (3.5V); SP does two all-gathers plus two reduce-scatters (3.5V). Same wire volume, strictly less memory. That is why SP is described as a ‘free’ complement to tensor parallelism rather than a trade-off.
The activation-memory saving, in one formula
Because SP shards the previously-replicated band by t as well, the per-layer activation memory collapses to a single clean expression — every term now divided by t:
A_TP = sbh · (10 + 24/t + 5as/(h·t))
A_TP+SP = (sbh / t) · (34 + 5as/h)The second line is just the full single-device formula sbh(34 + 5as/h) divided cleanly by t — the ideal 1/t scaling that pure TP failed to reach because of its 10·sbh floor. The gap between the two lines is entirely that floor: TP keeps 10·sbh, SP keeps 10·sbh/t. As t grows the saving approaches a full 10·sbh per layer — often the difference between fitting a long-context training step in device memory and recomputing activations from scratch.
A worked numeric example
Take a GPT-3-scale layer: s = 2048, b = 4, h = 12288, a = 96, with tensor/sequence degree t = 8. First 5as/h = 5 × 96 × 2048 / 12288 = 80, and sbh ≈ 1.007 × 10^8 elements.
TP only: 10 + 24/8 + 80/8 = 10 + 3 + 10 = 23 → 23 · sbh
TP + SP: (34 + 80) / 8 = 114 / 8 = 14.25 → 14.25 · sbhIn fp16 (2 bytes/element), per device per layer: TP holds 23 × 1.007e8 × 2 ≈ 4.63 GB; TP+SP holds 14.25 × 1.007e8 × 2 ≈ 2.87 GB. That is a 38% reduction in per-layer activation memory — about 1.76 GB saved per layer per device, on the order of 170 GB across a 96-layer stack. All of it comes from dividing that one replicated 10·sbh term, at zero extra communication.
Where SP stops: the attention core and long sequences
Be precise about what SP does not fix. Notice the attention term 5as/(h·t) is divided by t in both the TP and TP+SP formulas — that quadratic-in-sequence cost is sharded by tensor parallelism splitting heads, not by SP. Sequence parallelism all-gathers back to the full sequence length before the attention region, so each device’s attention still runs over all s positions for the heads it owns. The per-head O(s^2) score matrix is untouched by adding SP.
That is fine until s becomes very large — tens or hundreds of thousands of tokens — at which point the attention scores alone will not fit, no matter how many heads you shard. The sequence dimension has to be split through the attention itself. That is a different axis of parallelism, usually called context parallelism, and it is where ring attention comes in.
Context parallelism and ring attention
Context parallelism (CP) shards Q, K, and V along the sequence: each device owns a contiguous block of query tokens and the matching block of keys/values. The obstacle is that attention is all-to-all — every query must see every key — so a device holding only its own K/V block cannot compute full attention alone. Ring attention solves this by arranging the devices in a ring and passing K/V blocks around it step by step: at each step a device attends its local queries against the K/V block in hand, accumulates the partial (online-softmax) result, and forwards that block to its neighbor. After t steps every query has attended every key.
The win is twofold: no device ever materializes the full s × s score matrix (only s/t × s/t tiles), and the K/V transfer overlaps with compute, hiding most of the communication. This is how context windows of hundreds of thousands of tokens become trainable — SP for the LayerNorm/dropout band, CP for the attention core.
Practical notes and pitfalls
A few things to keep straight. First, SP is not standalone — it lives on top of tensor parallelism and reuses the exact same TP process group and degree t; you cannot have SP without TP under it. Second, do not conflate Megatron sequence parallelism with context parallelism: SP partitions the non-attention regions and gathers back to full sequence for the matmuls, while CP partitions the attention itself — they solve different memory problems and compose. Third, the swap only stays free if your interconnect handles all-gather and reduce-scatter as efficiently as a fused all-reduce; on weak fabrics the two-phase pattern can expose latency the fused primitive hid.
For a single-CPU small-model setting, SP is moot (t = 1 collapses every formula to the base case), but the accounting is exactly why activation memory, not weights, sets your maximum batch and sequence length. The single-device analog is gradient checkpointing — trade recompute for memory — and the same 34 + 5as/h arithmetic tells you what it buys.