Sequence parallelism gets explained as a derivation — shard the LayerNorm/dropout band along the token axis, watch one all-reduce split into an all-gather plus a reduce-scatter, admire that the byte count is unchanged. That derivation is settled, and a companion piece walks it. What is far less settled is the question a practitioner actually faces: you are over your per-device memory budget, and SP is one of three levers you could pull. This piece treats SP as a knob — how much it can move, what constrains it, why you cannot turn it independently, and how it stacks up against the two obvious alternatives of using a smaller micro-batch or recomputing activations from scratch.
Three levers on one budget
Training memory splits into two piles. The static pile — weights, gradients, optimizer state — is fixed by the model and is what ZeRO/FSDP sharding attacks. The dynamic pile is activations: every intermediate tensor autograd must keep until the backward pass reaches it. The dynamic pile is the one that scales with your batch and your context length, and it is almost always what actually blows up.
Without changing the model, there are exactly three ways to shrink it. You can make less of it (smaller micro-batch), throw it away and rebuild it (activation recomputation), or spread it across devices you already own (sequence parallelism). These are not equivalent. Two of them cost you throughput; one is close to free. Knowing which is which, and how far the free one reaches before it runs out, is the whole practical skill — and it is a different skill from being able to derive the formula.
What the knob buys, stated once
The activation count for one transformer layer, with sequence length s, micro-batch b, hidden size h, and a heads, follows Korthikanti et al. (2022). Written for a single device, for tensor parallelism at degree t, and for TP+SP:
A_1 = sbh · (34 + 5as/h) one device
A_TP = sbh · (10 + 24/t + 5as/(h·t)) TP alone, degree t
A_TP+SP = sbh · (34 + 5as/h) / t TP + sequence parallel
saving = 10·sbh · (1 − 1/t) per layer, per deviceRead the last line as the knob’s entire range. SP does not scale with t the way you might hope: it removes the replicated 10·sbh floor and nothing else, and even at t = ∞ that is all it can ever remove. The interesting question is therefore not ‘how much does SP save’ in the abstract, but what fraction of your particular total is that floor — a number that swings from trivial to dominant depending on the rest of your configuration.
Free on the wire, not free in sync points
The reason SP is described as free is that a ring all-reduce is already implemented as a reduce-scatter followed by an all-gather. SP just uses the two halves separately — all-gather entering a matmul region, reduce-scatter leaving it — so both the bytes moved, 2(t−1)/t · V per device, and the 2(t−1) ring steps are unchanged. Bandwidth-neutral and, in the ring model, latency-neutral too.
What does change is the count of collective calls. A TP layer issues two collectives; a TP+SP layer issues four. Same total traffic, twice as many synchronization points. In practice that shows up as extra kernel-launch overhead and slightly more exposure to stragglers and network jitter, because every collective is a barrier where the fastest rank waits for the slowest. It is a small tax, typically a low single-digit percentage, and it is why SP is ‘nearly free’ rather than free — but it is real, and it grows on noisy or oversubscribed fabrics.
The knob you cannot turn on its own
Here is the constraint that surprises people configuring a run: in Megatron-style SP there is no independent SP degree. The whole mechanism is a reshard between two views of the same t ranks — sequence-split in the norm band, hidden-split in the matmul band. SP degree is TP degree, always. In the config it appears as a boolean, not a number.
The consequence is that you cannot buy more activation savings by raising SP alone. Wanting a bigger divisor means raising t, which drags along all of tensor parallelism’s costs: more all-reduce traffic per layer, and the hard requirement that the TP group stay inside one NVLink domain. Two smaller constraints follow. s must be divisible by t, so odd context lengths need padding that you must then mask out of the loss. And after the reshard each rank feeds b · s/t tokens into its GEMMs — drive that too low and the matmuls go skinny and lose efficiency.
Context parallelism is the axis that is independent
SP is bounded because it only touches the token-independent band: LayerNorm normalizes each token across the hidden dimension and residual dropout is elementwise, so neither mixes positions and both shard for free. Attention is the opposite — every query must see every key — so SP leaves the attention core alone.
Context parallelism (ring attention) is the separate knob that does shard attention, by rotating K/V blocks around a ring so no device ever holds the full sequence; the mechanism is covered separately. The configuration point is that CP is a real mesh dimension with its own degree: N = DP × CP × TP × PP. So the two sit in a clean relationship. SP is free but bounded — it costs nothing and caps out at 10·sbh. CP is unbounded but not free — it scales to any context but adds a per-layer K/V ring and a causal-mask load-balance problem. Reach for SP first; reach for CP when the attention core, not the norm band, is what overflows.
SP versus simply using a smaller micro-batch
Every activation term is linear in b, so halving the micro-batch halves all of them — including the quadratic attention term that SP cannot reach. On pure memory-per-dollar, shrinking b is the stronger lever. So why is it the second choice?
Because it is the one that costs throughput. Holding the global batch fixed (it is set by convergence, not by your memory), a smaller micro-batch means proportionally more gradient-accumulation steps. Each step re-pays the fixed per-step overheads, and more importantly the GEMMs get narrower: arithmetic intensity falls, the matmuls drift from compute-bound toward memory-bound, and utilization sags well before you reach b = 1. SP has no equivalent penalty — it moves bytes you were already moving and leaves every GEMM the same shape. The rule that falls out: exhaust the free lever before you pay for the expensive one. Turn SP on, then shrink b only for the memory SP could not reach.
A worked configuration: 7B-class, 8K context
Take h = 4096, a = 32, 32 layers, s = 8192, b = 2, t = 8, fp16. Then 5as/h = 5 · 32 · 8192 / 4096 = 320 and sbh = 6.71 × 10^7 elements, so one unit of sbh is 0.134 GB per layer, or 4.30 GB across the 32-layer stack. Now count units four ways:
| Config | Units of sbh | Stack total |
|---|---|---|
| TP only | 10 + 3 + 40 = 53 | 228 GB |
| TP + SP | (34 + 320)/8 = 44.25 | 190 GB |
| TP + FlashAttention | 10 + 3 = 13 | 55.8 GB |
| TP + SP + FlashAttention | 34/8 = 4.25 | 18.3 GB |
The first pair is the disappointing case: SP buys only 16%, because at 8K context the quadratic 320·sbh term swamps the 10·sbh floor SP removes. Neither config fits an 80 GB device.
Why FlashAttention makes SP more valuable, not less
The second pair in that table is the one worth internalizing. FlashAttention (or selective recomputation, which achieves the same on this term) never materializes the attention score matrix, deleting the 5as/h term outright. That drops TP-only from 53 units to 13 — and of those 13, the replicated floor is 10, or 77% of what remains. Turning SP on now cuts 13 units to 4.25: a 3.1× reduction, 55.8 GB down to 18.3 GB.
This inverts the intuition that a memory optimization becomes redundant once you have another one. The two are complements: FlashAttention removes the term SP cannot touch, which promotes SP’s fixed saving from a rounding error to the dominant effect. The practical difference here is not cosmetic. With TP+SP+FA, activations plus a shard of weights and Adam state (roughly 14 GB at t = 8) land near 32 GB, leaving real headroom to raise the batch or the context. Without SP, the same run sits around 70 GB and fragments its way into OOM.
Where SP does nothing, and what it means on CPU
Four situations where the knob is simply inert, worth checking before you reach for it. Inference: no backward pass means no stored activations, and SP does not shard the KV cache — the actual serving constraint. No tensor parallelism: SP degree equals TP degree, so at t = 1 the saving is 10sbh(1 − 1) = 0. Full recomputation: if you already discard and rebuild every layer, the band SP shards is not being stored anyway — though SP composes beautifully with selective recompute, which is exactly the FlashAttention row above. Static memory: SP touches activations only; weights, gradients, and optimizer state are ZeRO’s problem.
For CPU and small-model work the direct answer is that t = 1 makes SP a no-op. The transferable idea survives, though: look for state that is replicated across workers rather than merely large, and remember that the token axis is the cheapest thing to split, because normalization and dropout never look sideways.