Ring Attention answers a brutally simple question: how do you run exact attention over a sequence so long its activations do not fit on one accelerator? The trick is to treat attention not as one big matrix multiply but as a stream. Split the sequence across P devices, give each a block of queries, keys, and values, then pass the K/V blocks around a ring while every device folds each visiting block into a running online-softmax accumulator. No device ever holds the whole sequence, none materializes the full attention matrix, and the network traffic hides behind the arithmetic. The result is mathematically identical to dense attention, yet the memory per device stays flat as context grows. This piece builds the idea from the softmax identity up, works a numeric example, and draws the line between the ring and single-device FlashAttention tiling.
Why one device runs out of room
Attention over a length-N sequence forms scores S = QK^T / √d_k of shape [N, N]. FlashAttention already taught us not to store that matrix: it tiles the computation and streams blocks through fast on-chip memory, so attention costs O(N) memory instead of O(N^2). But tiling does nothing about the other linear-in-N cost — the activations Q, K, V and the residual stream are each [N, d], and for a million-token context those tensors alone dwarf a single device’s memory before you compute a thing.
So the constraint that forces a ring is not the N^2 matrix (FlashAttention handles that) but the N·d activations. A sequence that cannot live on one device must be split across devices — and then every query still needs keys living on every other device. Ring Attention is the communication pattern that meets that need without ever gathering the full sequence anywhere.
Sharding the sequence: what each device holds
Cut the sequence into P contiguous blocks of size b = N/P, one per device. Device i owns block i: its Q_i, K_i, V_i, each of shape [b, d] — sequence parallelism, a split along the token axis.
The asymmetry that makes the ring work: queries stay home. Device i only ever computes outputs for its own query rows, so Q_i never moves. The keys and values must travel, because each query needs q·k for every key. Rather than broadcast all K/V at once — reassembling the full sequence in memory and defeating the purpose — the ring moves one block at a time, so a device ever holds only its own block plus one visiting block: two, not P.
The online-softmax identity that makes streaming legal
Attention’s output for one query is Σ_j softmax(s_j)·v_j = (Σ_j exp(s_j - m)·v_j) / (Σ_j exp(s_j - m)), where m is any constant — taken as max_j s_j for numerical safety, as the subtraction cancels in the ratio. That invariance is why attention can be computed incrementally: if a later block reveals a bigger maximum, you can retroactively fix up everything computed so far.
Keep three running quantities per query — the max m, the denominator l = Σ exp(s - m), and the unnormalized output o = Σ exp(s - m)·v. When a block arrives, raise m if its local max is bigger and rescale the old l and o by α = exp(m_old - m_new) before adding the block — correcting the earlier terms as if they had used the new max all along. This is exactly the FlashAttention accumulator; the ring just feeds it blocks arriving over the network instead of from HBM.
Per query q, keep running state (m, l, o):
m = running max of scores
l = running Σ exp(s - m) # denominator
o = running Σ exp(s - m)·v # vector, length d
For each incoming K/V block:
s_j = q · k_j / √d_k # this block’s scores
m' = max(m, max_j s_j) # new running max
α = exp(m - m') # rescale old state
l' = α·l + Σ_j exp(s_j - m')
o' = α·o + Σ_j exp(s_j - m')·v_j
(m, l, o) = (m', l', o')
After the final block: out = o / l
The ring schedule: passing K/V around
Arrange the devices in a logical ring: 0 → 1 → … → P-1 → 0. The computation runs P steps. On step t, device i holds the K/V block from device (i - t) mod P; it folds that block’s partial attention into its accumulator and — at the same time — passes the block to (i + 1) mod P while receiving the next from (i - 1) mod P.
After P steps each device has seen all P K/V blocks once, so Q_i has attended to the entire sequence. Every device sends and receives exactly one block per step, so traffic is uniform — no hotspots, no all-gather storm — which lets the ring scale to hundreds of devices.
Accumulation math, step by step
The schedule and the accumulator combine into the per-device inner loop, run simultaneously on every device’s own Q_i.
state = (m = -∞, l = 0, o = 0) # per query row in Q_i
kv = (K_i, V_i) # start with your own block
for t in 0 .. P-1:
if t < P-1: async send(kv → i+1), recv(next ← i-1)
S = Q_i · kv.K^T / √d_k # [b, b] scores
state = online_softmax(state, S, kv.V) # rescale + add
if t < P-1: wait(send, recv); kv = next
O_i = state.o / state.l # [b, d] final outputs
The send/recv is launched before the matmul and waited on after, so the next block’s transfer overlaps the current block’s compute. The online_softmax call is the rescaling recurrence above. Nothing here materializes an N×N matrix — the largest tensor any device touches is [b, b].
A worked numeric example
To see the rescale fire, follow one query whose four keys arrive as two blocks, arranged so the second block holds the larger score and forces a real correction. Values are scalars (d = 1) to keep the arithmetic legible; the vector case is identical component-wise.
One query, 4 keys arriving as 2 blocks (/√d_k folded into s):
Block A: s = [1, 2], v = [1, 2]
Block B: s = [3, 0], v = [3, 4]
Step A (m = -∞, l = 0, o = 0)
m' = 2, α = 0
exp(s - 2) = [0.368, 1.000]
l = 1.368 o = 0.368·1 + 1.000·2 = 2.368
Step B (bigger max → real rescale)
m' = 3, α = exp(2 - 3) = 0.368
exp(s - 3) = [1.000, 0.050]
l = 0.368·1.368 + (1.000 + 0.050) = 1.553
o = 0.368·2.368 + (1.000·3 + 0.050·4) = 4.070
out = o / l = 4.070 / 1.553 = 2.621
Direct softmax over [1, 2, 3, 0] = 2.621 ✓
Step A commits a provisional answer believing the max is 2; step B finds a 3, lifts the max, and multiplies the stored l and o by α = 0.368 before adding its terms. That one scalar multiply is the entire cost of processing blocks out of order, yet the result lands exactly on the dense-softmax answer — order-independence is what makes the ring exact, not an approximation.
Memory: O(1) extra per device
Tally what a device stores: its own shard Q_i, K_i, V_i (3·b·d), at most two K/V blocks at once (~2·b·d), plus an accumulator of b·d. Every term is proportional to the block size b = N/P, and none grows with the total context N once you hold b fixed and add devices.
That is the headline result: the extra memory to reach across the whole sequence, beyond a device’s own shard, is one visiting block — O(1) in block count, O(b) in elements, and constant in N. Double the context and the device count together and per-device memory is unchanged, so the ceiling is aggregate pod memory, not any single chip — which is why the ring enables near-arbitrary context.
Communication vs compute: why the transfer hides
The ring only pays off if moving a block is cheaper than computing on it. Per step a device transfers one K/V block — 2·b·d elements, so communication is O(b·d). The compute is the score and value matmuls, O(b^2·d) arithmetic. The compute-to-communication ratio is therefore O(b^2 d) / O(b d) = O(b).
That linear-in-block-size ratio is the design lever. Keep the block b large — N/P in the thousands of tokens — and the per-step matmul outlasts the per-step transfer, so the network cost vanishes behind compute. Below a hardware-dependent threshold (FLOP throughput over link bandwidth) the ring turns communication-bound and stalls — the single most important tuning knob in practice.
Ring Attention vs FlashAttention
They share one engine — the online-softmax accumulator — which is why they are so easily conflated, but they solve different problems. FlashAttention is a single-device memory optimization: it tiles QK^T so the N×N scores never leave fast SRAM, trading a full matrix for a stream of tiles read from that same GPU’s HBM — the whole sequence still on one device.
| FlashAttention | Ring Attention | |
|---|---|---|
| Scope | One device | P devices in a ring |
| What streams | Tiles from HBM | K/V blocks over the network |
| Solves | N×N matrix too big | N×d activations too big |
| Per-unit memory | O(N) on the GPU | O(N/P) per device |
| Bottleneck to hide | HBM bandwidth | Inter-device bandwidth |
They compose: each device in a ring typically runs FlashAttention locally on each [b, b] tile. Ring Attention is that same streaming idea lifted from the memory hierarchy of one chip to the interconnect of a whole pod — same math, one level up.
Causal masking, load balance, and other wrinkles
The clean uniform picture cracks under a causal mask. With autoregressive attention, query block i attends only to key blocks j ≤ i; the upper triangle is wasted work. In the naive ring the device holding the last query block does far more useful work than the one holding the first — left unaddressed, half the pod sits idle. Production variants (striped or zig-zag ring attention) reshuffle which tokens land on which device so every device carries a roughly equal count of unmasked block-pairs.
Two more notes. The ring is exact: unlike sparse or linear attention it computes true dense attention, changing cost structure, not model quality. And the accumulator must carry un-normalized o and running l separately until the end; normalizing per block and averaging is wrong, because early blocks used a denominator later blocks revise.