Retention: fading memory instead of softmax
Attention computes softmax(QK^T / √d_k) V, and its per-token decode cost grows with context because the KV cache grows. The softmax is the obstacle: exp does not decompose into anything you can accumulate incrementally, so you must keep every past key.
Retention removes the softmax and replaces learned, content-dependent weights with a fixed weight by distance: a token m steps back is scaled by γ^m for a constant 0 < γ < 1. The intuition is a leaky bucket — each step the memory you hold is multiplied by γ (it fades), and the new token is poured in on top. The critical algebraic property is that this decay factorizes — γ^(n-m) = γ^n / γ^m — so the past folds into a running accumulator instead of being recomputed. Relative position is carried in two channels: a phase, via an xPos/rotary-style rotation on Q and K, and a magnitude, via the decay.
The recurrent form: one state matrix, forever
Written as a recurrence, retention keeps a single state matrix S_n that stands in for the entire history:
S_n = γ · S_(n-1) + K_n^T V_n S_n : [d_k, d_v], S_0 = 0
o_n = Q_n · S_n o_n : [1, d_v]
Shapes per step: Q_n, K_n, V_n : [1, d]
K_n^T V_n : [d_k, d_v] rank-1 outer product
Q_n S_n : [1,d_k]×[d_k,d_v] → [1, d_v]Read it left to right: fade what you had, add an outer product for the new token, read the answer out by projecting the query through the state. And — the whole point — S_n is the same d_k × d_v shape at token 10 and at token 100,000, so decoding is O(1) time and O(1) memory per token: no cache to grow, nothing to stream that gets bigger, no eviction policy to tune. The recurrence also makes the model’s bias explicit — information is written into a lossy summary and continuously attenuated, never stored verbatim.
The parallel form: a decay mask in place of softmax
A sequential recurrence is fine for generation and terrible for training, where every position should be computed at once. Retention therefore has a second, fully parallel form: attention with the softmax swapped for an elementwise decay mask D.
Retention(X) = (Q K^T ⊙ D) V
D_(nm) = γ^(n-m) if n ≥ m (causal + geometric decay)
= 0 if n < m (no peeking ahead)
Q, K, V = X W_Q, X W_K, X W_V X : [N, d] → Q,K,V : [N, d]
QK^T : [N, N] ⊙ D → × V → [N, d_v]D is one lower-triangular matrix doing two jobs: zeros above the diagonal enforce causality, and the γ^(n-m) entries below apply the fade. Notice there is no row normalization — nothing forces a row of weights to sum to 1, which is why the block adds a GroupNorm afterwards to keep activation scale sane. Computationally this is the same dense-matmul shape attention kernels already target, so training throughput is Transformer-class.
Why the two forms give identical numbers
These are not approximations of each other. Unroll the recurrence from S_0 = 0 and the parallel form falls out:
S_n = Σ_(m=1..n) γ^(n-m) K_m^T V_m
o_n = Q_n S_n
= Σ_(m=1..n) γ^(n-m) (Q_n K_m^T) V_m
= Σ_(m=1..n) (Q_n K_m^T · D_(nm)) V_m ⇐ row n of (QK^T ⊙ D)VWhat makes it work is associativity plus the factorizable decay: because γ^(n-m) splits into a term depending only on n and one depending only on m, the double sum collapses into a single accumulator faded once per step. Softmax cannot do this, which is exactly why attention has no cheap recurrent twin. The practical payoff: you train with the parallel form and serve with the recurrent form on the very same weights — no distillation, no second model, no train/serve mismatch to debug.
Chunkwise-recurrent: the form you actually train long context with
The parallel form is O(N^2); the recurrent form is linear but sequential, leaving vector units idle. The chunkwise-recurrent form splits the difference: cut the sequence into chunks of size B, run the parallel form inside a chunk, carry a recurrent state between chunks.
For chunk i, inner positions j = 1..B:
inner (parallel) I_i = (Q_i K_i^T ⊙ D) V_i
cross (carry-in) C_i = (Q_i R_(i-1)) ⊙ ξ, ξ_j = γ^j
state (carry-out) R_i = γ^B R_(i-1) + K_i^T (V_i ⊙ ζ), ζ_j = γ^(B-j)
Retention(X_i) = I_i + C_i cost ≈ O(N B d + N d^2)Each token’s output sums what it retains from its own chunk and what it inherits from the compressed past. Cost is quadratic only in B, which you choose, so the total is linear in N while still being dominated by large matmuls. This is the form for prefill: a long prompt is ingested chunk by chunk at high arithmetic intensity, and what survives is one small state handed to the recurrent decoder.
Reading the decay factor γ as a memory horizon
γ is the personality of a retention head. Its weight on a token k steps back is γ^k, so it defines an effective receptive field. The weights sum to Σ_k γ^k = 1/(1-γ), which doubles as the rule of thumb for the horizon — how many recent tokens carry essentially all the mass. Worked out:
k : 1 10 50 200 1000 horizon 1/(1-γ)
0.90^k : 0.900 0.349 0.005 ~0 ~0 10
0.99^k : 0.990 0.904 0.605 0.134 0.00004 100
0.999^k : 0.999 0.990 0.951 0.819 0.368 1000The table shows the trap in a single fixed γ. At 0.9 the head is functionally a 10-token window; at 0.999 it barely distinguishes recent from distant, blurring everything into one average. Neither alone is a language model.
Multi-scale retention: many horizons at once
The fix is to stop choosing. Multi-scale retention (MSR) gives every head its own decay, spread geometrically across scales — roughly γ_h = 1 - 2^(-5-h) for head index h. With eight heads that yields horizons of about 32, 64, 128, 256, 512, 1024, 2048, and 4096 tokens, all computed in the same layer and concatenated.
This recreates something attention gets for free. Short-γ heads handle syntax, agreement, and the preceding clause; long-γ heads hold topic, persona, and document-level state. Because γ_h is a scalar multiplying the whole state, each head is still a clean fade — the three equivalent forms survive untouched. Note the decays are fixed constants, not learned parameters: learning them would add little expressivity while risking drift to 1 (state blows up) or toward 0 (head goes blind), and constants keep the decay mask precomputable.