Ask a Transformer for its ten-thousandth token and it must re-read all nine thousand nine hundred and ninety-nine before it. That is the price of a KV cache: memory that grows without bound and a per-token cost that creeps upward as the conversation gets useful. RetNet — the Retentive Network (Sun et al., 2023) — proposes a different memory: not a transcript you re-read, but a fixed-size summary you fade and update. The mechanism is retention, and its defining property is that it can be computed three ways — a parallel matmul, a step-by-step recurrence, or a hybrid over chunks — that all produce the same answer. This article builds retention from the ground up, works the decay factor γ numerically, walks the full block, and puts numbers on what constant-memory decoding is worth when RAM is the ceiling.
Retention: fading memory instead of softmax
Attention computes softmax(QK^T / √d_k) V, and its per-token decode cost grows with context because the KV cache grows. The softmax is the obstacle: exp does not decompose into anything you can accumulate incrementally, so you must keep every past key.
Retention removes the softmax and replaces learned, content-dependent weights with a fixed weight by distance: a token m steps back is scaled by γ^m for a constant 0 < γ < 1. The intuition is a leaky bucket — each step the memory you hold is multiplied by γ (it fades), and the new token is poured in on top. The critical algebraic property is that this decay factorizes — γ^(n-m) = γ^n / γ^m — so the past folds into a running accumulator instead of being recomputed. Relative position is carried in two channels: a phase, via an xPos/rotary-style rotation on Q and K, and a magnitude, via the decay.
The recurrent form: one state matrix, forever
Written as a recurrence, retention keeps a single state matrix S_n that stands in for the entire history:
S_n = γ · S_(n-1) + K_n^T V_n S_n : [d_k, d_v], S_0 = 0
o_n = Q_n · S_n o_n : [1, d_v]
Shapes per step: Q_n, K_n, V_n : [1, d]
K_n^T V_n : [d_k, d_v] rank-1 outer product
Q_n S_n : [1,d_k]×[d_k,d_v] → [1, d_v]Read it left to right: fade what you had, add an outer product for the new token, read the answer out by projecting the query through the state. And — the whole point — S_n is the same d_k × d_v shape at token 10 and at token 100,000, so decoding is O(1) time and O(1) memory per token: no cache to grow, nothing to stream that gets bigger, no eviction policy to tune. The recurrence also makes the model’s bias explicit — information is written into a lossy summary and continuously attenuated, never stored verbatim.
The parallel form: a decay mask in place of softmax
A sequential recurrence is fine for generation and terrible for training, where every position should be computed at once. Retention therefore has a second, fully parallel form: attention with the softmax swapped for an elementwise decay mask D.
Retention(X) = (Q K^T ⊙ D) V
D_(nm) = γ^(n-m) if n ≥ m (causal + geometric decay)
= 0 if n < m (no peeking ahead)
Q, K, V = X W_Q, X W_K, X W_V X : [N, d] → Q,K,V : [N, d]
QK^T : [N, N] ⊙ D → × V → [N, d_v]D is one lower-triangular matrix doing two jobs: zeros above the diagonal enforce causality, and the γ^(n-m) entries below apply the fade. Notice there is no row normalization — nothing forces a row of weights to sum to 1, which is why the block adds a GroupNorm afterwards to keep activation scale sane. Computationally this is the same dense-matmul shape attention kernels already target, so training throughput is Transformer-class.
Why the two forms give identical numbers
These are not approximations of each other. Unroll the recurrence from S_0 = 0 and the parallel form falls out:
S_n = Σ_(m=1..n) γ^(n-m) K_m^T V_m
o_n = Q_n S_n
= Σ_(m=1..n) γ^(n-m) (Q_n K_m^T) V_m
= Σ_(m=1..n) (Q_n K_m^T · D_(nm)) V_m ⇐ row n of (QK^T ⊙ D)VWhat makes it work is associativity plus the factorizable decay: because γ^(n-m) splits into a term depending only on n and one depending only on m, the double sum collapses into a single accumulator faded once per step. Softmax cannot do this, which is exactly why attention has no cheap recurrent twin. The practical payoff: you train with the parallel form and serve with the recurrent form on the very same weights — no distillation, no second model, no train/serve mismatch to debug.
Chunkwise-recurrent: the form you actually train long context with
The parallel form is O(N^2); the recurrent form is linear but sequential, leaving vector units idle. The chunkwise-recurrent form splits the difference: cut the sequence into chunks of size B, run the parallel form inside a chunk, carry a recurrent state between chunks.
For chunk i, inner positions j = 1..B:
inner (parallel) I_i = (Q_i K_i^T ⊙ D) V_i
cross (carry-in) C_i = (Q_i R_(i-1)) ⊙ ξ, ξ_j = γ^j
state (carry-out) R_i = γ^B R_(i-1) + K_i^T (V_i ⊙ ζ), ζ_j = γ^(B-j)
Retention(X_i) = I_i + C_i cost ≈ O(N B d + N d^2)Each token’s output sums what it retains from its own chunk and what it inherits from the compressed past. Cost is quadratic only in B, which you choose, so the total is linear in N while still being dominated by large matmuls. This is the form for prefill: a long prompt is ingested chunk by chunk at high arithmetic intensity, and what survives is one small state handed to the recurrent decoder.
Reading the decay factor γ as a memory horizon
γ is the personality of a retention head. Its weight on a token k steps back is γ^k, so it defines an effective receptive field. The weights sum to Σ_k γ^k = 1/(1-γ), which doubles as the rule of thumb for the horizon — how many recent tokens carry essentially all the mass. Worked out:
k : 1 10 50 200 1000 horizon 1/(1-γ)
0.90^k : 0.900 0.349 0.005 ~0 ~0 10
0.99^k : 0.990 0.904 0.605 0.134 0.00004 100
0.999^k : 0.999 0.990 0.951 0.819 0.368 1000The table shows the trap in a single fixed γ. At 0.9 the head is functionally a 10-token window; at 0.999 it barely distinguishes recent from distant, blurring everything into one average. Neither alone is a language model.
Multi-scale retention: many horizons at once
The fix is to stop choosing. Multi-scale retention (MSR) gives every head its own decay, spread geometrically across scales — roughly γ_h = 1 - 2^(-5-h) for head index h. With eight heads that yields horizons of about 32, 64, 128, 256, 512, 1024, 2048, and 4096 tokens, all computed in the same layer and concatenated.
This recreates something attention gets for free. Short-γ heads handle syntax, agreement, and the preceding clause; long-γ heads hold topic, persona, and document-level state. Because γ_h is a scalar multiplying the whole state, each head is still a clean fade — the three equivalent forms survive untouched. Note the decays are fixed constants, not learned parameters: learning them would add little expressivity while risking drift to 1 (state blows up) or toward 0 (head goes blind), and constants keep the decay mask precomputable.
The full block: gate, norm, and why they are needed
A RetNet layer wraps retention in machinery that earns its place: a gated multi-scale retention sublayer followed by a feed-forward network, each with a residual and pre-LayerNorm.
Y = GroupNorm_h( MSR(LN(X)) ) per-head norm, h groups
MSR_out = ( swish(X W_G) ⊙ Y ) W_O swish gate, then output proj
X' = X + MSR_out
out = X' + FFN(LN(X'))Both extras address the missing softmax. Without row normalization, heads produce wildly different scales — a head at γ=0.999 accumulates far more mass than one at γ=0.96 — so GroupNorm per head puts them on comparable footing before concatenation. The swish gate restores the content-dependent modulation that fixed decay gave up: decay decides how long information lives, the gate decides how much of it is admitted for this token. Drop either and training destabilizes or quality drops.
What constant memory is worth on CPU
Put numbers on it. Take a 1.5B-parameter model, 24 layers, 16 heads, d_head = 128, decoding in fp16 at 32K context.
KV cache = 2 · N · L · H · d_head · 2 B
per token : 2 · 24 · 16 · 128 · 2 B = 192 KB / token
@ N=32768 : 192 KB · 32768 ≈ 6.4 GB (and still rising)
RetNet state = L · H · d_k · d_v · 2 B
24 · 16 · 128 · 128 · 2 B ≈ 12.6 MB (flat, any N)
crossover : 12.6 MB / 192 KB per token ≈ 64 tokensRoughly 500× less state at 32K context — and the retention figure is identical at 1K or at 1M, while the cache keeps growing. The fixed state is bigger than the KV cache only for the first few dozen tokens; past about 64 the cache balloons past it without bound. On a bandwidth-bound CPU that gap is largely a token-rate gap too, since decode time is dominated by streaming state and weights from DRAM. Equally valuable operationally: memory is predictable, so you can size a box for N concurrent sessions without modelling a context-length distribution or implementing cache eviction.
Where retention sits, and where it breaks
Retention is best read as linear attention with a decay gate. Linear attention drops the softmax and accumulates S_n = S_(n-1) + φ(K_n)^T V_n; retention adds the fade, which bounds the state, prevents the dilution that plagues an unbounded running sum, and gives each head a horizon. It is equally a state-space model with the simplest transition: set A = γI in h_n = A h_(n-1) + B x_n and you have retention, where S4 learns a structured A and Mamba makes the transition input-dependent.
That simplicity is also the limitation. A content-blind decay means nothing can be protected from fading: a rare identifier 20K tokens back is attenuated by the same γ^k as filler, so exact needle-in-a-haystack recall is structurally harder than for attention, which can point at any token at full weight. Multi-scale heads soften this; they do not remove it. Retention trades unbounded precise lookup for bounded, predictable cost — know which your workload needs, and benchmark recall before committing.
γ^(n-m) over relative position, and because that decay factorizes, the same computation admits three equivalent schedules: a parallel form (QK^T ⊙ D)V for training, a recurrent form S_n = γS_(n-1) + K_n^T V_n with o_n = Q_n S_n for O(1)-per-token, constant-memory decoding, and a chunkwise form that makes long context linear. Train one way, serve another, same weights. The state is a fixed d_k × d_v matrix, so memory is flat and predictable where a KV cache grows without bound — about 12.6 MB versus 6.4 GB and climbing for a 1.5B model at 32K context, a roughly 500× reduction that bites hardest on bandwidth-bound CPUs. The cost of that determinism is a recency bias: each head sees a horizon of roughly 1/(1-γ) tokens, which multi-scale heads spread across scales but never fully escape. Reach for retention when predictable long-context serving matters more than exact long-range recall.