A linear time-invariant state space model applies the same dynamics to every token: the state matrices are fixed, so the model compresses history by a rule that cannot look at what the history actually says. That is a real ceiling — an LTI SSM cannot latch onto a rare name and discard a page of filler, because ‘latch’ and ‘discard’ are content-dependent decisions. Selectivity is the one-line modification that lifts it: make the SSM’s parameters functions of the input. The line is short; the consequences are not. Time-invariance dies, the convolution kernel dies with it, and the FFT training path must be replaced by a parallel scan and a carefully fused kernel. This piece works through the mechanism precisely: what becomes input-dependent and what does not, why Δ is secretly a gate, and what selection costs and buys.
What time-invariance forbids
Recall the discretized LTI recurrence: x_k = A_d x_{k-1} + B_d u_k, y_k = C x_k, where A_d, B_d, C are computed once and reused at every position. That constancy is exactly what buys the convolutional view — unrolling gives y_k = Σ_j (C A_d^{k-j} B_d) u_j, and because the coefficient depends only on the gap k−j, the whole layer is one fixed kernel applied by FFT.
But a fixed kernel is a fixed policy. Every token is written into the state with the same strength B_d, and every token’s influence decays at the same rate set by the eigenvalues of A_d. Two synthetic tasks expose this. Selective copying asks the model to reproduce a few marked tokens scattered among distractors: it must write some inputs and skip others, which a content-blind B_d cannot do. Induction heads ask it to recall what followed the previous occurrence of the current token, which a fixed geometric decay cannot hold. LTI SSMs fail both while attention solves both, and that gap is what selection was designed to close.
The mechanism: parameters as functions of the input
The change is deliberately minimal. In a selective SSM (the S6 layer at the heart of Mamba), three quantities become projections of the current token u_t:
B_t = Linear_N(u_t) B_t: [B, L, N]
C_t = Linear_N(u_t) C_t: [B, L, N]
Δ_t = softplus(d_bias + Linear_D(u_t)) Δ_t: [B, L, D]
A = −exp(A_log) (learned, NOT input-dependent) A: [D, N]with B = batch, L = length, D = channels, N = state size (typically 16). Two design choices deserve attention. First, A stays input-independent — input-dependence reaches it only through Δ_t at discretization, which keeps the parameterization small and the dynamics stable. Second, A is parameterized as −exp(A_log), forcing every diagonal entry negative so exp(ΔA) is always a contraction and the recurrence cannot explode. softplus plays the same role for Δ: it keeps the step size strictly positive and differentiable.
Discretization becomes a per-token operation
In the LTI case you discretize once. Here Δ changes every step, so zero-order hold must be applied at every position:
A_d,t = exp(Δ_t · A) [B, L, D, N]
B_d,t = (Δ_t A)^-1 (exp(Δ_t A) − I) · Δ_t B_t ≈ Δ_t · B_t
x_t = A_d,t ⊙ x_{t-1} + B_d,t · u_t x_t: [B, D, N]
y_t = Σ_n C_t[n] · x_t[:, n] y_t: [B, D]Because A is diagonal, exp(Δ_t A) is an elementwise exponential, not a matrix exponential — D×N scalar exp calls per token rather than an N×N decomposition. The first-order simplification B_d,t ≈ Δ_t B_t is what implementations actually use; it is the Euler limit of the ZOH formula. Note the shape that appears here and nowhere in an LTI SSM: [B, L, D, N]. The discretized matrices are now a full length-by-state tensor rather than a single reused pair, and that tensor is the central engineering problem the rest of the design exists to solve.
Δ is a gate, exactly
The most illuminating fact about selectivity is that it is not a new idea so much as a rediscovery. Take the degenerate case N = 1, A = −1, B_t = 1, with Δ_t = softplus(s_t) for some projection s_t = Linear(u_t). Then:
A_d,t = exp(−Δ_t) = exp(−softplus(s_t))
= 1 / (1 + exp(s_t)) = σ(−s_t) = 1 − σ(s_t)
B_d,t = 1 − A_d,t = σ(s_t) ≡ g_t
⇒ x_t = (1 − g_t) · x_{t-1} + g_t · u_tThat is precisely the update gate of a GRU — derived, not bolted on. Selectivity is therefore best understood as gating generalized to an N-dimensional state: classic RNNs gate a scalar per channel, a selective SSM gates an entire structured state with a learned continuous-time timescale. It also explains why Δ carries most of the weight in practice. B_t and C_t steer which direction of the state a token writes to and reads from; Δ_t decides whether the past survives at all.
Reading Δ as a memory horizon
Put numbers on it. Take a single channel with A = −1 and ask how much of the previous state survives one step, A_d = exp(−Δ), and how many steps until an input decays to 1/e of its initial contribution — an effective horizon of roughly 1/Δ tokens.
| Δ_t | A_d = exp(−Δ) | State retained | Horizon ≈ 1/Δ | Behaviour |
|---|---|---|---|---|
| 0.01 | 0.990 | 99% | 100 tokens | Ignore this token, hold memory |
| 0.1 | 0.905 | 90% | 10 tokens | Blend gently |
| 1.0 | 0.368 | 37% | 1 token | Strong write |
| 5.0 | 0.0067 | 0.7% | <1 token | Reset: focus on the present |
The two extremes are the whole story. As Δ_t → ∞ the state is flushed and replaced by the current input; as Δ_t → 0 the input is ignored and the state persists unchanged. A selective model can emit Δ ≈ 0.01 on filler and Δ ≈ 4 on a proper noun, giving it — per channel, per token — the write/skip decision an LTI kernel structurally cannot make. It is also why Δ initialization matters: implementations set the softplus bias so initial Δ lands near [0.001, 0.1], biasing a fresh model toward remembering rather than resetting.
The convolution dies
Now the bill. Unroll the time-varying recurrence and the coefficient on input u_j in output y_k is:
y_k = Σ_{j≤k} C_k · ( ∏_{i=j+1}^{k} A_d,i ) · B_d,j · u_jCompare with the LTI version, C A_d^{k-j} B_d. There the coefficient was a function of the gap k−j alone; here it depends on every A_d,i strictly between j and k, which depend on the actual tokens sitting there — so two identical gaps in different parts of the sequence get different weights. That is the definition of a time-varying system, and it means no single kernel K with y = K ∗ u exists. No kernel means no FFT, and the O(L log L) parallel training path that made S4 practical is gone. The recurrence is now the only formulation, and it must be made parallel by other means.
The associative scan restores parallel depth
The rescue is that a first-order linear recurrence is an associative scan. Write each step as a pair with a_t = A_d,t and b_t = B_d,t · u_t, so x_t = a_t x_{t-1} + b_t, and define the composition of two consecutive steps:
(a_1, b_1) ⊕ (a_2, b_2) = (a_2 · a_1, a_2 · b_1 + b_2)Substituting confirms this is exactly ‘apply step 1, then step 2’, and the operator is associative: composing steps 1…4 as ((1⊕2)⊕(3⊕4)) gives the same result as left-to-right. Associativity is the licence to parallelize. A Blelloch-style work-efficient scan computes all prefixes in O(L) total work and O(log L) depth, so the sequence resolves in a logarithmic number of rounds rather than L sequential ones. The scan does more raw arithmetic than the FFT convolution it replaces, but that is the price of input-dependence.
Why the fused kernel is the real contribution
Parallel depth is not enough, because the algorithm is memory-bound, not compute-bound. The offender is the [B, L, D, N] tensor. Take a modest layer: B = 1, L = 8192, D = 2048 (inner width after the usual 2× expansion), N = 16. Then
elements = 1 × 8192 × 2048 × 16 ≈ 2.7 × 10^8
bytes (fp16) ≈ 537 MB — for A_d alone, per layerWriting that to HBM and reading it back would swamp the runtime; the arithmetic is trivial by comparison. The fix is kernel fusion: load only the compact tensors (Δ, A, B_t, C_t, all O(L(D+N))) from HBM into SRAM, do the discretization and the scan and the output contraction there, and write back only y: [B, L, D]. The expanded state never touches main memory. The backward pass applies the same logic through recomputation — intermediate states are regenerated in SRAM rather than stored, trading cheap flops for expensive bandwidth.
What it costs at inference, and on a CPU
Inference is where the design pays off, and where the scan disappears. Autoregressive decoding is inherently sequential, so there is nothing to scan: you run the plain recurrence one token at a time. Per token per layer that is roughly D×N multiply-adds for the state update plus D×N for the readout — with D = 2048, N = 16, about 33k FMAs each, small next to the projections. The state is D×N numbers, about 128 KB in fp32, and it is constant: token 100 and token 100,000 cost the same, with no KV cache growing behind them.
For CPU-bound small models that profile is unusually friendly: decode is a tight loop over a cache-resident state instead of a streaming pass over an ever-growing cache, so it is bandwidth-cheap and latency-flat. The awkward case is prefill — without a convolution you cannot batch a long prompt into a GEMM, and a sequential loop over thousands of positions is slow. Practical implementations chunk the prompt: scan within a block, carry the block’s state forward.
Pitfalls, and what selection does not fix
Three failure modes recur. Numerical drift: the state is a long product of contractions, so fp16 accumulation loses low-order bits over thousands of steps; keep Δ and the state in fp32. Bad Δ initialization: start it too large and the model resets its state every token, behaving like a bag-of-words. Missing the kernel: a reference implementation that materializes [B, L, D, N] in HBM can be an order of magnitude slower than the fused version — the math is not the speed.
The deeper limit is structural. Selection changes what gets compressed, not how much: the state is still D×N numbers, so verbatim recall over arbitrary long spans stays bounded by that capacity in a way attention’s growing KV cache is not. Hence the pragmatic design — hybrids that interleave a few attention layers among many selective SSM layers, letting the SSM carry cheap linear-time context while attention supplies exact lookup where it is genuinely needed.