What time-invariance forbids
Recall the discretized LTI recurrence: x_k = A_d x_{k-1} + B_d u_k, y_k = C x_k, where A_d, B_d, C are computed once and reused at every position. That constancy is exactly what buys the convolutional view — unrolling gives y_k = Σ_j (C A_d^{k-j} B_d) u_j, and because the coefficient depends only on the gap k−j, the whole layer is one fixed kernel applied by FFT.
But a fixed kernel is a fixed policy. Every token is written into the state with the same strength B_d, and every token’s influence decays at the same rate set by the eigenvalues of A_d. Two synthetic tasks expose this. Selective copying asks the model to reproduce a few marked tokens scattered among distractors: it must write some inputs and skip others, which a content-blind B_d cannot do. Induction heads ask it to recall what followed the previous occurrence of the current token, which a fixed geometric decay cannot hold. LTI SSMs fail both while attention solves both, and that gap is what selection was designed to close.
The mechanism: parameters as functions of the input
The change is deliberately minimal. In a selective SSM (the S6 layer at the heart of Mamba), three quantities become projections of the current token u_t:
B_t = Linear_N(u_t) B_t: [B, L, N]
C_t = Linear_N(u_t) C_t: [B, L, N]
Δ_t = softplus(d_bias + Linear_D(u_t)) Δ_t: [B, L, D]
A = −exp(A_log) (learned, NOT input-dependent) A: [D, N]with B = batch, L = length, D = channels, N = state size (typically 16). Two design choices deserve attention. First, A stays input-independent — input-dependence reaches it only through Δ_t at discretization, which keeps the parameterization small and the dynamics stable. Second, A is parameterized as −exp(A_log), forcing every diagonal entry negative so exp(ΔA) is always a contraction and the recurrence cannot explode. softplus plays the same role for Δ: it keeps the step size strictly positive and differentiable.
Discretization becomes a per-token operation
In the LTI case you discretize once. Here Δ changes every step, so zero-order hold must be applied at every position:
A_d,t = exp(Δ_t · A) [B, L, D, N]
B_d,t = (Δ_t A)^-1 (exp(Δ_t A) − I) · Δ_t B_t ≈ Δ_t · B_t
x_t = A_d,t ⊙ x_{t-1} + B_d,t · u_t x_t: [B, D, N]
y_t = Σ_n C_t[n] · x_t[:, n] y_t: [B, D]Because A is diagonal, exp(Δ_t A) is an elementwise exponential, not a matrix exponential — D×N scalar exp calls per token rather than an N×N decomposition. The first-order simplification B_d,t ≈ Δ_t B_t is what implementations actually use; it is the Euler limit of the ZOH formula. Note the shape that appears here and nowhere in an LTI SSM: [B, L, D, N]. The discretized matrices are now a full length-by-state tensor rather than a single reused pair, and that tensor is the central engineering problem the rest of the design exists to solve.
Δ is a gate, exactly
The most illuminating fact about selectivity is that it is not a new idea so much as a rediscovery. Take the degenerate case N = 1, A = −1, B_t = 1, with Δ_t = softplus(s_t) for some projection s_t = Linear(u_t). Then:
A_d,t = exp(−Δ_t) = exp(−softplus(s_t))
= 1 / (1 + exp(s_t)) = σ(−s_t) = 1 − σ(s_t)
B_d,t = 1 − A_d,t = σ(s_t) ≡ g_t
⇒ x_t = (1 − g_t) · x_{t-1} + g_t · u_tThat is precisely the update gate of a GRU — derived, not bolted on. Selectivity is therefore best understood as gating generalized to an N-dimensional state: classic RNNs gate a scalar per channel, a selective SSM gates an entire structured state with a learned continuous-time timescale. It also explains why Δ carries most of the weight in practice. B_t and C_t steer which direction of the state a token writes to and reads from; Δ_t decides whether the past survives at all.
Reading Δ as a memory horizon
Put numbers on it. Take a single channel with A = −1 and ask how much of the previous state survives one step, A_d = exp(−Δ), and how many steps until an input decays to 1/e of its initial contribution — an effective horizon of roughly 1/Δ tokens.
| Δ_t | A_d = exp(−Δ) | State retained | Horizon ≈ 1/Δ | Behaviour |
|---|---|---|---|---|
| 0.01 | 0.990 | 99% | 100 tokens | Ignore this token, hold memory |
| 0.1 | 0.905 | 90% | 10 tokens | Blend gently |
| 1.0 | 0.368 | 37% | 1 token | Strong write |
| 5.0 | 0.0067 | 0.7% | <1 token | Reset: focus on the present |
The two extremes are the whole story. As Δ_t → ∞ the state is flushed and replaced by the current input; as Δ_t → 0 the input is ignored and the state persists unchanged. A selective model can emit Δ ≈ 0.01 on filler and Δ ≈ 4 on a proper noun, giving it — per channel, per token — the write/skip decision an LTI kernel structurally cannot make. It is also why Δ initialization matters: implementations set the softplus bias so initial Δ lands near [0.001, 0.1], biasing a fresh model toward remembering rather than resetting.
The convolution dies
Now the bill. Unroll the time-varying recurrence and the coefficient on input u_j in output y_k is:
y_k = Σ_{j≤k} C_k · ( ∏_{i=j+1}^{k} A_d,i ) · B_d,j · u_jCompare with the LTI version, C A_d^{k-j} B_d. There the coefficient was a function of the gap k−j alone; here it depends on every A_d,i strictly between j and k, which depend on the actual tokens sitting there — so two identical gaps in different parts of the sequence get different weights. That is the definition of a time-varying system, and it means no single kernel K with y = K ∗ u exists. No kernel means no FFT, and the O(L log L) parallel training path that made S4 practical is gone. The recurrence is now the only formulation, and it must be made parallel by other means.
The associative scan restores parallel depth
The rescue is that a first-order linear recurrence is an associative scan. Write each step as a pair with a_t = A_d,t and b_t = B_d,t · u_t, so x_t = a_t x_{t-1} + b_t, and define the composition of two consecutive steps:
(a_1, b_1) ⊕ (a_2, b_2) = (a_2 · a_1, a_2 · b_1 + b_2)Substituting confirms this is exactly ‘apply step 1, then step 2’, and the operator is associative: composing steps 1…4 as ((1⊕2)⊕(3⊕4)) gives the same result as left-to-right. Associativity is the licence to parallelize. A Blelloch-style work-efficient scan computes all prefixes in O(L) total work and O(log L) depth, so the sequence resolves in a logarithmic number of rounds rather than L sequential ones. The scan does more raw arithmetic than the FFT convolution it replaces, but that is the price of input-dependence.