What ALiBi actually changes
Standard attention computes a score for every query–key pair, s_ij = q_i · k_j / sqrt(d_k), then softmaxes each query’s row into weights. On its own this score is permutation-invariant: shuffle the tokens and the scores are unchanged, because a dot product knows nothing about order. Every position scheme exists to break that symmetry. Sinusoidal and learned embeddings do it at the input; RoPE does it inside attention, rotating q and k by a position-dependent angle.
ALiBi does it later still, and more bluntly. It leaves q, k, and the embeddings completely alone and instead adds a fixed, non-learned bias directly to the score matrix before softmax. The bias depends only on the distance between the two positions and on which head is looking — no parameter to train, no table to index, nothing about token content. That single relocation, from ‘encode position in the vectors’ to ‘bias the scores by distance,’ is what gives ALiBi all of its properties.
The bias formula
For a query at position i attending to a key at position j, ALiBi modifies the pre-softmax score to:
s_ij = q_i · k_j / sqrt(d_k) - m · |i - j|
attn_i = softmax( s_i,0 , s_i,1 , ... , s_i,i )In a causal (decoder) language model a query only sees keys at or before it, so j ≤ i and the distance |i - j| = i - j is always ≥ 0. The bias term -m · (i - j) is therefore zero for the token attending to itself and grows linearly more negative the further back the key is. The scalar m > 0 is the slope: it sets how hard the penalty bites per step of distance. Written as a row, the biases added to one query’s scores are m · [ -(i), ..., -2, -1, 0 ] — a ramp, flat at the present token and sliding downhill into the past. Because it is added inside the softmax, a bias of -m·k multiplies that key’s unnormalized weight by exp(-m·k), an exponential decay with distance.
The per-head slope schedule
If every head used the same slope, the model could only look back at one characteristic range. ALiBi instead hands each head a different slope, chosen as a geometric sequence. For a model with H heads (a power of two), head h = 1, 2, ..., H gets:
m_h = 2^(-8h/H)
H = 8 -> m = 1/2, 1/4, 1/8, 1/16, 1/32, 1/64, 1/128, 1/256
= 2^-1, 2^-2, 2^-3, ... , 2^-8The sequence starts at 2^(-8/H) and multiplies by that same ratio each step, so the slopes span three orders of magnitude. A head with a steep slope (m = 1/2) penalizes distance hard and effectively attends only to the last few tokens — a sharp, local, recency head. A head with a shallow slope (m = 1/256) barely penalizes distance at all, keeping a nearly flat view across hundreds of tokens — a long-range head. The geometric spread guarantees the ensemble covers many scales of context at once. The slopes are fixed constants, not learned; the authors found that making them trainable gave no benefit, so they stay hard-coded. For head counts that are not a power of two, the schedule is interpolated, but the power-of-two case is the one to remember.
A worked example
Take head h with slope m = 1/4 and a query at position i = 4 attending over keys j = 0..4. Suppose the raw scaled scores q·k/sqrt(d_k) all happen to equal 2.0 — i.e. content alone is indifferent about which key to pick. The distances and biases are:
| key j | distance i−j | bias −m·(i−j) | biased score | softmax weight |
|---|---|---|---|---|
| 0 | 4 | −1.00 | 1.00 | 0.114 |
| 1 | 3 | −0.75 | 1.25 | 0.146 |
| 2 | 2 | −0.50 | 1.50 | 0.188 |
| 3 | 1 | −0.25 | 1.75 | 0.241 |
| 4 | 0 | 0.00 | 2.00 | 0.310 |
The weights come from exp(biased score) normalized (exp(1.0)=2.72, exp(1.25)=3.49, ... , exp(2.0)=7.39, summing to ≈23.8). Even though the content scores were identical, the most recent token now receives 0.310 of the attention and the oldest only 0.114 — a 2.7× preference, produced entirely by the linear bias. Swap in the shallow head m = 1/256 and the biases become tiny (−0.016 at distance 4), the weights flatten toward uniform, and that head sees the whole span almost evenly.
Why it biases toward recency
The recency effect is not a heuristic bolted on top — it falls straight out of the exponential in softmax. A key that is k steps back has its unnormalized attention weight scaled by exp(-m · k) relative to the present token. That is geometric decay: each additional step back multiplies the ceiling on a token’s influence by a constant factor exp(-m) less than one. For m = 1/2, ten tokens back the factor is exp(-5) ≈ 0.0067 — that key is essentially muted unless its content score is enormous. Content can still win: a strongly relevant distant token with a high q·k can overcome the penalty. But all else equal, nearer tokens dominate.
This encodes a useful prior for language: the next token depends most on what was just said, with a soft, smoothly fading memory of everything earlier. The per-head slopes turn that into a spectrum — steep heads enforce tight locality, shallow heads preserve long-range signal — so the model gets a whole bank of memory lengths at once rather than choosing one.
Why it extrapolates to longer sequences
ALiBi’s headline result is train short, test long: a model trained on, say, 1024-token sequences keeps its perplexity stable when run on 3072 tokens, where embedding-based schemes fall apart. The reason is that ALiBi has nothing that runs out. A learned positional embedding table has no row for position 2000 if you trained to 1024. Sinusoidal encodings do have values at every position, but the model never saw those high-frequency phase combinations, so the input distribution shifts and quality degrades.
ALiBi’s bias, by contrast, is the same linear function at any distance: -m · (i - j) is well defined at distance 5000 even if training never exceeded 1024. And because the exponential decay makes far-back tokens contribute almost nothing, a query at a distant test position sees an attention distribution that looks, in its dominant near region, just like training. The effective receptive field is bounded by the slope, not by the sequence length, so stretching the sequence never creates an unfamiliar regime — extrapolation is a free side effect of a bias that never had a maximum position baked into it.