ALiBi — Attention with Linear Biases is a way to tell a transformer where each token sits without ever adding a positional vector to anything. It touches neither the token embeddings nor the query and key vectors. Instead it does one small thing, right before the softmax: it subtracts a penalty proportional to how far apart two tokens are from their raw attention score. Closer tokens keep their score; distant tokens are pushed down by a straight line whose steepness is fixed per attention head. That is the entire idea. From it flow three surprisingly large consequences: attention is biased toward recent tokens, the model extrapolates to sequences far longer than it ever saw in training, and it costs almost nothing — no embedding table, no extra matrix multiply. This piece derives the bias, lays out the per-head slope schedule, works a small numeric example by hand, and draws the sharp line between ALiBi (a bias on scores) and RoPE or sinusoidal encodings (transformations of the vectors).
What ALiBi actually changes
Standard attention computes a score for every query–key pair, s_ij = q_i · k_j / sqrt(d_k), then softmaxes each query’s row into weights. On its own this score is permutation-invariant: shuffle the tokens and the scores are unchanged, because a dot product knows nothing about order. Every position scheme exists to break that symmetry. Sinusoidal and learned embeddings do it at the input; RoPE does it inside attention, rotating q and k by a position-dependent angle.
ALiBi does it later still, and more bluntly. It leaves q, k, and the embeddings completely alone and instead adds a fixed, non-learned bias directly to the score matrix before softmax. The bias depends only on the distance between the two positions and on which head is looking — no parameter to train, no table to index, nothing about token content. That single relocation, from ‘encode position in the vectors’ to ‘bias the scores by distance,’ is what gives ALiBi all of its properties.
The bias formula
For a query at position i attending to a key at position j, ALiBi modifies the pre-softmax score to:
s_ij = q_i · k_j / sqrt(d_k) - m · |i - j|
attn_i = softmax( s_i,0 , s_i,1 , ... , s_i,i )In a causal (decoder) language model a query only sees keys at or before it, so j ≤ i and the distance |i - j| = i - j is always ≥ 0. The bias term -m · (i - j) is therefore zero for the token attending to itself and grows linearly more negative the further back the key is. The scalar m > 0 is the slope: it sets how hard the penalty bites per step of distance. Written as a row, the biases added to one query’s scores are m · [ -(i), ..., -2, -1, 0 ] — a ramp, flat at the present token and sliding downhill into the past. Because it is added inside the softmax, a bias of -m·k multiplies that key’s unnormalized weight by exp(-m·k), an exponential decay with distance.
The per-head slope schedule
If every head used the same slope, the model could only look back at one characteristic range. ALiBi instead hands each head a different slope, chosen as a geometric sequence. For a model with H heads (a power of two), head h = 1, 2, ..., H gets:
m_h = 2^(-8h/H)
H = 8 -> m = 1/2, 1/4, 1/8, 1/16, 1/32, 1/64, 1/128, 1/256
= 2^-1, 2^-2, 2^-3, ... , 2^-8The sequence starts at 2^(-8/H) and multiplies by that same ratio each step, so the slopes span three orders of magnitude. A head with a steep slope (m = 1/2) penalizes distance hard and effectively attends only to the last few tokens — a sharp, local, recency head. A head with a shallow slope (m = 1/256) barely penalizes distance at all, keeping a nearly flat view across hundreds of tokens — a long-range head. The geometric spread guarantees the ensemble covers many scales of context at once. The slopes are fixed constants, not learned; the authors found that making them trainable gave no benefit, so they stay hard-coded. For head counts that are not a power of two, the schedule is interpolated, but the power-of-two case is the one to remember.
A worked example
Take head h with slope m = 1/4 and a query at position i = 4 attending over keys j = 0..4. Suppose the raw scaled scores q·k/sqrt(d_k) all happen to equal 2.0 — i.e. content alone is indifferent about which key to pick. The distances and biases are:
| key j | distance i−j | bias −m·(i−j) | biased score | softmax weight |
|---|---|---|---|---|
| 0 | 4 | −1.00 | 1.00 | 0.114 |
| 1 | 3 | −0.75 | 1.25 | 0.146 |
| 2 | 2 | −0.50 | 1.50 | 0.188 |
| 3 | 1 | −0.25 | 1.75 | 0.241 |
| 4 | 0 | 0.00 | 2.00 | 0.310 |
The weights come from exp(biased score) normalized (exp(1.0)=2.72, exp(1.25)=3.49, ... , exp(2.0)=7.39, summing to ≈23.8). Even though the content scores were identical, the most recent token now receives 0.310 of the attention and the oldest only 0.114 — a 2.7× preference, produced entirely by the linear bias. Swap in the shallow head m = 1/256 and the biases become tiny (−0.016 at distance 4), the weights flatten toward uniform, and that head sees the whole span almost evenly.
Why it biases toward recency
The recency effect is not a heuristic bolted on top — it falls straight out of the exponential in softmax. A key that is k steps back has its unnormalized attention weight scaled by exp(-m · k) relative to the present token. That is geometric decay: each additional step back multiplies the ceiling on a token’s influence by a constant factor exp(-m) less than one. For m = 1/2, ten tokens back the factor is exp(-5) ≈ 0.0067 — that key is essentially muted unless its content score is enormous. Content can still win: a strongly relevant distant token with a high q·k can overcome the penalty. But all else equal, nearer tokens dominate.
This encodes a useful prior for language: the next token depends most on what was just said, with a soft, smoothly fading memory of everything earlier. The per-head slopes turn that into a spectrum — steep heads enforce tight locality, shallow heads preserve long-range signal — so the model gets a whole bank of memory lengths at once rather than choosing one.
Why it extrapolates to longer sequences
ALiBi’s headline result is train short, test long: a model trained on, say, 1024-token sequences keeps its perplexity stable when run on 3072 tokens, where embedding-based schemes fall apart. The reason is that ALiBi has nothing that runs out. A learned positional embedding table has no row for position 2000 if you trained to 1024. Sinusoidal encodings do have values at every position, but the model never saw those high-frequency phase combinations, so the input distribution shifts and quality degrades.
ALiBi’s bias, by contrast, is the same linear function at any distance: -m · (i - j) is well defined at distance 5000 even if training never exceeded 1024. And because the exponential decay makes far-back tokens contribute almost nothing, a query at a distant test position sees an attention distribution that looks, in its dominant near region, just like training. The effective receptive field is bounded by the slope, not by the sequence length, so stretching the sequence never creates an unfamiliar regime — extrapolation is a free side effect of a bias that never had a maximum position baked into it.
The compute cheapness: no embeddings, no matmuls
ALiBi is close to free, in three senses. First, parameters: there is no positional embedding table (which for large models and long contexts can be a non-trivial chunk of weights) and the slopes are constants, so ALiBi adds exactly zero learnable parameters. Second, FLOPs: RoPE has to rotate every query and key vector at every layer, an elementwise multiply-add across the head dimension; ALiBi does none of that. Its bias is a single element-wise addition to the score matrix that already exists.
Third, memory and precompute: the bias matrix is a static, distance-only Toeplitz pattern — every diagonal is a constant — so it can be computed once per head and reused, or fused directly into the attention kernel and the causal mask, and need never be materialized as a full N×N tensor. For a CPU-bound small language model, where every avoided multiply and skipped lookup matters, ALiBi is attractive precisely because it moves position handling out of the hot vector math and into a cheap additive constant.
ALiBi vs RoPE and sinusoidal: a bias, not an encoding
The cleanest way to hold the difference is where in the pipeline the position information lives. Sinusoidal and learned encodings live in the input embeddings: a position vector is added to each token before layer one and propagates through the network. RoPE lives in the query and key vectors: it rotates q and k by an angle proportional to position, so their dot product comes to depend on relative position. Both are encodings — they change what the vectors are.
ALiBi changes none of the vectors. It lives in the score matrix as an additive scalar bias applied after the dot product and before softmax. Position is never carried in a representation; it exists only as a penalty on attention weights.
| Scheme | Acts on | Learned params | Extra compute | Extrapolates |
|---|---|---|---|---|
| Learned pos. emb. | input embeddings | yes (table) | lookup + add | poorly |
| Sinusoidal | input embeddings | no | add | weakly |
| RoPE | q and k vectors | no | rotate q,k each layer | moderately |
| ALiBi | attention scores | no | one add to scores | strongly |
Practical implications and pitfalls
ALiBi is a natural fit for autoregressive, decoder-only models — the recency prior and the causal mask agree, and the additive bias drops cleanly into a KV-cache decode loop because a new query’s biases against all cached keys are just -m · (i - j), computed on the fly with no state. Its cheapness and extrapolation make it appealing for long-context CPU-SLM settings.
But keep the limits honest. The recency bias is an assumption: when a crucial token sits far in the past, a steep-slope head actively suppresses it and the model must lean on the shallow heads to carry that signal. ‘Extrapolates’ means perplexity stays stable, not that the model exploits unlimited range — the effective window is bounded by the slopes, so ALiBi is no magic key to true long-range recall. The recency intuition also does not transfer unchanged to bidirectional encoders, which need a symmetric variant, and the 2^(-8h/H) schedule is a prior baked in, not learned from your data. Within those bounds, ALiBi buys position handling, length generalization, and lower cost with one subtraction before the softmax.