Attention is permutation invariant
Start with the core computation. Self-attention takes an input matrix X: [N, d] of N token vectors, projects it into queries, keys, and values, and computes softmax(QK^T / sqrt(d_k)) V. Now ask: what happens if we shuffle the rows of X?
Because every row is projected by the same weight matrices and the attention scores are computed pairwise, permuting the input tokens simply permutes the output tokens in the same way — the actual values are unchanged. Formally, for any permutation matrix P, attention satisfies Attn(PX) = P · Attn(X). The layer is permutation equivariant: it has no built-in notion that token 3 comes after token 2. A bag of words and a sentence look identical to it. That is the hole positional encoding exists to fill.
Injecting position into the tokens
The transformer’s answer is disarmingly simple: before the first attention layer, give every token a second vector that depends only on its position, and combine it with the token’s content embedding. The original design adds them: h_i = E(token_i) + PE(i), where E is the content embedding and PE(i) is the positional encoding for slot i, both of dimension d.
After this addition the two identical tokens at different positions produce different input vectors, so the query and key derived from them differ, and attention can finally distinguish order. The elegance is that nothing in the attention math changes — we simply enrich the input so that ‘where’ is baked into the same vectors that carry ‘what.’ Everything downstream then has position available to use or ignore as the data demands.
The sinusoidal construction
The original transformer used a fixed, non-learned encoding built from sines and cosines. For position pos and dimension index i (with 0 ≤ i < d/2):
PE(pos, 2i) = sin( pos / 10000^(2i/d) )
PE(pos, 2i+1) = cos( pos / 10000^(2i/d) )Each pair of dimensions (2i, 2i+1) is a sine/cosine pair spinning at its own angular frequency ω_i = 1 / 10000^(2i/d). Low indices spin fast (short wavelength); high indices spin slowly (wavelength up to roughly 2π × 10000 tokens). The result is that each position gets a unique d-dimensional fingerprint, and reading across the dimensions is like reading a number in a mixed-radix, continuous form — a smooth multi-scale ‘clock’ where different hands tick at different rates.
Why sinusoids: the relative-shift property
The real magic is not uniqueness — a lookup table gives that too. It is that sinusoids make relative position a linear operation. For a fixed offset k, the encoding at pos + k is a linear function of the encoding at pos. Using the angle-addition identities, for each frequency ω:
sin(ω(pos+k)) = cos(ωk)·sin(ωpos) + sin(ωk)·cos(ωpos)
cos(ω(pos+k)) = cos(ωk)·cos(ωpos) - sin(ωk)·sin(ωpos)That is exactly a 2×2 rotation by angle ωk applied to the (sin, cos) pair. So shifting a position by k is a rotation that depends on k but not on pos. A linear attention head can therefore learn to attend ‘three tokens back’ uniformly across the sequence — the geometry of the encoding hands relative position to the model for free.
A small worked example
Take d = 4, so two frequency pairs, and use base 10000. The frequencies are ω_0 = 1 and ω_1 = 1 / 10000^(2/4) = 1/100 = 0.01. Then:
PE(0) = [ sin 0, cos 0, sin 0, cos 0 ] = [0, 1, 0, 1]
PE(1) = [ sin 1, cos 1, sin .01, cos .01 ] ≈ [0.841, 0.540, 0.010, 1.000]
PE(2) = [ sin 2, cos 2, sin .02, cos .02 ] ≈ [0.909, -0.416, 0.020, 1.000]Notice the first pair changes quickly from position to position while the second pair barely moves — the slow, high-index dimensions encode coarse, long-range location, and the fast, low-index dimensions encode fine, local offsets. Every position vector is distinct, and neighbouring positions have similar vectors, so distance in sequence maps to distance in encoding space.
Fixed vs learned encodings
The sinusoidal scheme is fixed: it is a deterministic function, with zero parameters and defined for any position. The main alternative is a learned positional embedding — a trainable table with one vector per position, exactly like the token embedding table but indexed by slot. BERT and GPT-2 use this.
Learned embeddings are flexible: the model discovers whatever positional geometry the task rewards, and they often match or slightly beat sinusoids in-distribution. Their weakness is hard-edged: the table has a fixed number of rows, so a position beyond the maximum training length has no embedding at all. Fixed sinusoids, by contrast, are defined for every position, giving them a fighting chance at longer sequences. This is the first appearance of the tension that dominates modern positional-encoding research: in-distribution quality versus length extrapolation.
Absolute vs relative position
Both schemes above are absolute: they answer ‘which slot is this, counting from the start?’ But language is largely governed by relative structure — agreement, dependencies, and phrase boundaries depend on how far apart two tokens are, not on their absolute indices. The sentence means the same whether it starts at position 0 or position 500.
This motivates relative positional encoding, which conditions attention on the offset i - j between a query at i and a key at j rather than on i and j individually. Shaw et al. (2018) did this by adding learned vectors keyed on relative distance to the attention computation. The sinusoidal shift property we derived hints at why relative framing is natural: the encoding already turns a shift into a fixed rotation. Relative methods make that structure explicit rather than hoping a model recovers it from absolute inputs.
Where position enters the network
A second design axis is where position is injected. The classic recipe adds PE once, at the input, and lets it propagate through every layer. That is simple and cheap, but the positional signal is added to content and then repeatedly transformed, so by deep layers it is entangled and partly washed out.
An alternative is to inject position inside each attention layer, modifying Q, K, or the scores directly at every depth. This keeps positional information crisp throughout the stack and expresses it exactly where it is used — in the query-key interaction. This is precisely the move RoPE and ALiBi make, and a big part of why they extrapolate and scale better than a single input-side addition.