Self-attention has a surprising blind spot: on its own, it cannot tell “dog bites man” from “man bites dog.” The mechanism treats a sentence as an unordered set of vectors, so word order — the thing that carries much of a language’s meaning — is invisible to it. Positional encoding is the fix: a way of stamping each token with information about where it sits, so attention can reason about sequence, distance, and direction. This article is the foundation of the topic. We derive why the problem exists, build the original sinusoidal encoding from first principles, work a small numeric example, and lay out the design axes — fixed vs learned, absolute vs relative — that later methods like RoPE and ALiBi navigate.
Attention is permutation invariant
Start with the core computation. Self-attention takes an input matrix X: [N, d] of N token vectors, projects it into queries, keys, and values, and computes softmax(QK^T / sqrt(d_k)) V. Now ask: what happens if we shuffle the rows of X?
Because every row is projected by the same weight matrices and the attention scores are computed pairwise, permuting the input tokens simply permutes the output tokens in the same way — the actual values are unchanged. Formally, for any permutation matrix P, attention satisfies Attn(PX) = P · Attn(X). The layer is permutation equivariant: it has no built-in notion that token 3 comes after token 2. A bag of words and a sentence look identical to it. That is the hole positional encoding exists to fill.
Injecting position into the tokens
The transformer’s answer is disarmingly simple: before the first attention layer, give every token a second vector that depends only on its position, and combine it with the token’s content embedding. The original design adds them: h_i = E(token_i) + PE(i), where E is the content embedding and PE(i) is the positional encoding for slot i, both of dimension d.
After this addition the two identical tokens at different positions produce different input vectors, so the query and key derived from them differ, and attention can finally distinguish order. The elegance is that nothing in the attention math changes — we simply enrich the input so that ‘where’ is baked into the same vectors that carry ‘what.’ Everything downstream then has position available to use or ignore as the data demands.
The sinusoidal construction
The original transformer used a fixed, non-learned encoding built from sines and cosines. For position pos and dimension index i (with 0 ≤ i < d/2):
PE(pos, 2i) = sin( pos / 10000^(2i/d) )
PE(pos, 2i+1) = cos( pos / 10000^(2i/d) )Each pair of dimensions (2i, 2i+1) is a sine/cosine pair spinning at its own angular frequency ω_i = 1 / 10000^(2i/d). Low indices spin fast (short wavelength); high indices spin slowly (wavelength up to roughly 2π × 10000 tokens). The result is that each position gets a unique d-dimensional fingerprint, and reading across the dimensions is like reading a number in a mixed-radix, continuous form — a smooth multi-scale ‘clock’ where different hands tick at different rates.
Why sinusoids: the relative-shift property
The real magic is not uniqueness — a lookup table gives that too. It is that sinusoids make relative position a linear operation. For a fixed offset k, the encoding at pos + k is a linear function of the encoding at pos. Using the angle-addition identities, for each frequency ω:
sin(ω(pos+k)) = cos(ωk)·sin(ωpos) + sin(ωk)·cos(ωpos)
cos(ω(pos+k)) = cos(ωk)·cos(ωpos) - sin(ωk)·sin(ωpos)That is exactly a 2×2 rotation by angle ωk applied to the (sin, cos) pair. So shifting a position by k is a rotation that depends on k but not on pos. A linear attention head can therefore learn to attend ‘three tokens back’ uniformly across the sequence — the geometry of the encoding hands relative position to the model for free.
A small worked example
Take d = 4, so two frequency pairs, and use base 10000. The frequencies are ω_0 = 1 and ω_1 = 1 / 10000^(2/4) = 1/100 = 0.01. Then:
PE(0) = [ sin 0, cos 0, sin 0, cos 0 ] = [0, 1, 0, 1]
PE(1) = [ sin 1, cos 1, sin .01, cos .01 ] ≈ [0.841, 0.540, 0.010, 1.000]
PE(2) = [ sin 2, cos 2, sin .02, cos .02 ] ≈ [0.909, -0.416, 0.020, 1.000]Notice the first pair changes quickly from position to position while the second pair barely moves — the slow, high-index dimensions encode coarse, long-range location, and the fast, low-index dimensions encode fine, local offsets. Every position vector is distinct, and neighbouring positions have similar vectors, so distance in sequence maps to distance in encoding space.
Fixed vs learned encodings
The sinusoidal scheme is fixed: it is a deterministic function, with zero parameters and defined for any position. The main alternative is a learned positional embedding — a trainable table with one vector per position, exactly like the token embedding table but indexed by slot. BERT and GPT-2 use this.
Learned embeddings are flexible: the model discovers whatever positional geometry the task rewards, and they often match or slightly beat sinusoids in-distribution. Their weakness is hard-edged: the table has a fixed number of rows, so a position beyond the maximum training length has no embedding at all. Fixed sinusoids, by contrast, are defined for every position, giving them a fighting chance at longer sequences. This is the first appearance of the tension that dominates modern positional-encoding research: in-distribution quality versus length extrapolation.
Absolute vs relative position
Both schemes above are absolute: they answer ‘which slot is this, counting from the start?’ But language is largely governed by relative structure — agreement, dependencies, and phrase boundaries depend on how far apart two tokens are, not on their absolute indices. The sentence means the same whether it starts at position 0 or position 500.
This motivates relative positional encoding, which conditions attention on the offset i - j between a query at i and a key at j rather than on i and j individually. Shaw et al. (2018) did this by adding learned vectors keyed on relative distance to the attention computation. The sinusoidal shift property we derived hints at why relative framing is natural: the encoding already turns a shift into a fixed rotation. Relative methods make that structure explicit rather than hoping a model recovers it from absolute inputs.
Where position enters the network
A second design axis is where position is injected. The classic recipe adds PE once, at the input, and lets it propagate through every layer. That is simple and cheap, but the positional signal is added to content and then repeatedly transformed, so by deep layers it is entangled and partly washed out.
An alternative is to inject position inside each attention layer, modifying Q, K, or the scores directly at every depth. This keeps positional information crisp throughout the stack and expresses it exactly where it is used — in the query-key interaction. This is precisely the move RoPE and ALiBi make, and a big part of why they extrapolate and scale better than a single input-side addition.
The extrapolation problem
Extrapolation — running on sequences longer than anything seen in training — is where encodings diverge sharply. Learned absolute embeddings fail outright: no row exists for the new positions. Sinusoidal encodings are at least defined everywhere, but they still tend to degrade, because attention heads were only ever tuned on the frequency patterns present at training lengths and behave unpredictably on unseen combinations.
The practical impact is direct: a model’s useful context window is bounded not just by memory and compute but by whether its positional scheme still makes sense past the training horizon. That single requirement drives much of the design of ALiBi (which biases attention by raw distance) and the RoPE-scaling tricks (position interpolation, NTK-aware scaling) used to stretch context windows after training.
Cost, shapes, and CPU-SLM notes
Positional encoding is computationally almost free. Fixed sinusoids can be precomputed once into a [max_len, d] table and cached; adding them is an O(N × d) element-wise operation, negligible beside the O(N^2 × d) attention itself. Learned embeddings add max_len × d parameters — usually a small slice of the model.
For CPU-hosted small models the choices still matter. A precomputed sinusoidal table costs no parameters and nothing per step beyond a cache read, friendly to tight memory budgets. Methods that rotate Q and K per layer (RoPE) add a handful of multiply-adds per token per layer — cheap, but not zero when you are counting flops on a laptop-class deployment. The encoding you pick also fixes your achievable context length, often the real constraint on a small model.
Common pitfalls
A few mistakes recur. First, forgetting position entirely in a from-scratch implementation — the model trains, loss drops, and it silently behaves like a bag-of-words, because attention gave you no error, just no ordering. Second, off-by-one and base confusion: mixing up the 2i/d exponent or the 10000 base yields frequencies that do not span the intended range.
Third, assuming any scheme extrapolates: deploying well beyond the training length and being surprised by garbage output. The safe habit is to be explicit about which axis you are on — fixed or learned, absolute or relative, input-side or in-layer — and to test at, and beyond, your target sequence length before trusting the result.