Self-attention is permutation-equivariant: shuffle the tokens and the outputs shuffle with them, unchanged. Something has to say where each token sits. Rotary Position Embedding (RoPE) does it with one suspiciously simple idea — chop each query and key vector into two-dimensional pairs and rotate each pair by an angle proportional to the token’s position. Nothing is added, nothing is concatenated, no parameters are learned. The payoff is a property of rotations: the dot product of two rotated vectors depends only on the difference of their angles, so a score written in absolute positions collapses into a function of the relative offset.
Rotating a single 2-D pair
Start with the smallest case: a two-dimensional vector x = (x_0, x_1) belonging to a token at position m. RoPE replaces it with the rotation of x by the angle mθ, where θ is a fixed constant chosen per pair (never learned):
R(mθ) = [ cos(mθ) -sin(mθ) ]
[ sin(mθ) cos(mθ) ]
x’_0 = x_0·cos(mθ) − x_1·sin(mθ)
x’_1 = x_0·sin(mθ) + x_1·cos(mθ)Two facts about rotation matrices carry the entire construction. They compose by adding angles, R(α)·R(β) = R(α+β); and they are orthogonal, so R(α)^T = R(α)^(-1) = R(−α). Orthogonality also means ||x’|| = ||x|| — RoPE never changes the norm of a query or key, only its direction. That is why it drops in without rescaling anything downstream — only the angular relationship carries the new signal.
The complex-exponential form
A 2-D rotation is multiplication by a unit complex number, which shortens the algebra considerably. Identify the pair (x_0, x_1) with z = x_0 + i·x_1. Then
R(mθ)·x ≡ z · e^(i·mθ)
(a + ib)(cosφ + i sinφ) = (a cosφ − b sinφ) + i(a sinφ + b cosφ)The real and imaginary parts of that product are exactly the two rows of the matrix form above. The other identity we need: the real dot product of two 2-vectors is the real part of one complex number times the conjugate of the other, a · b = Re[z_a · conj(z_b)]. Since conj(e^(iφ)) = e^(−iφ), conjugation flips the sign of a rotation angle — which is precisely how a difference of positions is about to fall out of a product of two absolute positions.
The relative-position theorem
Here is the whole reason RoPE exists. Let a query q sit at position m and a key k at position n. Rotate each by its own position and take the score:
⟨R(mθ)q, R(nθ)k⟩
= q^T R(mθ)^T R(nθ) k
= q^T R(−mθ) R(nθ) k [orthogonality]
= q^T R((n − m)θ) k [angles add]
complex view:
Re[ (z_q e^(imθ)) · conj(z_k e^(inθ)) ]
= Re[ z_q conj(z_k) · e^(i(m − n)θ) ]Both lines say the same thing: the absolute positions have cancelled, and the score is a function of q, k, and the offset m − n alone. Expanding gives (q·k)·cos((n−m)θ) + (q×k)·sin((n−m)θ), where q × k = q_1k_0 − q_0k_1 is the 2-D cross product (cos is even, so its argument’s sign is free; sin is odd, so it is not). Absolute rotations go in, a relative-position-dependent score comes out, and no approximation was made — it is an exact identity.
The frequency ladder: theta_i = base^(-2i/d)
One rotation frequency would encode position modulo one wavelength and nothing else. RoPE therefore runs d/2 pairs at geometrically spaced frequencies. For head dimension d and pair index i = 0 … d/2 − 1:
θ_i = base^(−2i/d), wavelength_i = 2π/θ_i = 2π·base^(2i/d)
base = 10000, d = 128:
i = 0 → θ = 1.0 → wavelength ≈ 6.3 tokens
i = 16 → θ = 0.1 → wavelength ≈ 63 tokens
i = 32 → θ = 0.01 → wavelength ≈ 628 tokens
i = 63 → θ = 1.155e-4 → wavelength ≈ 54,400 tokensThe fastest pair spins a full turn every ~6 tokens and resolves fine local order; the slowest barely moves across the whole sequence, behaving like a monotone long-range signal. Together they make the offset m − n legible at every scale at once. The base is the only knob: it sets the ratio between fastest and slowest wavelength, so the ladder spans roughly 2π to 2π·base tokens. A larger base lengthens the slowest wavelength but crowds adjacent positions in the low-frequency pairs.
Full-dimension RoPE and the shapes involved
Stacking the per-pair rotations gives one block-diagonal orthogonal matrix acting on the whole head vector:
R_m = blockdiag( R(mθ_0), …, R(mθ_(d/2−1)) ) : [d, d]
q’_m = R_m q_m, k’_n = R_n k_n, score = q’_m · k’_n / sqrt(d)
⟨q’_m, k’_n⟩ = Σ_i [ (q·k)_i cos((n−m)θ_i) + (q×k)_i sin((n−m)θ_i) ]Because R_m is block diagonal, the one-pair theorem applies to every pair independently and the full score is their sum — each term relative, so the total is relative. Note that d here is the head dimension, not the model width: RoPE is applied per head, after the QKV projection reshapes to [B, H, N, d_h], and the ladder is built from d_h. Computing θ_i from d_model by mistake yields a silently wrong — but still trainable — model.
Why Q and K, and never V
RoPE touches the query and key tensors only, and the reason falls straight out of the derivation: the identity that makes positions cancel is about an inner product, and the only inner product in attention is q · k. The value vector never enters it — it is averaged with the resulting weights, out = Σ_j a_j v_j.
Rotating V would inject an absolute position into the output with no cancellation to rescue it: the same token at position 9000 would emit values pointing in a systematically different direction than at position 3. Worse, a weighted average of rotated values is not the rotation of the average, so the interference between positions is nonlinear. Keeping RoPE on the score path makes it a pure positional bias on attention rather than a distortion of the content the model carries forward.
A worked numeric example with d = 4
Take head dimension d = 4 (two pairs) and, for readable numbers, base = 100, giving θ_0 = 1.0 and θ_1 = 100^(−1/2) = 0.1. Let q = k = (1, 0 | 1, 0), query at m = 2, key at n = 1.
pair 0: q’ = (cos 2.0, sin 2.0) = (−0.4161, 0.9093)
k’ = (cos 1.0, sin 1.0) = ( 0.5403, 0.8415)
dot = −0.2248 + 0.7652 = 0.5403 = cos(1.0)
pair 1: q’ = (cos 0.2, sin 0.2) = ( 0.9801, 0.1987)
k’ = (cos 0.1, sin 0.1) = ( 0.9950, 0.0998)
dot = 0.9752 + 0.0198 = 0.9950 = cos(0.1)
score = 1.5353 (unrotated q·k = 2.0)Each pair returned exactly cos((m−n)θ_i), as the theorem predicts for this q and k. Move the pair to m = 5001, n = 5000 and every number above changes except the two dot products. Note also the scale separation: the fast pair has already decayed to 0.54 at an offset of one token, while the slow pair is still at 0.995 and needs an offset of 10 to decay as much.
Cost, and the form you actually implement
Nobody materialises the [d, d] block-diagonal matrix. The standard implementation is two elementwise multiplies against precomputed tables plus one cheap permutation:
cos_tab, sin_tab : [N_max, d_h] built once at load time
rotate_half(q) = concat( −q[d_h/2:], q[:d_h/2] )
q’ = q * cos_tab[pos] + rotate_half(q) * sin_tab[pos]
cost: ~3 FLOPs/element → 3·N·H·d_h per tensor, for Q and K
tables (fp32): 2·N_max·d_h·4 B; N_max=4096, d_h=64 → 2 MBSet that against attention’s O(N^2·d) and the projections’ O(N·d^2): RoPE is arithmetically free. What it is not free in is memory traffic. A naive implementation reads and writes the whole [B, H, N, d_h] Q and K tensors an extra time at arithmetic intensity near 1 — on a bandwidth-bound CPU, pure loss. Fuse the rotation into the epilogue of the QKV projection instead, while the vectors are still hot in registers or L1.
Precision and CPU-SLM implementation notes
The one genuinely fragile step is computing the angle m·θ_i: positions grow without bound while θ_i can be as small as 1e-4, and argument error feeds straight into the trigonometric output. In bf16, with 8 mantissa bits, a position index of 4096 is already quantised to steps of 32. Build the tables in fp32 and only then cast. Casting the results is far safer, since cos and sin live in [−1, 1] where every format has full relative precision; even so, bf16 tables carry ~4e-3 relative error per entry, accumulating over the d_h-term dot product.
Two equivalent pairing conventions are in the wild: interleaved, pairing dimensions (2i, 2i+1), and half-split, pairing (i, i + d_h/2). They differ only by a permutation of the dimension axis, but half-split makes rotate_half two contiguous block copies that vectorise cleanly under AVX2/NEON, whereas interleaved needs a shuffle. The two are not weight-compatible. During decoding, keys enter the KV cache already rotated at their absolute position, so each step rotates only the new query and key — O(d_h) per head — and cached entries are never touched again.
Pitfalls checklist
The failure modes are consistent enough to enumerate:
| Mistake | Symptom |
|---|---|
| Rotating V as well as Q and K | Quality degrades further out in the sequence |
| θ_i built from d_model, not d_head | Wrong wavelengths; trains, generalises poorly |
| Interleaved vs half-split mismatch | Fluent but incoherent output after conversion |
| Re-rotating cached keys each step | Angles double; attention collapses in a few tokens |
| Angles computed in fp16/bf16 | Quiet degradation, visible only at long positions |
| Ignoring partial rotary (rotary_pct < 1) | Some models rotate only the first fraction of dims |
Almost all of these are silent: RoPE has no learned parameters and no loss term of its own, so a broken implementation does not diverge, it just gets worse. The cheapest guard is the theorem itself — score a random q at m against k at n, then repeat at m + 1000, n + 1000. If the two numbers disagree beyond floating-point tolerance, the implementation is wrong. Four lines, instead of a wasted training run.
mθ_i, with θ_i = base^(−2i/d) giving a geometric ladder of wavelengths from ~6 tokens up to 2π·base. Because rotation matrices are orthogonal and compose by adding angles, ⟨R_m q, R_n k⟩ = q^T R_(n−m) k — an exact identity in which absolute positions cancel and only the offset survives. That is also why it touches Q and K but never V: the cancellation is a property of the inner product, and V is not in one. Implementation is two elementwise multiplies against precomputed cos/sin tables, arithmetically free but worth fusing into the QKV epilogue on bandwidth-bound CPUs. Build the angle tables in fp32, match your checkpoint’s pairing convention exactly, rotate keys once before they enter the KV cache, and verify with the shift test: the same score at (m, n) and (m+1000, n+1000).