Rotating a single 2-D pair

Start with the smallest case: a two-dimensional vector x = (x_0, x_1) belonging to a token at position m. RoPE replaces it with the rotation of x by the angle mθ, where θ is a fixed constant chosen per pair (never learned):

R(mθ) = [ cos(mθ)   -sin(mθ) ]
           [ sin(mθ)    cos(mθ) ]

x’_0 = x_0·cos(mθ) − x_1·sin(mθ)
x’_1 = x_0·sin(mθ) + x_1·cos(mθ)

Two facts about rotation matrices carry the entire construction. They compose by adding angles, R(α)·R(β) = R(α+β); and they are orthogonal, so R(α)^T = R(α)^(-1) = R(−α). Orthogonality also means ||x’|| = ||x|| — RoPE never changes the norm of a query or key, only its direction. That is why it drops in without rescaling anything downstream — only the angular relationship carries the new signal.

Advertisement

The complex-exponential form

A 2-D rotation is multiplication by a unit complex number, which shortens the algebra considerably. Identify the pair (x_0, x_1) with z = x_0 + i·x_1. Then

R(mθ)·x   ≡   z · e^(i·mθ)

(a + ib)(cosφ + i sinφ) = (a cosφ − b sinφ) + i(a sinφ + b cosφ)

The real and imaginary parts of that product are exactly the two rows of the matrix form above. The other identity we need: the real dot product of two 2-vectors is the real part of one complex number times the conjugate of the other, a · b = Re[z_a · conj(z_b)]. Since conj(e^(iφ)) = e^(−iφ), conjugation flips the sign of a rotation angle — which is precisely how a difference of positions is about to fall out of a product of two absolute positions.

Advertisement

The relative-position theorem

Here is the whole reason RoPE exists. Let a query q sit at position m and a key k at position n. Rotate each by its own position and take the score:

⟨R(mθ)q, R(nθ)k⟩
  = q^T R(mθ)^T R(nθ) k
  = q^T R(−mθ) R(nθ) k     [orthogonality]
  = q^T R((n − m)θ) k           [angles add]

complex view:
  Re[ (z_q e^(imθ)) · conj(z_k e^(inθ)) ]
  = Re[ z_q conj(z_k) · e^(i(m − n)θ) ]

Both lines say the same thing: the absolute positions have cancelled, and the score is a function of q, k, and the offset m − n alone. Expanding gives (q·k)·cos((n−m)θ) + (q×k)·sin((n−m)θ), where q × k = q_1k_0 − q_0k_1 is the 2-D cross product (cos is even, so its argument’s sign is free; sin is odd, so it is not). Absolute rotations go in, a relative-position-dependent score comes out, and no approximation was made — it is an exact identity.

The frequency ladder: theta_i = base^(-2i/d)

One rotation frequency would encode position modulo one wavelength and nothing else. RoPE therefore runs d/2 pairs at geometrically spaced frequencies. For head dimension d and pair index i = 0 … d/2 − 1:

θ_i = base^(−2i/d),   wavelength_i = 2π/θ_i = 2π·base^(2i/d)

base = 10000, d = 128:
  i = 0   → θ = 1.0       → wavelength ≈ 6.3 tokens
  i = 16  → θ = 0.1       → wavelength ≈ 63 tokens
  i = 32  → θ = 0.01      → wavelength ≈ 628 tokens
  i = 63  → θ = 1.155e-4  → wavelength ≈ 54,400 tokens

The fastest pair spins a full turn every ~6 tokens and resolves fine local order; the slowest barely moves across the whole sequence, behaving like a monotone long-range signal. Together they make the offset m − n legible at every scale at once. The base is the only knob: it sets the ratio between fastest and slowest wavelength, so the ladder spans roughly 2π to 2π·base tokens. A larger base lengthens the slowest wavelength but crowds adjacent positions in the low-frequency pairs.

Full-dimension RoPE and the shapes involved

Stacking the per-pair rotations gives one block-diagonal orthogonal matrix acting on the whole head vector:

R_m = blockdiag( R(mθ_0), …, R(mθ_(d/2−1)) )   : [d, d]

q’_m = R_m q_m,   k’_n = R_n k_n,   score = q’_m · k’_n / sqrt(d)

⟨q’_m, k’_n⟩ = Σ_i [ (q·k)_i cos((n−m)θ_i) + (q×k)_i sin((n−m)θ_i) ]

Because R_m is block diagonal, the one-pair theorem applies to every pair independently and the full score is their sum — each term relative, so the total is relative. Note that d here is the head dimension, not the model width: RoPE is applied per head, after the QKV projection reshapes to [B, H, N, d_h], and the ladder is built from d_h. Computing θ_i from d_model by mistake yields a silently wrong — but still trainable — model.

Why Q and K, and never V

RoPE touches the query and key tensors only, and the reason falls straight out of the derivation: the identity that makes positions cancel is about an inner product, and the only inner product in attention is q · k. The value vector never enters it — it is averaged with the resulting weights, out = Σ_j a_j v_j.

Rotating V would inject an absolute position into the output with no cancellation to rescue it: the same token at position 9000 would emit values pointing in a systematically different direction than at position 3. Worse, a weighted average of rotated values is not the rotation of the average, so the interference between positions is nonlinear. Keeping RoPE on the score path makes it a pure positional bias on attention rather than a distortion of the content the model carries forward.

A worked numeric example with d = 4

Take head dimension d = 4 (two pairs) and, for readable numbers, base = 100, giving θ_0 = 1.0 and θ_1 = 100^(−1/2) = 0.1. Let q = k = (1, 0 | 1, 0), query at m = 2, key at n = 1.

pair 0:  q’ = (cos 2.0, sin 2.0) = (−0.4161, 0.9093)
         k’ = (cos 1.0, sin 1.0) = ( 0.5403, 0.8415)
         dot = −0.2248 + 0.7652 = 0.5403 = cos(1.0)

pair 1:  q’ = (cos 0.2, sin 0.2) = ( 0.9801, 0.1987)
         k’ = (cos 0.1, sin 0.1) = ( 0.9950, 0.0998)
         dot =  0.9752 + 0.0198 = 0.9950 = cos(0.1)

score = 1.5353          (unrotated q·k = 2.0)

Each pair returned exactly cos((m−n)θ_i), as the theorem predicts for this q and k. Move the pair to m = 5001, n = 5000 and every number above changes except the two dot products. Note also the scale separation: the fast pair has already decayed to 0.54 at an offset of one token, while the slow pair is still at 0.995 and needs an offset of 10 to decay as much.