Position Interpolation Math

The setup: what RoPE actually encodes

Rotary Position Embedding does not add a position vector to the token embedding — it rotates it. Split each head’s query and key vector (q, k: [d], typically d = 128) into d/2 pairs. Pair i at absolute position m is rotated by m · θ_i, with frequencies on a geometric schedule:

θ_i = base^(-2i/d),   i = 0 .. d/2-1,   base = 10000
wavelength  λ_i = 2π / θ_i

i = 0   →  θ = 1.0        λ ≈ 6.3 tokens     (fastest)
i = 32  →  θ = 0.01       λ ≈ 628 tokens
i = 63  →  θ ≈ 1.16e-4   λ ≈ 54000 tokens   (slowest)

This works because a dot product of two rotated vectors depends only on the difference of their angles: q_m · k_n is a function of (m - n) · θ_i. Absolute positions in, relative distance out. Fast pairs resolve neighbours; slow pairs carry long-range order.

Why direct extrapolation breaks

Now ask what the model has actually seen. With L = 2048, pair i was only ever evaluated at angles in [0, 2048 · θ_i]. Setting that to a full turn, 2048 · θ_i = 2π gives θ_i ≈ 0.00307, around i = 40. So for the top ~24 of 64 pairs — the slow, long-range ones — the rotation never completed a single revolution during training. They have seen only a slice of the unit circle.

Feeding position 6000 to such a head asks it to evaluate cos and sin in a region of angle space that is entirely out of distribution. The learned attention-score function, well-behaved on the trained arc, is unconstrained outside it: scores swing to arbitrary values and perplexity explodes within a few hundred tokens past L. A failure of coverage, not of capacity.

The Position Interpolation trick

PI’s observation is that the problem is about the argument, not the function. If the model is reliable for angles generated by positions in [0, L], keep every angle inside that range by rescaling the index before it reaches the rotation:

s = L’ / L        (L’ = target window, L = pretrained window)

m’ = m / s      instead of  m

rotation angle for pair i:   (m / s) · θ_i

A token at position 6000 in an 8192-token prompt is treated by RoPE as if it sat at position 1500 — inside the trained interval. Positions become non-integer: 0, 0.25, 0.5, 0.75, 1.0, …. The name is literal: rather than extrapolating the position function past its domain, we interpolate between integer positions the model already understands, assuming the learned function is smooth in between. Weights, attention, and KV cache layout are untouched.

The same move, seen as a frequency change

Because the angle is a product, dividing the position is algebraically identical to dividing the frequency:

(m / s) · θ_i  =  m · (θ_i / s)

⇒  θ’_i = θ_i / s      λ’_i = s · λ_i (every wavelength stretched by s)

This second view is the more useful one, because it makes the design decision explicit: PI divides every frequency by the same s — fastest pair and slowest pair compressed identically. Nothing in the derivation forces that; it is simply the simplest thing to do, and exactly the assumption later methods revisit. Read PI as uniform frequency compression.

A worked example: 2048 → 8192

Model pretrained at L = 2048, head dimension d = 128, target L’ = 8192, so s = 4:

token at m = 6000, pair i = 0 (θ = 1.0)
  extrapolation:  angle = 6000 · 1.0  = 6000 rad   ← never trained
  interpolation:  angle = 1500 · 1.0  = 1500 rad   ← inside [0, 2048]

adjacent tokens m = 100, 101  →  m’ = 25.00, 25.25
  phase gap for pair 0:  1.00 rad  →  0.25 rad

slowest pair (i = 63):  λ = 54000  →  λ’ = 216000 tokens

Both halves of the trade are visible at once. Every angle now lies in the trained band — the win. The phase separation between neighbouring tokens has shrunk by exactly 4× — the bill.

Why interpolation is provably safer

The intuition — stay in the trained range — can be made quantitative, and the original paper does. It bounds how far the attention score, as a function of relative distance, can deviate under each regime. Under extrapolation the bound grows essentially without limit, because the basis functions are evaluated at unconstrained angles. Under interpolation the score is sampled between points where it is already pinned down, so deviation is governed by smoothness rather than distance. For the standard setting (d = 128, base = 10000) the paper reports the interpolation bound to be roughly 600× smaller — which also explains why, even with no fine-tuning, PI degrades gracefully where extrapolation collapses outright.

What you pay: high-frequency resolution

The cost lands on the fast pairs. Pair 0 has a natural wavelength of about 6.3 tokens, which is what gives attention a crisp sense of “the token immediately before this one.” At s = 4 that becomes about 25 tokens: the signal separating token n from n+1 is a quarter as strong. At s = 16 (2k → 32k), a sixteenth.

This is why PI-extended models regress slightly on short-context tasks, worst for anything needing fine local order: exact copying, code indentation, tight syntactic agreement. Reach is linear in s; local resolution is inversely linear in s. You trade them at a fixed exchange rate.

Why a short fine-tune is required

Interpolated angles are in-distribution, but their joint pattern is not: the model has never seen consecutive tokens sitting 0.25 positions apart, and its heads were tuned against the original phase spacing. A brief adaptation phase recalibrates them to the compressed geometry.

The good news is how cheap this is. The original work extended LLaMA models from 2048 to 32768 with roughly 1000 steps of ordinary next-token fine-tuning on long documents — no architectural change, no new objective — most of the recovery arriving in the first few hundred. PI’s practical significance is largely that ratio: context extension for a rounding error of the original training budget.

The implementation is three lines

In a standard rotary implementation the only thing that changes is the position tensor fed to the frequency outer product:

inv_freq = 1.0 / (base ** (torch.arange(0, d, 2).float() / d))   # [d/2]

t = torch.arange(seq_len, dtype=torch.float32)
t = t / scaling_factor            # <-- this line IS Position Interpolation

freqs = torch.outer(t, inv_freq)  # [seq_len, d/2]
emb   = torch.cat((freqs, freqs), dim=-1)   # [seq_len, d]
cos, sin = emb.cos(), emb.sin()   # cached, applied to q and k

Equivalently, scale inv_freq. Hugging Face configs expose this declaratively as rope_scaling = {"type": "linear", "factor": 4.0}. Note the float32: positions are now fractional and the largest angles run to thousands of radians, so a half-precision table loses real bits of phase. Build in fp32, cast cos/sin afterwards.

Shapes, memory, and compute

PI adds zero parameters and zero FLOPs — it is a divide on a cached table. What it costs is the longer sequence itself. Prefill attention is O(N^2 · d), so 4× the context is 16× the attention compute; projections and MLP scale linearly. The KV cache scales linearly and dominates memory:

bytes/token = 2 (K,V) × n_kv_heads × d_head × n_layers × bytes_per_elem

1.5B-class GQA model: 2 × 2 × 128 × 28 × 2 B  ≈ 28 KiB / token
  at 2048 tokens  →   56 MiB
  at 8192 tokens  →  224 MiB   (PI changes nothing here except N)

The honest accounting for “extend this model to 8k with PI”: free at the RoPE layer, 4× the cache, 16× the prefill attention term, and a small loss of local positional acuity.

Choosing the scale factor

Pick the smallest s that covers the window you actually need. An over-large factor is pure loss: s = 16 for a workload that peaks at 6k tokens buys nothing and blunts every short prompt.

And s is a property of the model, not the request: it must be identical during fine-tuning and at every inference call, because it defines the coordinate system the weights adapted to. If you extend in stages, re-fine-tune at each new s.

Pitfalls that bite

Skipping the fine-tune. Inference-only scaling beats extrapolation but lands well below a tuned checkpoint, and the gap widens with s.

Mismatched cache and query positions. A KV cache built with unscaled positions continued with scaled ones — common in chunked prefill and sliding-window paths — corrupts relative distance for every cached token.

Integer position assumptions. Code that indexes a precomputed cos/sin table by integer position, or rounds m/s, quietly destroys the interpolation.

Judging by perplexity alone. Perplexity is dominated by local prediction and stays respectable long after long-range retrieval has failed. Pair it with a positional probe at several depths across the full window.

PI on a CPU small language model

On CPU-class deployments the binding constraint is memory bandwidth, not FLOPs. PI is free; the window it enables is not. A 4× larger KV cache means 4× more bytes streamed from RAM per generated token, which on a machine with no HBM is close to 4× slower decoding, and quadratic prefill bites harder still — an 8k prompt on a laptop core is seconds of wall clock before the first token.

So pair PI with what attacks the cache: grouped-query attention, KV quantization to int8 or int4, honest windowing. PI buys the ability to attend across 8k tokens; whether you can afford to is a separate, bandwidth-shaped question.

Where PI sits in the lineage

Position Interpolation is the baseline every later context-extension method is measured against, and it earned that by being almost embarrassingly simple. Its one strong assumption is uniformity — that all d/2 frequencies should be compressed equally. That is where value is left on the table: the fast pairs were already comfortably in distribution and never needed compressing; only the slow pairs, the ones that never completed a revolution during pretraining, were ever the problem. Every serious successor is a rebuttal of that single line — while sharing PI’s skeleton: keep the rotation angles inside the trained band, and pay in phase resolution. PI just pays uniformly.