NTK-aware interpolation extends a RoPE model’s context window by changing exactly one number: the rotary base. Where linear position interpolation squeezes every rotary frequency by the same factor s, NTK-aware scaling squeezes them unevenly — the highest-frequency dimensions are left untouched, the lowest-frequency ones are compressed by the full s, and everything in between is interpolated geometrically. The whole method reduces to replacing b = 10000 with b’ = b · s^(d/(d-2)). It costs no FLOPs, no memory, and no retraining. This piece derives the formula, works a numeric example, shows which dimensions it protects and which it quietly leaves out of bounds, and explains what it does and does not buy you on a CPU.
RoPE in one screen: frequencies, not positions
Rotary position embedding does not add a vector to the token embedding; it rotates pairs of channels inside Q and K by an angle proportional to the absolute position. For a head of dimension d, split the head into d/2 pairs indexed by i = 0 … d/2 - 1 and give each pair its own angular frequency:
θ_i = b^(-2i/d), b = 10000 (the RoPE "base")
angle(m, i) = m · θ_i for token at position m
λ_i = 2π / θ_i = 2π · b^(2i/d) (wavelength, in tokens)Because attention compares q_m with k_n, the rotation collapses to a function of m - n alone — the elegance of RoPE. What matters here is that a RoPE model never learns ‘position 4000’; it learns a bank of d/2 sinusoids whose wavelengths span roughly 2π tokens at i = 0 up to tens of thousands at i = d/2 - 1. Extending context is a question about frequencies, not about a length constant.
Why naive extrapolation fails
Train with a window L = 2048 and every dimension sees a bounded range of angles: dimension i sweeps 0 to L · θ_i radians. Divide by 2π for the number of complete rotations the model observed:
r_i = L / λ_i = L · θ_i / (2π)For d = 128, b = 10000, L = 2048: r_0 ≈ 326 full turns, r_32 ≈ 3.3 turns, r_63 ≈ 0.038 — the slowest dimension never completes even four percent of a circle. High-frequency dimensions have seen every phase many times, so position 8192 shows them nothing new. Low-frequency dimensions have only ever seen a thin arc of their circle, so position 8192 lands on angles that are genuinely out of distribution. That asymmetry is the entire story, and it is why raw extrapolation blows perplexity up almost immediately past the trained length.
What linear interpolation does, and overpays for
Position interpolation (PI) fixes this by rescaling position itself: m → m/s, equivalently θ_i → θ_i / s for every i. Position 8191 under a 4× extension now produces angles training already covered, so nothing is out of range. It works, and with a short fine-tune it works very well.
The bill arrives at the fast end of the spectrum. Dimension 0 had wavelength 2π ≈ 6.3 tokens; PI stretches it to 25 tokens. The model’s crispest local signal — the one distinguishing ‘the previous token’ from ‘three tokens back’ — is blurred by the same factor as the coarse, document-scale signal that actually needed help. Two neighbouring tokens once a full radian apart in that channel are now a quarter of that. PI pays for long-range validity with short-range resolution everywhere, and it is that overpayment, not interpolation itself, that NTK-aware scaling targets.
The NTK argument: don't destroy what the net can't relearn
The name comes from neural tangent kernel / Fourier-feature theory, and the borrowed result is spectral bias: a network fed a low-dimensional input learns low-frequency functions of it quickly and high-frequency functions extremely slowly, unless those high frequencies are handed over explicitly as Fourier features. RoPE’s frequency bank is that feature map for position.
Read PI through that lens and the damage is clear: uniform interpolation attenuates the high-frequency components of the positional feature map — exactly the ones the network cannot cheaply rebuild by gradient descent. So the prescription writes itself: apply the compression where the model has slack (the slow dimensions, which have barely explored their period) and leave it off where the model has none (the fast dimensions, which encode fine-grained locality and already tile their circle many times over). The only remaining question is how to spread that budget smoothly across i.
The trick: scale the base, not the positions
Here is the move that makes NTK-aware scaling a one-line patch: leave m alone, leave each θ_i alone, and change the base b:
b’ = b · s^(d/(d-2))
θ’_i = b’^(-2i/d)
= ( b · s^(d/(d-2)) )^(-2i/d)
= b^(-2i/d) · s^(-2i/(d-2))
= θ_i · s^(-2i/(d-2))The per-dimension compression factor is therefore s^(-2i/(d-2)): a geometric ramp in i, not the flat 1/s of PI. Check the endpoints. At i = 0 the exponent is 0, so θ’_0 = θ_0 — the fastest dimension is untouched and pure-extrapolates. At i = d/2 - 1 the exponent is -2(d/2 - 1)/(d - 2) = -1, so θ’ = θ/s — the slowest dimension gets full PI. The odd-looking d/(d-2) exists for exactly that reason: it is the value that makes the last dimension land on PI.
A worked example: 2048 to 8192 on a 128-dim head
Take d = 128, b = 10000, s = 4. Then b’ = 10000 · 4^(128/126) ≈ 40891 — a bump of about 4.09×, slightly more than s itself. Wavelengths in tokens:
| dim i | λ_i (original) | λ_i (PI, s=4) | λ_i (NTK-aware) | NTK factor |
|---|---|---|---|---|
| 0 | 6.3 | 25.1 | 6.3 | 1.00× |
| 16 | 62.8 | 251 | 89.3 | 1.42× |
| 32 | 628 | 2513 | 1271 | 2.02× |
| 48 | 6283 | 25133 | 18068 | 2.88× |
| 63 | 54410 | 217640 | 217640 | 4.00× |
The shape of the intervention is now visible: local-detail channels keep their original resolution, the mid-band absorbs a roughly 2× stretch, and only the document-scale channels take the full 4×. You have bought context length almost entirely out of the part of the spectrum that had capacity to spare.
Which dimensions are actually safe
Use r_i = L/λ_i as the diagnostic. Dimensions with λ_i < L completed at least one full turn in training and need no interpolation; those with λ_i > L did not, and any position past L pushes them into unseen angles. The crossover:
λ_i = L ⇒ i* = (d/2) · log_b( L / 2π )
d = 128, b = 10000, L = 2048 ⇒ i* ≈ 40 of 64So dimensions 0–40 are ‘safe’ and 41–63 must be interpolated by at least s to stay in range. NTK-aware scaling gets the ordering right but not the thresholds: it interpolates the safe dimensions slightly (harmless) and gives dimension 41 only a 4^(82/126) ≈ 2.47× stretch when it needed 4×. A smooth geometric ramp cannot match a sharp wavelength boundary, and that mismatch is the method’s real weakness.
Implementation: literally one line
Every RoPE implementation builds an inverse-frequency vector once and caches cos/sin tables from it. NTK-aware scaling edits the base feeding that vector, and nothing else:
# original: inv_freq[i] = theta_i, shape [d/2]
idx = arange(0, d, 2) / d # = 2i/d
inv_freq = 1.0 / (base ** idx)
# NTK-aware: same code, new base
base_ntk = base * s ** (d / (d - 2)) # 10000 -> ~40891 for s=4, d=128
inv_freq = 1.0 / (base_ntk ** idx)
# cos/sin tables: [L_new, d/2] each — L=8192, d=128, fp32 → ~4 MB total
t = arange(L_new)
freqs = outer(t, inv_freq) # [L_new, d/2]
cos_t, sin_t = cos(freqs), sin(freqs)No weight changes, no architecture changes, no extra tensors in the hot path. Hugging Face exposes it as config (rope_scaling = {"type": "dynamic", "factor": s}); GGUF runtimes expose a rope_freq_base you can pass at load time. That is why the technique spread in days rather than months.
The dynamic variant
Fixing s = 4 up front means even a 200-token prompt is decoded with stretched frequencies, needlessly degrading short-context quality. Dynamic NTK makes the factor track the sequence seen so far:
s_t = max(1, L_current / L_train)
b’_t = b · s_t^(d/(d-2))Below the trained length s_t = 1 and the model is bit-for-bit its original self; past it, the base ramps up smoothly. That is strictly better on short prompts, and it is the default in most inference stacks. One honest caveat: in a standard KV cache the keys are stored after rotation, so tokens embedded while s_t = 1 keep those angles even after the base has grown, and the cache ends up holding keys from several bases. It works because s_t drifts slowly, but it is an approximation, not an identity — which is why some runtimes prefer a fixed factor derived from the advertised maximum length.
Cost on a CPU SLM
The scaling itself is free. You recompute the [L, d/2] tables once — about 4 MB in fp32 for L = 8192, d = 128 — and every token afterwards costs exactly what it cost before. NTK-aware scaling never shows up in a FLOP count.
What is not free is the context you unlocked. The KV cache grows linearly: bytes = 2 · L · n_layers · n_kv_heads · d_head · sizeof(dtype). A 7B model (32 layers, 32 KV heads, width 128, fp16) costs 512 KB per token, so 8192 tokens is 4 GB — more than the quantized weights. A GQA 1.1B SLM (22 layers, 4 KV heads, d_head = 64) costs ~22 KB per token, so the same window is ~180 MB. Prefill attention is still O(N^2 · d), so 4× the context is ~16× the prompt-processing time. On CPU, cache size and prefill — not the rotary math — bound you.
Pitfalls, and the road to YaRN
Three things regularly bite. First, the effective factor is smaller than the nominal one: because mid-to-slow dimensions are under-interpolated, s = 4 does not reliably deliver 4× usable context, so practitioners dial s up (8 or 16 for a nominal 4×). Second, NTK-aware wins without fine-tuning and loses with it — once you are paying for a fine-tune anyway, plain PI usually ends up ahead. Third, base scaling is a model-level change — mixing a scaled base with an unscaled draft model, or with pre-rotated cached keys, silently corrupts positions.
YaRN is the direct answer to all three: it replaces the smooth geometric ramp with an explicit wavelength-based partition (interpolate the dimensions with λ > L, extrapolate those with λ < L, blend between), and adds an attention temperature t = 0.1 · ln(s) + 1 to correct the entropy drift that longer contexts introduce. NTK-aware scaling is the idea; YaRN is the idea done precisely.
b’ = b · s^(d/(d-2)), which is algebraically identical to compressing each frequency by s^(-2i/(d-2)): nothing at the fastest dimension, full 1/s at the slowest, geometric in between. The justification is spectral bias: high-frequency positional features are what a network learns least easily, so uniform interpolation destroys the signal most expensive to rebuild, while long-wavelength dimensions were under-trained anyway and can absorb the squeeze. It is a one-line change with no FLOP or memory cost, and it beats linear PI when you cannot fine-tune. Its limitation: a smooth ramp cannot honour a sharp λ_i vs L boundary, so dimensions just past the crossover stay slightly out of range — which is why the nominal factor over-promises, why PI catches up once fine-tuning is on the table, and why YaRN replaced the ramp with an explicit wavelength partition plus attention temperature. It also does not touch the real cost: the KV cache still grows linearly and prefill is still quadratic.