Ablation Study Math

Ablation as a treatment effect

Formally, an ablation estimates a treatment effect. Let M(c) be the metric (say, validation loss) of a model trained with configuration c, and let c\{k} be the same configuration with component k removed or replaced by a neutral baseline. The quantity you want is

Δ_k = E[M(c\{k})] − E[M(c)]

The expectation matters: a single training run is one draw from a distribution over random seeds, data order, and initialization. What you can actually measure is a sample of runs on each side, so every ablation is secretly a two-sample estimation problem. Framing it this way immediately tells you what can go wrong: biased comparisons (the two sides differ in more than k), high variance (too few seeds), and over-interpretation (reading meaning into a difference smaller than the noise floor).

Matched controls: what must be held equal

The ablated and baseline runs must differ in exactly one thing, and deciding what to hold equal is the first design choice. Removing a component usually changes parameter count and step cost, so you must pick a matching rule: parameter-matched (rescale widths so both models have the same N parameters), compute-matched (equal training FLOPs C ≈ 6ND, so the cheaper model sees more tokens), or token-matched (equal D).

These give different answers. Ablating an FFN reduces FLOPs per token; token-matched comparison then handicaps the baseline, while compute-matched lets the smaller model train longer and often flatters it. Papers that disagree about whether a component “helps” frequently just used different matching rules. State the rule explicitly, and prefer compute-matched when the question is “what should I spend my budget on?” — because that is the decision the ablation is meant to inform.

Seed variance: the noise floor

Two runs of the identical configuration do not produce identical losses. Different seeds change initialization, dropout masks, and data shuffling, and small models are especially noisy. If run-to-run metric values are roughly M ~ N(μ, σ^2), then σ is your noise floor: any ablation effect smaller than about one σ is invisible to a single-run comparison.

Estimate it directly: run the baseline n times and compute the sample standard deviation

s = sqrt( Σ_i (M_i − M̄)^2 / (n − 1) )

For a 100M-parameter model on a modest corpus, seed-to-seed validation-loss spread of 0.005–0.02 nats is typical. That number is worth measuring once and reusing: it converts every later single-seed comparison into an honest statement of what you can and cannot conclude.

Standard error and how many seeds you need

Averaging n seeds shrinks the uncertainty of the mean by sqrt(n): the standard error is SE = s / sqrt(n). For a difference between two independent groups of n runs each,

SE_Δ = sqrt( s_A^2/n + s_B^2/n ) ≈ s · sqrt(2/n)

To resolve an effect of size Δ with roughly 95% confidence you need Δ ≥ 2 · SE_Δ, which rearranges to a sample-size rule:

n ≥ 8 · (s / Δ)^2

If seed noise is s = 0.01 and you care about a 0.01 loss difference, n ≥ 8 runs per arm. If you only care about effects twice the noise, n = 2 suffices. This single formula is the difference between an ablation section you can trust and one that is reading tea leaves.

A worked example, end to end

Suppose you ablate RoPE → learned absolute positions in a 60M model, with 3 seeds per arm. Validation losses:

baseline (RoPE):   3.412  3.398  3.405   → mean 3.405, s_A = 0.0070
ablated (learned): 3.431  3.446  3.425   → mean 3.434, s_B = 0.0108
Δ = 3.434 − 3.405 = 0.029
SE_Δ = sqrt(0.0070^2/3 + 0.0108^2/3) = sqrt(0.0000163 + 0.0000389) ≈ 0.0074
t = Δ / SE_Δ ≈ 3.9

With roughly 3–4 degrees of freedom, t ≈ 3.9 clears the ~3.2 threshold for p < 0.05: RoPE genuinely helps here, by about 0.03 nats ± 0.015. Note what three seeds bought you: with one seed per arm you would have reported some Δ between 0.013 and 0.048 depending on which pair of runs you happened to get — a 3.7× spread in the headline number.

Confidence intervals beat p-values

A p-value answers “is the effect distinguishable from zero?” but the engineering question is “how big is it?” Report the interval:

Δ ± t_crit · SE_Δ   (t_crit ≈ 2 for large n, larger for few seeds)

An interval of 0.029 ± 0.023 says the component helps but the magnitude is uncertain by nearly its own size — a very different message from “p < 0.05, significant.” Intervals also make negative results informative: 0.002 ± 0.004 is evidence the component does roughly nothing, whereas 0.002 ± 0.040 is evidence you did not run enough seeds. In loss terms, a useful yardstick is that a 0.01-nat improvement in per-token loss corresponds to about a 1% reduction in perplexity (e^0.01 ≈ 1.01), which grounds “how much do we care?” in something legible.

Multiple comparisons: the grid inflates false positives

A real ablation section tests many components at once. If you run m independent comparisons at significance level α = 0.05, the chance that at least one is a false positive is

P(≥1 false positive) = 1 − (1 − α)^m
m = 10  →  1 − 0.95^10 ≈ 0.40

Forty percent. A ten-row ablation table run at the conventional threshold will, on average, contain a spurious “winner” or “loser” nearly half the time. The blunt fix is the Bonferroni correction — test each row at α/m — which is conservative but honest. A gentler practice: treat the grid as a screening pass, then rerun only the apparent winners with fresh seeds. An effect that survives replication on new randomness is worth believing; one that does not was probably the 40% talking.

One-at-a-time vs interactions

Standard practice ablates components one at a time from a full configuration. That measures each component’s marginal effect in the presence of everything else — which is not the same as its standalone value, because components interact. Write the metric as an additive model with an interaction term:

M(a, b) = μ + α·a + β·b + γ·a·b,   a, b ∈ {0, 1}
γ = M(1,1) − M(1,0) − M(0,1) + M(0,0)

Estimating γ requires all four corners — a 2×2 factorial — not two ablations. Classic interactions in transformers: pre-norm placement changes whether warmup matters; weight decay interacts with learning-rate schedule; data deduplication changes the value of more epochs. When two ablations each look neutral but the pair matters, only the factorial design sees it. Budget for the corners you genuinely suspect interact, and say plainly that the rest are marginal effects.

Proxy-scale ablations and extrapolation risk

Nobody ablates at full scale; you ablate a cheap proxy and hope the ranking transfers. The math that justifies this is the scaling law: if both variants follow L(N) ≈ L_∞ + A·N^(−α) with similar exponents, a constant-offset advantage at small N tends to persist. The danger is when curves cross: a component that helps a 50M model can be neutral or harmful at 7B if it changes α or L_∞ rather than A.

The defensible protocol is to ablate at two or three proxy sizes (say 25M, 60M, 150M), fit the trend of Δ(N), and check its sign and slope. A gap that widens with scale is strong evidence; a gap that shrinks toward zero is a warning that you are measuring a small-model artifact. One extra proxy size roughly doubles ablation cost and multiplies the credibility of the conclusion.

Budgeting an ablation grid

Ablation cost is a simple product, and writing it down prevents wishful design. With m configurations, n seeds each, and per-run cost C_run ≈ 6ND FLOPs:

C_total = m · n · 6 · N · D
example: m = 8, n = 3, N = 60M, D = 1.2B tokens
C_total = 8 · 3 · 6 · 6×10^7 · 1.2×10^9 ≈ 1.0×10^19 FLOPs

The levers, in order of leverage: shrink N (quadratic win, since smaller models also tolerate shorter D at the compute-optimal ratio D ≈ 20N); shrink the grid m by screening with single seeds first; only then cut seeds n, because that is the lever that costs you statistical power. A common allocation is two-thirds of budget on a coarse single-seed screen and one-third on multi-seed confirmation of the top candidates.

Ablations on CPU-class SLMs

For the CPU small-language-model regime this series cares about, the arithmetic above is unusually friendly. A 25–60M parameter model trained on 0.5–1.2B tokens is ~10^17–10^19 FLOPs per run — days on a modern many-core CPU with a well-optimized stack, hours on one modest GPU. That makes properly seeded ablations affordable precisely where they are most needed, because small models are the noisiest: relative seed variance grows as models shrink, so the n ≥ 8(s/Δ)^2 rule bites hardest at SLM scale.

Two practical notes. First, fix the data order across arms when you can — pairing runs on the same shuffled stream turns the two-sample test into a lower-variance paired test, since the difference M_A,i − M_B,i cancels shared data noise. Second, evaluate on a large held-out set: evaluation noise adds σ_eval^2 ≈ Var(loss) / T_eval to your error budget, and a few million held-out tokens makes it negligible.

Choosing the metric: loss vs downstream

Validation loss is the right primary metric for ablations: it is smooth, low-variance, and every token contributes signal. Benchmark accuracies are the opposite — a 1000-question task has a binomial standard error of

SE_acc = sqrt( p(1 − p) / Q ) = sqrt(0.5 · 0.5 / 1000) ≈ 1.6 percentage points

so accuracy differences under ~3 points on a single benchmark are consistent with pure chance, regardless of seeds. Worse, small models sit near chance on hard benchmarks, where the metric is almost all noise. The workable pattern: rank and select on loss, then confirm the final choice on an aggregate of several benchmarks, which averages down the binomial noise. If loss and downstream metrics disagree persistently, that itself is a finding — usually a sign the ablated component affects formatting or in-context behavior rather than raw prediction quality.

Common pitfalls

The failure modes are predictable enough to checklist. Untuned comparisons: the baseline was tuned for months and the ablated variant inherits its hyperparameters; if the optimal learning rate shifts when a component is removed, you are measuring mistuning, not the component. Re-tune at least the learning rate per arm. Peeking: comparing curves mid-training and stopping when your favorite is ahead is a sequential-testing bias; fix the token budget in advance. Cherry-picked checkpoints: report the final or averaged checkpoint, not the best-ever eval. Silent confounds: a changed component that alters throughput changes how many tokens fit your wall-clock budget — a compute confound hiding inside an infrastructure detail. Survivorship in the grid: reporting only ablations that produced clean stories. Each of these has appeared in published work; none survives the discipline of pre-registered matching rules, fixed budgets, and reported intervals.

Reading other people's ablation tables

The same math turns you into a better reader. When a paper shows an ablation table, ask four questions. How many seeds? If unstated, assume one, and mentally add ±σ error bars to every row. What was matched? Parameter-, compute-, and token-matching answer different questions; a row that flips under a different matching rule is fragile. How many rows? Ten rows at α = 0.05 means a ~40% chance the table contains at least one artifact. What scale? A 100M-parameter ablation justifying a 70B design decision is an extrapolation, and the paper owes you evidence the gap persists across at least two proxy sizes.

Tables that survive all four questions are rare and precious. Tables that fail them are not worthless — they are screening evidence, the first pass of the protocol above — but they license a hypothesis, not a conclusion. Calibrating that distinction is most of what “ablation study math” buys you.