Ablation as a treatment effect
Formally, an ablation estimates a treatment effect. Let M(c) be the metric (say, validation loss) of a model trained with configuration c, and let c\{k} be the same configuration with component k removed or replaced by a neutral baseline. The quantity you want is
Δ_k = E[M(c\{k})] − E[M(c)]
The expectation matters: a single training run is one draw from a distribution over random seeds, data order, and initialization. What you can actually measure is a sample of runs on each side, so every ablation is secretly a two-sample estimation problem. Framing it this way immediately tells you what can go wrong: biased comparisons (the two sides differ in more than k), high variance (too few seeds), and over-interpretation (reading meaning into a difference smaller than the noise floor).
Matched controls: what must be held equal
The ablated and baseline runs must differ in exactly one thing, and deciding what to hold equal is the first design choice. Removing a component usually changes parameter count and step cost, so you must pick a matching rule: parameter-matched (rescale widths so both models have the same N parameters), compute-matched (equal training FLOPs C ≈ 6ND, so the cheaper model sees more tokens), or token-matched (equal D).
These give different answers. Ablating an FFN reduces FLOPs per token; token-matched comparison then handicaps the baseline, while compute-matched lets the smaller model train longer and often flatters it. Papers that disagree about whether a component “helps” frequently just used different matching rules. State the rule explicitly, and prefer compute-matched when the question is “what should I spend my budget on?” — because that is the decision the ablation is meant to inform.
Seed variance: the noise floor
Two runs of the identical configuration do not produce identical losses. Different seeds change initialization, dropout masks, and data shuffling, and small models are especially noisy. If run-to-run metric values are roughly M ~ N(μ, σ^2), then σ is your noise floor: any ablation effect smaller than about one σ is invisible to a single-run comparison.
Estimate it directly: run the baseline n times and compute the sample standard deviation
s = sqrt( Σ_i (M_i − M̄)^2 / (n − 1) )
For a 100M-parameter model on a modest corpus, seed-to-seed validation-loss spread of 0.005–0.02 nats is typical. That number is worth measuring once and reusing: it converts every later single-seed comparison into an honest statement of what you can and cannot conclude.
Standard error and how many seeds you need
Averaging n seeds shrinks the uncertainty of the mean by sqrt(n): the standard error is SE = s / sqrt(n). For a difference between two independent groups of n runs each,
SE_Δ = sqrt( s_A^2/n + s_B^2/n ) ≈ s · sqrt(2/n)
To resolve an effect of size Δ with roughly 95% confidence you need Δ ≥ 2 · SE_Δ, which rearranges to a sample-size rule:
n ≥ 8 · (s / Δ)^2
If seed noise is s = 0.01 and you care about a 0.01 loss difference, n ≥ 8 runs per arm. If you only care about effects twice the noise, n = 2 suffices. This single formula is the difference between an ablation section you can trust and one that is reading tea leaves.
A worked example, end to end
Suppose you ablate RoPE → learned absolute positions in a 60M model, with 3 seeds per arm. Validation losses:
baseline (RoPE): 3.412 3.398 3.405 → mean 3.405, s_A = 0.0070
ablated (learned): 3.431 3.446 3.425 → mean 3.434, s_B = 0.0108
Δ = 3.434 − 3.405 = 0.029
SE_Δ = sqrt(0.0070^2/3 + 0.0108^2/3) = sqrt(0.0000163 + 0.0000389) ≈ 0.0074
t = Δ / SE_Δ ≈ 3.9
With roughly 3–4 degrees of freedom, t ≈ 3.9 clears the ~3.2 threshold for p < 0.05: RoPE genuinely helps here, by about 0.03 nats ± 0.015. Note what three seeds bought you: with one seed per arm you would have reported some Δ between 0.013 and 0.048 depending on which pair of runs you happened to get — a 3.7× spread in the headline number.
Confidence intervals beat p-values
A p-value answers “is the effect distinguishable from zero?” but the engineering question is “how big is it?” Report the interval:
Δ ± t_crit · SE_Δ (t_crit ≈ 2 for large n, larger for few seeds)
An interval of 0.029 ± 0.023 says the component helps but the magnitude is uncertain by nearly its own size — a very different message from “p < 0.05, significant.” Intervals also make negative results informative: 0.002 ± 0.004 is evidence the component does roughly nothing, whereas 0.002 ± 0.040 is evidence you did not run enough seeds. In loss terms, a useful yardstick is that a 0.01-nat improvement in per-token loss corresponds to about a 1% reduction in perplexity (e^0.01 ≈ 1.01), which grounds “how much do we care?” in something legible.
Multiple comparisons: the grid inflates false positives
A real ablation section tests many components at once. If you run m independent comparisons at significance level α = 0.05, the chance that at least one is a false positive is
P(≥1 false positive) = 1 − (1 − α)^m
m = 10 → 1 − 0.95^10 ≈ 0.40
Forty percent. A ten-row ablation table run at the conventional threshold will, on average, contain a spurious “winner” or “loser” nearly half the time. The blunt fix is the Bonferroni correction — test each row at α/m — which is conservative but honest. A gentler practice: treat the grid as a screening pass, then rerun only the apparent winners with fresh seeds. An effect that survives replication on new randomness is worth believing; one that does not was probably the 40% talking.
One-at-a-time vs interactions
Standard practice ablates components one at a time from a full configuration. That measures each component’s marginal effect in the presence of everything else — which is not the same as its standalone value, because components interact. Write the metric as an additive model with an interaction term:
M(a, b) = μ + α·a + β·b + γ·a·b, a, b ∈ {0, 1}
γ = M(1,1) − M(1,0) − M(0,1) + M(0,0)
Estimating γ requires all four corners — a 2×2 factorial — not two ablations. Classic interactions in transformers: pre-norm placement changes whether warmup matters; weight decay interacts with learning-rate schedule; data deduplication changes the value of more epochs. When two ablations each look neutral but the pair matters, only the factorial design sees it. Budget for the corners you genuinely suspect interact, and say plainly that the rest are marginal effects.
Proxy-scale ablations and extrapolation risk
Nobody ablates at full scale; you ablate a cheap proxy and hope the ranking transfers. The math that justifies this is the scaling law: if both variants follow L(N) ≈ L_∞ + A·N^(−α) with similar exponents, a constant-offset advantage at small N tends to persist. The danger is when curves cross: a component that helps a 50M model can be neutral or harmful at 7B if it changes α or L_∞ rather than A.
The defensible protocol is to ablate at two or three proxy sizes (say 25M, 60M, 150M), fit the trend of Δ(N), and check its sign and slope. A gap that widens with scale is strong evidence; a gap that shrinks toward zero is a warning that you are measuring a small-model artifact. One extra proxy size roughly doubles ablation cost and multiplies the credibility of the conclusion.
Budgeting an ablation grid
Ablation cost is a simple product, and writing it down prevents wishful design. With m configurations, n seeds each, and per-run cost C_run ≈ 6ND FLOPs:
C_total = m · n · 6 · N · D
example: m = 8, n = 3, N = 60M, D = 1.2B tokens
C_total = 8 · 3 · 6 · 6×10^7 · 1.2×10^9 ≈ 1.0×10^19 FLOPs
The levers, in order of leverage: shrink N (quadratic win, since smaller models also tolerate shorter D at the compute-optimal ratio D ≈ 20N); shrink the grid m by screening with single seeds first; only then cut seeds n, because that is the lever that costs you statistical power. A common allocation is two-thirds of budget on a coarse single-seed screen and one-third on multi-seed confirmation of the top candidates.
Ablations on CPU-class SLMs
For the CPU small-language-model regime this series cares about, the arithmetic above is unusually friendly. A 25–60M parameter model trained on 0.5–1.2B tokens is ~10^17–10^19 FLOPs per run — days on a modern many-core CPU with a well-optimized stack, hours on one modest GPU. That makes properly seeded ablations affordable precisely where they are most needed, because small models are the noisiest: relative seed variance grows as models shrink, so the n ≥ 8(s/Δ)^2 rule bites hardest at SLM scale.
Two practical notes. First, fix the data order across arms when you can — pairing runs on the same shuffled stream turns the two-sample test into a lower-variance paired test, since the difference M_A,i − M_B,i cancels shared data noise. Second, evaluate on a large held-out set: evaluation noise adds σ_eval^2 ≈ Var(loss) / T_eval to your error budget, and a few million held-out tokens makes it negligible.