Pretraining scaling laws tell you how a model’s pretraining loss falls as you add parameters, data, and compute — a clean power law. But almost nobody ships pretraining loss. What you actually care about is how a pretrained base transfers: how well it does on a downstream task after a little fine-tuning, and how much fine-tuning data that takes. This article is about the second scaling story — the one that governs transfer. The central result is that pretraining is worth a measurable number of fine-tuning tokens — an ‘effective data transferred’ that itself scales as a power of model size — and that downstream metrics can scale differently, and far less smoothly, than the loss curve that produced them.
Pretraining loss is a proxy, not the target
A pretraining scaling law predicts cross-entropy loss L as a smooth power law in parameters N, data D, or compute C — roughly L(C) ≈ (C_0 / C)^α_C + L_∞. That curve is remarkably regular, but it measures the model’s next-token prediction on its pretraining distribution, not its usefulness on your task.
Transfer scaling asks a different question: given a base pretrained to some scale, how good is it on a downstream task, and how much task-specific data does it need? The two are linked — lower pretraining loss generally means better transfer — but the link is neither one-to-one nor equally smooth. A model can shave a few percent off pretraining loss and jump a downstream benchmark by twenty points, or move the loss and barely move the metric. Reasoning about deployment means reasoning about transfer, not just about L.
Effective data transferred: the core idea
The cleanest way to quantify transfer comes from Hernandez et al.’s Scaling Laws for Transfer (2021). Fine-tune a pretrained model on D_F tokens of task data, and ask: how many tokens would a from-scratch model of the same size have needed to reach the same loss? Call the gap the effective data transferred, D_T.
The total effective data the fine-tuned model behaves as if it saw is D_E = D_F + D_T. Pretraining, in other words, is worth D_T free fine-tuning tokens — you get them without collecting a single labeled example. In the low-data regime, where task data is scarce, D_T >> D_F: nearly all of the model’s competence is transferred, not fine-tuned. That is precisely the regime most practitioners live in, which is what makes the framing so useful.
The transfer power law
The striking empirical finding is that D_T is itself a clean power law — jointly in the fine-tuning set size and the model size:
D_T = k · (D_F)^α · (N)^β
D_T = effective data transferred (tokens)
D_F = fine-tuning dataset size (tokens)
N = non-embedding parameters
k, α, β = fitted constants (task-pair specific)For the one transfer pair they measured carefully — English text pretraining transferring to Python code — the fit was roughly α ≈ 0.18, β ≈ 0.38, and k ≈ 1.9×10^4. The exponents, not the constant, carry the insight: D_T grows only weakly with how much task data you have (α is small) but strongly with how big the pretrained model is (β is more than twice as large). Bigger base, more transfer.
Reading the exponents
Because β > α, model scale is the dominant lever on transfer. Consider what each exponent means in isolation. Doubling the fine-tuning data multiplies D_T by 2^0.18 ≈ 1.13 — a 13% gain for twice the labeling effort. Growing the model ten-fold multiplies D_T by 10^0.38 ≈ 2.4 — the pretraining becomes worth almost two-and-a-half times as many free fine-tuning tokens.
Since pretraining compute scales as C ≈ 6 N D, a larger N is also a larger pretraining-compute investment; the N^β term is why people say effective data transferred scales as a power of pretraining compute. The practical reading: if you are data-starved on your downstream task, you get far more mileage from starting with a larger, better-pretrained base than from scraping together a bit more task-specific data.
A worked example
Put numbers on it. Take a base with N = 10^9 non-embedding parameters and a modest fine-tuning set of D_F = 10^6 tokens, using the text→code constants above.
D_T = 1.9e4 × (10^6)^0.18 × (10^9)^0.38
= 1.9e4 × 10^1.08 × 10^3.42
= 1.9e4 × 12.0 × 2630
≈ 6.0 × 10^8 tokensSo this base behaves as if it had seen ~600 million fine-tuning tokens — roughly 600× the one million you actually supplied. Nearly all of its task competence is transferred. Now scale the model to N = 10^10: D_T rises to about 1.4×10^9 — the same 2.4× jump the exponent predicts. Ten times the model, not ten times the data, is what bought the extra effective data.
Why bigger models fine-tune more efficiently
The N^β term restates a fact practitioners feel constantly: larger pretrained models need less task data to hit a target. A big base has already learned transferable structure — syntax, factuality, reasoning scaffolds, representations that a new task can reuse — so fine-tuning mostly re-points existing features rather than learning them from scratch.
This inverts the intuition that bigger models are hungrier. They are hungrier during pretraining, but on the downstream task they are the opposite: sample-efficient, because the expensive learning already happened. It is also why parameter-efficient fine-tuning (LoRA, adapters, prompt tuning) works so well on large bases — you are nudging a rich feature set, not rebuilding one, so a tiny number of trainable weights and a tiny dataset suffice. The scaling law and the PEFT toolkit are two views of the same underlying transfer economy.
The high-data regime and ossification
Transfer is not free upside everywhere. The power law describes the low-data regime, where D_T dominates. As D_F grows, the from-scratch model catches up, D_T becomes a smaller share of D_E, and the pretraining advantage shrinks toward the margin.
Worse, in the very-high-data regime pretraining can actively hurt — a phenomenon Hernandez et al. call ossification. A heavily-pretrained model’s weights are set in patterns tuned to the pretraining distribution; when abundant target data would let a fresh model reach a lower loss, those frozen-in priors act like a rigid initialization the fine-tuning cannot fully escape. The base ossifies. The lesson is a regime map: pretrained bases are a huge win when task data is scarce, roughly neutral in the middle, and occasionally a liability when you have effectively unlimited in-distribution data and enough compute to train from scratch.
Downstream metrics don’t scale like loss
Even where transfer works, the metric you report rarely inherits the loss curve’s smoothness. Pretraining loss is an average log-probability over every token — a continuous quantity that responds to tiny improvements. A downstream metric like exact-match accuracy, pass@1, or BLEU is a thresholded, discrete transform of those probabilities.
Consider a task needing a five-token answer exactly right. Accuracy only moves when the model’s per-token probability crosses the point where the whole sequence becomes the most likely output. Halving the loss might shift accuracy from near-zero to substantial in one scale step, then plateau. So the same smooth power law in L can render as a flat line that suddenly lifts, or a sharp knee, in the accuracy plot. The compute-to-loss relationship is lawful; the loss-to-metric relationship is a task-specific, often nonlinear, squashing function.
Emergence, or a metric mirage?
This gap fuels the debate over emergent abilities — capabilities that appear absent in small models and present in large ones, seeming to switch on at a scale threshold. Wei et al. catalogued many such benchmarks where performance stays at chance, then jumps sharply once the model is big enough.
Schaeffer et al. offered a deflationary counter: much apparent emergence is an artifact of metric choice. Swap a harsh all-or-nothing metric (exact match) for a smooth one (token edit distance, per-token likelihood) and the ‘sudden’ jump often resolves into the same gradual improvement the loss showed all along. Both readings share the practical caution: you cannot extrapolate a downstream benchmark the way you extrapolate loss. A metric sitting at zero may be one scale-up away from lifting — or genuinely stuck — and the loss curve alone will not tell you which.
What this means for CPU-class SLMs
For small models meant to run on CPUs, transfer scaling is the whole game. You will not out-pretrain a frontier lab, so your leverage is choosing the strongest available base for its size and transferring it efficiently. The N^β term warns that a smaller base transfers less per token — so an SLM genuinely needs more task data, or a narrower task, to compensate.
That points to a concrete playbook: pick a base already strong on your domain (a code-heavy pretrain for a coding SLM), use PEFT to adapt on a modest, high-quality dataset, and evaluate on the real downstream metric rather than trusting the fine-tuning loss. Because a compact base sits closer to the high-data regime for a narrow task, you also spend less on data before hitting diminishing returns — and you sidestep ossification, which mostly threatens large bases swimming in target data.
Pitfalls when reasoning about transfer
Three traps recur. First, treating the exponents as universal: α, β, and k were fit for one transfer pair (text→code). A distant transfer — different modality, different language — has its own constants, and the more the target differs from the pretraining distribution, the less transfers. The shape generalizes; the numbers do not.
Second, optimizing pretraining loss as if it were the deliverable, then being surprised when the downstream metric barely moves — or moves in a jump you did not predict. Third, extrapolating a benchmark curve linearly: downstream metrics bend, plateau, and occasionally leap, so a straight-line forecast over-promises or under-promises depending on where the knee sits. The safe habit is to hold two models in mind at once — the smooth, extrapolable loss law, and the lumpy, task-specific metric it feeds — and never confuse one for the other.
D_T = k (D_F)^α (N)^β — free fine-tuning tokens you get without labeling anything. The exponents matter more than the constant: D_T grows weakly with task data (small α) but strongly with model size (larger β), so a larger, better-pretrained base is the biggest lever on data-scarce transfer — and larger bases are more fine-tune-data-efficient, not less. But the story has edges: in the high-data regime pretraining can ossify and hurt, and downstream metrics scale less smoothly than loss, bending and leaping where the loss curve glides. Choose the strongest base you can, adapt it efficiently, and always judge it on the real downstream metric — never on pretraining loss alone.