Scaling laws are usually quoted as if only two knobs exist: how big the model is and how many tokens it sees. But there is a third knob hiding inside the constants — how good the tokens are. Improving data quality through filtering, deduplication, and curation does not move you along the familiar loss-vs-compute curve; it shifts the whole curve, down and to the left, so you reach the same loss with fewer tokens and less compute. This piece treats quality as an effective-data multiplier, works a concrete example, and walks the evidence — the Phi ‘textbooks’ models, deduplication studies, and classifier-filtered corpora like FineWeb-Edu and DCLM. It is deliberately the counterpart to raw data scaling: that story is about quantity; this one is about the value packed into each token.
The curve everyone quotes hides a variable
The compute-optimal scaling law says test loss falls as a power of the compute you spend: roughly L(C) ≈ E + (C_c / C)^α_C, with a matching form in model size N and token count D, L(N, D) = E + A/N^α + B/D^β. The Chinchilla fit puts α ≈ 0.34 and β ≈ 0.28, and the E term is the irreducible entropy of the data. Every one of those constants — A, B, C_c — is measured on a particular corpus. Change the data distribution and you re-fit the constants.
That is the whole point. Adding more tokens moves you along a fixed curve toward its floor. Improving the data redraws the curve: the B/D^β term shrinks for every token count, so the same compute buys a lower loss. Quantity walks down the slope you were handed; quality hands you a better slope. Treating data quality as a fixed background assumption — the mistake baked into a bare reading of the law — hides the single highest-leverage lever a small team actually controls.
Quality as an effective-data multiplier
The cleanest way to reason about curation is to fold it into a single number. Say a curated corpus is worth q times a baseline web corpus token-for-token, so the model behaves as if it trained on D_eff = q · D tokens. The data-limited loss term becomes B / (q · D)^β. To hit a target loss you now need only D = D_eff / q real tokens, and since training compute is C ≈ 6 · N · D at fixed model size N, your compute bill falls by roughly the same factor q.
target data-term: B / D_eff^β (D_eff = effective tokens)
quality q: D_eff = q · D (q > 1 for better-than-web data)
real tokens: D = D_eff / q
compute (fixed N): C ≈ 6 · N · D → drops ~q×
worked: need D_eff = 300B effective tokens for the target loss.
baseline web, q = 1.0 → D = 300B raw tokens
curated data, q = 2.5 → D = 120B raw tokens (2.5× less compute)The multiplier is an approximation — q is not constant across the whole curve and it saturates — but it captures the mechanism exactly: better data is indistinguishable, to the loss curve, from simply having more of the baseline data.
Textbooks are all you need: the Phi evidence
The sharpest demonstration is Microsoft’s Phi line. phi-1 (1.3B parameters) was trained on only about 7B tokens of ‘textbook-quality’ data — a filtered slice of web code plus model-generated textbook and exercise text — and reached roughly 50% pass@1 on HumanEval, rivaling models trained on one to two orders of magnitude more tokens. phi-1.5, phi-2, and phi-3 pushed the same thesis into general reasoning: a small model on curated data punches far above its parameter and token class.
In effective-data-multiplier terms, the Phi recipe is a large q. The authors argue that most web text is a poor teacher — boilerplate, noise, and low-density prose — while text that is self-contained, correct, and pedagogically ordered delivers far more signal per token. The provocative title ‘textbooks are all you need’ overstates it (you still need scale and breadth), but the empirical point stands: curation and targeted synthetic generation moved the curve enough that a sub-2B model did what much larger models did, at a fraction of the training compute.
Deduplication: the cheapest multiplier
Before any clever filtering, the highest return-on-effort quality win is removing duplicates. Large web scrapes contain enormous redundancy — boilerplate, mirrored pages, near-identical documents. Lee et al. (2021) showed that deduplicating training data makes language models better: models trained on de-duplicated data reach the same or better accuracy in fewer steps, emit far less memorized training text verbatim, and need fewer parameters to match a baseline.
The mechanism fits the multiplier picture. A document repeated 40 times does not give you 40 tokens’ worth of signal; it gives you one document’s signal plus 39 wasted passes that push the model toward memorization instead of generalization. Deduplication — exact-match plus near-duplicate detection via MinHash / LSH on n-gram shingles — raises the unique-information density of the corpus, so every token the optimizer sees carries more novelty. It costs a one-time preprocessing pass and no training compute, yet it can meaningfully lift q. It is the first thing any serious data pipeline does, and the easiest to under-rate.
Filtering and curation: FineWeb-Edu and DCLM
Beyond dedup, the modern lever is classifier-based filtering: train a lightweight model to score how ‘valuable’ a document is, then keep the high-scoring tail. FineWeb-Edu applies an educational-quality classifier to Common Crawl, keeping text that looks like it teaches something. Models trained on it reach a given accuracy on knowledge and reasoning benchmarks (MMLU, ARC) with roughly an order of magnitude fewer tokens than raw Common Crawl — a large effective-data multiplier, bought purely by selection.
DCLM (DataComp-LM) turned this into a controlled competition: fix the model and compute, vary only the data pipeline, and measure. Its best filtered subset, DCLM-Baseline, trained competitive models with several times less compute than weaker-filtered baselines, confirming that the data-processing recipe — not just token count — is a first-class scaling variable. The takeaway across both: a good quality classifier plus aggressive deduplication is, in effect, a compute multiplier you apply once, offline, before a single gradient step. (Exact multipliers vary by benchmark and setup; treat the figures as approximate.)
The quantity-quality trade under a fixed budget
Filtering has a cost: it shrinks the pool. Keep only the top 10% of documents and you raise average quality but throw away 90% of your unique tokens. Under a fixed training-token budget this creates a genuine trade. If you filter so hard that the surviving corpus is smaller than your budget, you must repeat it for several epochs — and repetition has sharply diminishing returns, the subject of the companion data scaling article, which is about quantity and token repetition rather than per-token value.
So there is an optimum. Too little filtering and each token is weak (q low); too much filtering and you run out of unique tokens and burn the budget re-reading the same shrunken set (effective q collapses as repetition dominates). The quality frontier is the threshold that maximizes total useful signal = unique-token count × per-token value, subject to your compute. This is precisely why the two articles are complementary: quantity scaling moves along the curve; quality scaling shifts it — and the token budget is where the two meet.
Pitfalls: over-filtering, contamination, collapse
The multiplier is not free money, and three failure modes bite. Over-filtering kills diversity. A classifier that rewards ‘textbook-like’ text will quietly delete valid dialects, informal registers, and long-tail domains, hurting robustness and tail performance even as headline benchmarks rise. Quality is multi-dimensional; a single scalar score is a lossy proxy.
Benchmark contamination is the second trap: a filter that selects for text resembling the eval set inflates scores without real capability, so any curated corpus must be decontaminated against the benchmarks before you trust the numbers. Third, model collapse: leaning on synthetic, model-generated data — a key ingredient in the Phi recipe — degrades quality if fed back recursively, because errors and low-variance patterns compound (Shumailov et al.). The fix is grounding synthetic generation in real seeds, enforcing diversity, and verifying correctness. Read together, these say the effective-data multiplier is real but bounded: curation can shift the curve substantially, but it cannot replace scale, and pushed carelessly it degrades the very quality it chases.
What it means for CPU-class small models
For small language models — the ones meant to run on a CPU or at the edge, where parameter count is capped by memory and latency — data quality is not one lever among many; it is the lever. You cannot out-scale a bigger model on parameters or on raw compute, so the only axis left is packing more capability into the budget you have, and that is exactly what a high q buys: the same loss at fewer tokens, or a better model at the same tokens.
The practical playbook falls straight out of the sections above. Deduplicate aggressively (free multiplier). Filter with a quality classifier tuned to your target domain, but stop before you starve the corpus of unique tokens. Add carefully grounded synthetic data to cover gaps, and decontaminate against your evals. A 1–3B model trained this way on a curated, in-domain corpus routinely beats a generic 7B on the task you care about — not because scaling laws were broken, but because you moved to a better curve before you started climbing it.
q: the model behaves as if it saw q · D tokens, so you reach the same loss with fewer real tokens and roughly q times less compute — the whole curve shifts down and left rather than you merely sliding along it. The Phi ‘textbooks’ models, deduplication studies, and FineWeb-Edu / DCLM all show large multipliers from selection alone. But q saturates and can backfire: over-filtering kills diversity, quality filters can smuggle in benchmark contamination, and recursive synthetic data invites model collapse. The discipline is to raise quality until unique tokens run short, then stop. For CPU-class small models that cannot out-scale anyone, this is the single highest-leverage move — get onto a better curve before you spend a token climbing it.