Data quality is the one knob in a training run that shifts the scaling curve itself rather than moving you along it. Add more parameters or more FLOPs and you slide down a fixed loss curve; swap in cleaner data and the whole curve drops. The trouble is that ‘quality’ sounds like a taste judgement, and taste does not fit in a training budget. This article is about turning it into arithmetic: how to measure the quality of a corpus, how deduplication changes the token count that actually matters, how to price the trade between a cleaner token and an extra token, and what a single token is worth in units of loss — enough of a model to answer the question every small-model builder faces: clean the data, or collect more of it?
Quality is a curve shift, not a curve move
The Chinchilla-style loss law says test loss falls smoothly as a power of parameters N and training tokens D: L(N, D) ≈ E + A/N^α + B/D^β. Every term in that expression assumes the data is fixed; it describes how loss moves along one corpus. Data quality is different in kind — it changes the constants E, A, and B, and it changes what the model learns per token.
Concretely, a cleaner corpus behaves as if you had trained on more tokens than you actually did. We can capture that with a single scalar, an effective-data multiplier q, so that a run on D real tokens of a curated set lands near the loss you would expect from q · D tokens of the baseline set. Microsoft’s Phi models and HuggingFace’s FineWeb-Edu are the loud evidence: they reach quality that raw-web models need several times more tokens to match. If q > 1, curation is free scale — the rest of this article estimates q and the numbers that feed it.
Perplexity as a quality yardstick
The oldest quantitative proxy for ‘is this text good?’ is the perplexity a trusted reference model assigns to it. Take a small language model trained on known-clean text (a KenLM n-gram model or a compact neural LM), and score each document by its per-token loss PPL(x) = exp( -(1/T) Σ_t log p(x_t | x_<t) ).
Fluent, well-formed prose is predictable to such a model, so it earns low perplexity. Boilerplate, keyword spam, OCR garble, and broken markup are erratic and score high. The trick is that this gives a continuous, cheap, per-document number you can threshold or bin. A common recipe keeps documents whose reference perplexity falls in a middle band — not too high (garbage) and not too low, because pathologically low perplexity flags repetitive or templated text that a model can predict precisely because it is monotonous. Quality, quantified this way, is a distribution you can plot, not an opinion.
Classifier scores and the quality distribution
Perplexity captures fluency but not usefulness. The modern refinement is a lightweight quality classifier: train a small model (often a linear head on fastText or embedding features) to separate a ‘good’ reference set — think textbook, wiki, or human-rated educational pages — from random web text. Each document then gets a score s(x) ∈ [0, 1].
FineWeb-Edu is exactly this: an educational-quality classifier scored the whole corpus, and keeping only high-scoring pages produced a set on which small models learn markedly faster. The score distribution is the quality metric. You can read its mean, its spread, and — crucially — how much probability mass sits above whatever threshold you pick. Two corpora of identical token count can have wildly different score histograms, and that histogram, not the raw size, predicts downstream loss. This is what it means to make quality measurable: reduce a vague adjective to a scalar per document and a distribution over the corpus.
The deduplication problem
Web crawls are drowning in duplicates: the same article syndicated across hundreds of sites, boilerplate headers, mirror pages, quote chains. Left in, duplicates silently inflate your token count while adding no information. A corpus that is nominally D tokens but 40% duplicated carries only 0.6 D tokens of distinct signal — the rest is unlabeled repetition.
Why does that hurt? Because repeated data is memorized rather than generalized. Passing a model the same passage many times spends gradient steps sharpening recall of that exact string instead of learning transferable structure, and it inflates the risk of verbatim regurgitation at inference time. Deduplication is therefore not housekeeping; it is a quality operation with a measurable payoff. The question is how to detect near-duplicates at scale, since exact string match misses the 98%-identical syndicated copy that differs only in a byline — and that is where the hashing math comes in.
Near-duplicate math: Jaccard and MinHash
Represent a document as its set of overlapping word k-grams (shingles). Two documents’ similarity is the Jaccard index J(A, B) = |A ∩ B| / |A ∪ B|, which is 1 for identical shingle sets and near 0 for unrelated text. Computing it for every pair is O(n^2) and hopeless at web scale.
MinHash makes it cheap. Hash every shingle with a random permutation and keep the minimum; the probability that two documents share that minimum equals their Jaccard similarity exactly: Pr[min h(A) = min h(B)] = J(A, B). Repeat with K independent hashes and the fraction of matches estimates J with standard error about 1/√K. Bucketing these signatures with locality-sensitive hashing (LSH) then surfaces candidate pairs in near-linear time. So a whole-corpus dedupe reduces to: shingle, MinHash to a short signature, LSH-bucket, and drop documents above a Jaccard threshold like 0.8. The quality gain is the duplicate mass you removed.
Effective epochs and repetition decay
Dedup connects to a second number: how many times the model sees each token. If a corpus has duplication factor r (average copies per distinct document), then one nominal pass is really r passes over the unique content. That matters because repeated tokens have sharply diminishing value.
Empirically, the loss reduction from the k-th exposure to a token decays fast — the first pass teaches almost everything, the fourth or fifth adds little, and beyond a handful of repeats extra epochs can even hurt as the model overfits. A useful bookkeeping model treats effective tokens as D_eff = Σ_k u · f(k), where u is unique token count and f(k) ≤ 1 is a decaying weight on the k-th sighting. Undetected duplicates quietly push you up the repetition curve without your consent, which is precisely why dedup buys effective scale: it converts phantom repeats back into the first, high-value exposures the loss law actually rewards.
Quality versus quantity as a budget trade
Now the central tradeoff. You have a fixed compute budget, which caps the tokens you can train on. Do you spend an hour of pipeline time cleaning what you have, or an hour crawling more? Frame both sides in the same currency: expected loss.
Adding raw tokens moves you down the B/D^β term — real, but sublinear, since β is well under 1, so doubling tokens cuts that term by only 2^-β (roughly 20-30%). Cleaning multiplies effective tokens by q, giving the same improvement as a q× data increase but at the cost of a filter pass rather than a crawl. The trade tips toward quality whenever q from a filter exceeds the data-growth factor you could otherwise achieve — and because aggressive filtering routinely reports q of 2× to 5× on small models, quality usually wins for the compute-limited builder. The exception is the extreme low-data regime, where you are token-starved and even mediocre tokens beat none.
Filtering has a precision-recall cost
Filtering is not free, and its cost is also quantifiable. Every quality filter is a classifier with a threshold, and every threshold trades precision (fraction of kept tokens that are truly good) against recall (fraction of good tokens you keep). Crank the threshold up and you keep only pristine text but throw away swathes of usable data; loosen it and you retain volume but readmit junk.
The optimum is not maximum purity. If the top 10% by score yields a tiny, over-filtered corpus, the top 30% may win — trading slightly lower mean quality for three times the tokens, whichever combination minimizes final loss. That is an empirical sweep: train small proxies at several thresholds and read the loss. Quality and quantity are not opponents to be maximized separately but a single joint objective — you are choosing the point on the score distribution that maximizes q · D_kept’s effect on loss, not q alone.
The value of a token, quantified
Push the framing to its limit: what is one token worth? Differentiate the loss law with respect to data. From L ≈ E + B/D^β, the marginal loss reduction from one more token is -dL/dD = β · B / D^(β+1). Two facts fall out immediately. First, the value of a token shrinks as the corpus grows — the billionth token is worth far less than the millionth, decaying like D^-(β+1). Second, a quality token is worth a constant factor more than a baseline token, because raising effective data by q is identical to being q further along that curve.
So ‘the value of a token’ is not one number but a schedule: high early, falling fast, and uniformly lifted by whatever quality multiplier your pipeline achieves. Curation does not add tokens; it makes each token you already have behave like q of them at that point on the curve — which is exactly why cleaning early data beats crawling late data.
A worked estimate
Put numbers to it. Suppose a filter keeps 30% of a 100B-token web dump, leaving 30B curated tokens, and small-proxy runs show the curated set matches the loss of roughly 90B baseline tokens. Then the effective-data multiplier is q = 90 / 30 = 3: each curated token is doing the work of three raw ones.
raw dump = 100B tokens
kept (top 30%) = 30B tokens
loss-equivalent = 90B baseline tokens
quality multiplier q = 90B / 30B = 3.0x
to beat curation by crawling instead,
you would need > 3x more raw tokens (300B)
at the raw-token quality level — far more
crawl+storage+compute than one filter pass.The comparison is stark: a single classifier pass over 100B tokens is cheap next to crawling, deduping, and training on 300B. Whenever the measured q exceeds the multiplier you could realistically add by collecting more, the arithmetic says clean, do not crawl.
Why small models feel it most
All of this bites hardest for the CPU-friendly small language model. A large model has enough capacity to absorb noise and still extract signal — it can afford to see junk and route around it. A 1-to-3B model trained for CPU inference has no such slack: every parameter is precious, and gradient steps spent modelling boilerplate or memorizing duplicates are steps not spent on transferable skill.
That is the throughline of the Phi results: a small model on textbook-grade data punches far above its parameter count, because at small scale the quality multiplier q dominates the modest gains from raw size. If you are building a model to run on a laptop CPU, your leverage is overwhelmingly on the data side — a sharper quality classifier and a tighter dedupe move your loss more than another turn of the parameter crank, and they cost a filter pass rather than a bigger machine you may not have.
Pitfalls in measuring quality
The numbers mislead if you are careless. Reference leakage: if your quality classifier’s ‘good’ examples overlap your evaluation set, you will filter toward the benchmark and overstate quality — contamination masquerading as curation. Over-filtering: chasing a pristine mean can collapse diversity, leaving a model fluent on textbook prose but brittle on the messy inputs real users type. Metric myopia: low reference perplexity rewards text that a small model finds predictable, which can quietly bias a corpus toward the bland and repetitive.
The safeguard is to keep quality a distribution, not a single knob turned to maximum. Measure the score histogram, hold out a genuinely independent eval, sweep the filter threshold empirically, and validate that dedup removed copies rather than diversity. Quality is quantifiable — but only if you measure the whole distribution you are shaping, not just the mean you are proud of.