Training Data Filtering

The pipeline: four filters in series

Production filtering is a sequence of cheap-to-expensive stages, each removing documents so the next stage runs on less data. A typical order:

raw crawl
  -> language ID        (keep target languages)
  -> heuristic rules    (drop obvious junk, cheap)
  -> deduplication      (remove near-copies)
  -> quality classifier (keep ‘reference-like’ docs)
  -> safety / PII       (drop toxic, redact PII)
  -> curated corpus

Order matters for cost and correctness. Language ID and cheap heuristics go first because they discard the most data for the least compute. Deduplication usually precedes the quality classifier so the classifier is not scoring a thousand copies of the same document, and so its score distribution is not skewed by duplicates. The expensive, learned filters run last, on the smallest surviving set. Each arrow is a place where a threshold decides what lives and what dies.

Heuristic rules: cheap and surprisingly strong

Before any learned model, a handful of rule-based filters remove the most obvious garbage almost for free. These are the Gopher/C4-style rules: drop documents that are too short or absurdly long, have a low ratio of alphabetic characters, a high symbol-to-word ratio, too few stopwords (real prose is full of ‘the’, ‘and’, ‘of’), or excessive repetition of lines and n-grams.

Each rule encodes a cheap prior about what natural language looks like. A page that is 80% punctuation is a menu; a document where half the lines are identical is a log dump; a ‘paragraph’ with no stopwords is keyword-stuffed spam. None of these needs a GPU, all run in a single streaming pass, and together they remove a large slice of the crawl before you spend a cent on inference.

Classifier-based quality filtering

Heuristics catch obvious junk but cannot tell a thoughtful article from competent-but-empty filler. That is the job of a quality classifier. The classic recipe (GPT-3, RefinedWeb) is a binary classifier: treat a trusted corpus — Wikipedia, books, curated references — as positives, sample raw crawl as negatives, and train a fast linear model (fastText over word/char n-grams) to separate them.

The classifier never learns ‘quality’ in the abstract; it learns ‘does this document look like the reference set?’ That framing is the whole trick and its whole risk. FineWeb-Edu refined it by labelling documents for educational value with an LLM, then distilling those labels into a small classifier that scores every page 0–5 and keeps those above a threshold. The output is a scalar score s(d) per document; filtering just keeps documents with s(d) ≥ τ. Everything interesting hides in the choice of that threshold and in what your reference set silently considers ‘good.’

The math of a threshold

A fastText-style classifier outputs a probability that a document belongs to the positive (reference-like) class:

s(d) = σ(w · x_d + b),   σ(z) = 1 / (1 + exp(-z))

keep d  iff  s(d) ≥ τ

Here x_d is the document’s n-gram feature vector and τ is the retention threshold. Because the model is linear over n-gram features, its confidence tracks surface cues — vocabulary, formality, sentence structure — more than deep meaning, so a very high τ over-selects for a particular register (encyclopedic, formal) rather than genuine substance. In practice you do not pick τ by eyeballing scores; you pick the retention rate you can afford — ‘keep the top 40% by score’ — and let that fix τ. That reframes filtering as a budgeting decision, which is where yield enters.

Perplexity filtering with an n-gram model

A complementary, classifier-free signal is perplexity under a cheap language model — typically a KenLM n-gram model trained on clean text (Wikipedia, or the target domain). For a document of tokens w_1…w_N:

PPL(d) = exp( -(1/N) Σ_i log P(w_i | w_{i-n+1..i-1}) )

Low perplexity means the document looks like the clean reference distribution; high perplexity means it is surprising — which can mean gibberish, a rare language variant, heavy code, or an unusual but legitimate topic. The CCNet pipeline did exactly this, bucketing documents into head/middle/tail and keeping the head. It is unsupervised and runs on a CPU, but ‘surprising’ and ‘bad’ are not the same thing: perplexity quietly penalises dialects, jargon, and minority topics a Wikipedia-trained model finds unfamiliar, so it works best as one signal among several rather than the sole gate.

Deduplication: why copies are worse than they look

Web crawls are shockingly redundant — mirrors, reposts, boilerplate, and templated pages mean the same text appears hundreds of times. Duplicates hurt twice. First, they waste budget: a token seen a hundred times teaches far less than a hundred distinct tokens. Second, over-represented text is effectively up-weighted, so the model memorises it, increasing verbatim regurgitation and risking leakage of benchmark test sets that appear in the crawl.

Deduplication splits into exact and near. Exact removal is easy: hash each document (or each fixed-length span) and drop repeats. The hard, valuable case is near-duplication — two documents that differ only in an ad, a timestamp, or a reworded sentence — because a one-character change gives a completely different hash. Catching those at web scale, without comparing every pair, is what MinHash and LSH are built for.

Jaccard similarity and MinHash

Represent each document as a set of shingles — its distinct n-grams (say 5-word spans). Two near-duplicates share most shingles, so their overlap is measured by Jaccard similarity:

J(A, B) = |A ∩ B| / |A ∪ B|   ∈ [0, 1]

Computing J for every pair is quadratic and hopeless at billions of documents. MinHash makes it cheap. Apply a random hash to every shingle and keep the minimum value; the beautiful fact is that the probability two sets produce the same minimum equals their Jaccard similarity:

P[ min h(A) = min h(B) ] = J(A, B)

Repeat with k independent hashes and each document becomes a length-k signature of small integers. The fraction of matching positions is an unbiased estimate of J, with standard error about 1/√k. A 128-permutation signature compresses a whole document to 128 numbers while preserving similarity.

LSH banding: finding pairs without comparing all pairs

Signatures shrink each document, but you still must not compare all pairs. Locality-sensitive hashing (LSH) solves that. Split each k-element signature into b bands of r rows (k = b × r) and hash each band; two documents become candidate duplicates if they collide in any band, and only candidates are checked exactly.

The probability that a pair with Jaccard J becomes a candidate is:

P(candidate) = 1 - (1 - J^r)^b

This is an S-curve in J with a sharp threshold near (1/b)^(1/r). With k = 128 split as b = 16, r = 8, the curve turns on around J ≈ 0.69: a pair at J = 0.9 is caught with probability > 0.99, while a pair at J = 0.5 is almost always skipped. More bands catches looser duplicates at the cost of more candidate pairs to verify.

The precision / recall / yield tradeoff

Every filter is a binary decision, so score it like one. Treat ‘genuinely good document’ as the positive class. Then precision is the fraction of kept documents that really are good, and recall is the fraction of all good documents you kept. A third quantity matters just as much at scale: yield, the sheer number of tokens that survive.

These fight each other. Push a threshold up and precision rises — what you keep is cleaner — but recall and yield fall, because you also throw away good documents that merely looked borderline. Push it down and you keep more good tokens at the cost of admitting more junk. No setting maximises all three; filtering is choosing where on that surface to sit for a given corpus and model size.

A worked example: when high precision starves the model

Suppose a crawl has 100B tokens, of which 40B are genuinely good. Compare two thresholds:

The aggressive filter yields a pristine 20B-token corpus — wonderful, until your compute budget wants 50B tokens. Now you must either repeat the 20B (risking memorisation) or dip back into lower-quality data anyway. The lenient filter keeps 60B tokens at lower average quality but never starves training. The right choice depends on the ratio of compute budget to good-token supply: when good tokens are plentiful, favour precision; when they are scarce, yield wins.

Filtering for CPU-scale small models

For small models the arithmetic tilts hard toward quality. A CPU-scale model trained past the compute-optimal point — normal, because you over-train small models to make inference cheap — sees each good token many times, so the marginal value of a clean corpus is high. Small models also have less capacity to absorb noise: a large model can partition its parameters and quarantine spam, while a small one lets that noise degrade everything.

The practical upshot: for a small model, prefer a higher-precision filter and accept a smaller corpus, then reach the token count you need by repeating the good data a controlled number of times — a handful of epochs is fine; dozens invites memorisation. Filtering, deduplication, and epoch count are one coupled decision.

Pitfalls that quietly corrupt a corpus

Filtering is easy to get subtly wrong. The most damaging failure is benchmark contamination: if evaluation sets leak into training, your scores are fiction — so contamination checks must run against known benchmarks, not just within the crawl. A second trap is classifier bias: a filter trained to look like Wikipedia inherits Wikipedia’s blind spots, demoting informal, dialectal, or under-represented text and narrowing the model’s range.

Two more: over-filtering, where chasing precision strips out so much diversity the model becomes bland off the reference distribution; and double-counting the same signal — stacking a perplexity filter and a Wikipedia-based classifier that both reward the same register, compounding one bias instead of adding independent judgement. Good filtering is not a single aggressive gate. It is a series of cheap, diverse, individually-lenient filters whose thresholds are chosen against a real token budget — and then audited on samples you actually read.