Why it matters
Curation shapes model quality. Understanding shapes data ops.
Advertisement
The architecture
Dedup: near-duplicate removal.
Quality: perplexity + classifier filters.
Language classify.
Safety: PII + toxicity + illegal.
Advertisement
How it works end to end
MinHash for near-dedup.
FastText for language.
Perplexity filters.
Trained quality classifiers.
PII scrub.