Why architecture matters here
Bad moderation is one of two things: too aggressive (users bounce because everything is refused) or too permissive (a violation ships and the team is on the news). The architecture matters because you must tune both sides simultaneously.
A layered pipeline lets you tune. Cheap heuristics catch the trivially bad without paying for an ML call. ML classifiers handle nuance. The policy engine encodes what your product actually cares about (a health app has different rules than a coding assistant). Humans handle ambiguity.
Without layers, your only lever is a single threshold, and moving it hurts one side to help the other. With layers, you tune each independently and prove the change with metrics.
The policy taxonomy is the product
Everything downstream is typed against the category list. Classifier heads, thresholds, reviewer rubrics, per-category dashboards, the enforcement matrix, the appeal response a user reads, the numbers in a transparency report - all of them key off the same taxonomy. Get it wrong and model quality cannot rescue you, because a classifier can never be more consistent than its labels. Label noise, not model capacity, sets the ceiling on most moderation systems that plateau.
A category is well-formed when two trained reviewers, given only the definition and the item, reach the same verdict. That is the whole test, and it is cheap to run before you build anything: write the definition, hand 200 sampled items to three reviewers who cannot see each other's answers, and measure agreement. If they agree 65% of the time you do not have a category, you have a mood - and shipping it to a labelling vendor buys tens of thousands of examples with that disagreement baked in permanently.
The categories that fail this test are always the abstract ones: harmful, inappropriate, unsafe, offensive. They feel like categories because a person can nod at them, but they carry no decision procedure. A well-formed category names a specific harm and the subject it lands on, and comes with an explicit out-of-scope list. Not violence but five categories: operational instructions for committing violence, a credible threat against an identifiable person, glorification of a past violent act, depiction inside clearly framed fiction, and factual description in a news or historical register. Those five have different base rates, different evidence in the text, different severities, and different correct actions. Collapsed into one label they average into a classifier mediocre at all five and a rubric nobody can apply twice the same way.
Two more properties make a definition operational. State what the label is over - a single message, the assembled conversation, or the model output only - because a category scoped to the wrong unit is unlabellable by construction. And ship boundary examples, ten to twenty per category, drawn from the items reviewers actually argued about, with the resolution and the reason; these do more for agreement than another page of prose.
Severity tiers are not confidence
The single most common design bug in a moderation stack is one global cutoff applied to a score that mixes two independent quantities. Severity is how bad the item is if the label is true. Confidence is how sure you are that it is true. They are orthogonal, they come from different places - severity from policy, confidence from a calibrated model - and a system that multiplies them into one number can no longer distinguish a high-confidence mild violation from a low-confidence catastrophic one. Those two cases deserve opposite actions.
Assign severity to the category, once, in the policy document, and keep it fixed:
S0 note only log the verdict, take no user-visible action
S1 restrict degrade capability - drop tool access, force the
output filter on, tighten the system prompt
S2 block + explain refuse this request, tell the user which policy
and offer an appeal path
S3 block + review refuse, and enqueue for human review; repeated
S3 in a window escalates to account action
S4 block + route refuse, page the on-call safety owner, follow the
documented out-of-band procedure for this category
The decision surface is then a table, not an if-statement: (category, severity_tier, confidence_band) -> action, stored as versioned data with an owner per row, a change-approver, and a note recording what evidence justified the current setting. Two things follow immediately. A rarely-triggered S4 category can run at a permissive threshold because the cost of a false positive there is one annoyed user and the cost of a miss is unbounded; a high-volume S1 category runs strict because a percentage point of over-blocking there is measured in thousands of frustrated sessions a day. And when someone asks six months later why a particular request was refused, the answer is a row in a versioned table with a named owner rather than an archaeology expedition through classifier weights.
The architecture: every piece explained
The top strip is the decision path. Input is a user prompt. Pre-classifier runs cheap heuristics — length, regex for obvious keywords, prior-bad-actor lookups. This catches 40-60% of clear violations at microsecond cost. ML classifier scores across categories — toxicity, PII, jailbreak, self-harm, weapons — with calibrated confidence. Policy engine combines the scores with product rules: what is acceptable in your product, what is required by jurisdiction, what user tier grants what latitude.
The middle row is the response path. Model call uses a system prompt reinforced by guardrails from the classifier output. Output classifier checks the response — sometimes the model produces something the input classifier missed. Escalation queue receives cases where the classifiers and policy engine cannot decide with high confidence; these route to Human review by trained moderators with rubrics and calibration checks.
The bottom rows are the learning loop. Feedback + labels capture moderator decisions as training data; hard examples are worth 10x easy ones. Metrics + audit track precision, recall, latency, and human-review SLA per category. Policy versioning + release train ensures every rule change is reviewed, tested against a golden set, staged, and only then promoted.
End-to-end flow
End-to-end: a user asks "how do I hack into my ex's Facebook?" The pre-classifier flags "hack into" as a soft keyword. The ML classifier returns high confidence on unauthorized-access intent. The policy engine returns block-with-explanation. The model is prompted to refuse politely with resources. The output classifier verifies the refusal is well-formed. No escalation needed. Now consider "how do I hack my morning routine?" — the pre-classifier flags the same keyword; the ML classifier sees benign intent (routine, morning); the policy engine allows; the model responds helpfully. Difference: nuance, not a keyword. Every decision is logged; sampling drives audit and moderator training the next week.
Choosing a classifier per category, not per system
Teams pick one classification technology and apply it everywhere. That is a category error, because the three available technologies have envelopes that barely overlap and most taxonomies contain categories from all three regimes.
Deterministic matchers - regex, term lists, checksum-validated patterns, perceptual or exact hashes of known-bad media. Sub-millisecond, effectively free, perfectly reproducible, and trivially auditable: you can point at the rule that fired. Recall against paraphrase is close to zero and substring matching produces the classic false positives that make a term list a liability. Correct for exact identifiers, known-bad hashes, structured PII shapes where a checksum can confirm the match, and named entities you are legally required to catch. Never correct for intent.
Small fine-tuned classifiers - a distilled encoder or small decoder with a classification head. Single-digit to low-tens of milliseconds, batched cheaply enough to run on 100% of traffic, and - the property that matters most - they emit a score you can calibrate into a probability and threshold deliberately. The cost is data and inertia: thousands of labelled examples per category, a label set frozen into the weights, and policy changes that mean a retrain, an eval, and a re-derived threshold rather than a config commit.
LLM-as-judge - a general model given the policy text and asked to adjudicate against it. Hundreds of milliseconds to a couple of seconds, one to two orders of magnitude more expensive per call, and non-deterministic. What you buy is nuance and agility: the policy lives in prose, so a new category ships the day it is written with no labelled data, and the judge can weigh context a fixed-label classifier structurally cannot. What you must engineer around is that the judge is itself a model reading attacker-controlled text - present the content as clearly delimited data rather than instructions, constrain the output to a schema, and never let judge output reach an interpreter. See spotlighting for the delimiting mechanics.
Cascade them and the economics resolve: deterministic matchers gate the small classifier, and the small classifier's uncertain band gates the judge. If only 2% of traffic lands in the band where a judge adds information, its amortised cost falls fiftyfold and its latency lands on an already-suspect slice rather than the median request. Then assign per category - a hash list is the only sane answer for known-bad media, and a judge is the only thing that will ever handle "medical advice exceeding what this product is allowed to give".
Where moderation sits in the request path
Each position sees a different artifact and can afford a different latency, and no single position covers a whole taxonomy.
Input, before generation. The only thing available is the user's request, which makes this position correct for categories whose evidence is in the ask - solicitation of prohibited instructions, targeted harassment of a named person, categories with legal reporting duties. It cannot see harm the model invents unprompted. Its latency budget is whatever your time-to-first-token target can spare, which is why the practical answer is to overlap it with generation rather than serialise it; the parallel-classification and fail-open-versus-fail-closed mechanics are worked through in jailbreak defense and apply unchanged here.
Output, on the complete response. This position sees the artifact that actually reaches the user, so it is the one that finally matters for harm categories. For a non-streaming surface it is nearly free to place. For a streaming surface it costs either buffering or lateness.
Streaming partial output. Scoring token windows keeps a streaming UI alive and introduces a failure mode teams consistently miss: a classifier trained on complete texts is being asked to score truncated fragments, which is a distribution shift rather than a smaller version of the same problem. A half-emitted sentence can score far above or far below the full passage. Measure window-level precision and recall as separate metrics with their own threshold, and always re-score the assembled text at end-of-stream. Then decide explicitly whether the category tolerates retraction: once text has rendered it has been delivered, so for any category where delivery is itself the harm the only correct design is to buffer and never rely on pulling text back.
Asynchronous, after the fact. Expensive judges and expensive categories run on a sample of completed traffic. Nothing here protects the response that already shipped; the output is account-level enforcement, labels, and drift signal, so budget it as a data pipeline rather than request-path latency.
Calibration, and why one global threshold is wrong
A classifier's raw output is a score, not a probability. A sigmoid reading 0.8 does not mean 80% of items scoring 0.8 are violations, and modern networks are typically overconfident. Fit a calibration map on held-out data - Platt scaling for a well-behaved single head, isotonic regression when you have enough data and a monotone but oddly-shaped mapping - then verify it with a reliability diagram and a per-category expected-calibration-error figure. Until that is done, no cost argument about thresholds means anything, because you are multiplying costs by numbers that are not probabilities.
Once scores are calibrated, a threshold stops being a knob and becomes arithmetic: act when p(violation) * cost(miss) > (1 - p) * cost(false_block). That cost ratio is category-specific by orders of magnitude - a false block on a security-research question costs one irritated engineer, while a miss in a self-harm category carrying a duty of care is not denominated in the same units at all. A global cutoff tuned to aggregate F1 picks one cost ratio for every category at once, and it is wrong for all of them except the one dominating your traffic volume.
Report precision and recall at the operating point alongside the false-positive rate, never AUC - AUC averages over regions you will never operate in and flatters exactly the low-FPR corner where you actually live. The base-rate arithmetic explaining why a low-precision operating point can still be sound as a soft action is worked through in jailbreak defense and applies unchanged. The moderation-specific consequence is the one worth internalising: precision decides which intake a threshold feeds, not just how hard the action is. A category whose operating point yields single-digit precision has no business issuing an S2 block, but it is an excellent uncertainty-sampling feed for the review queue below - and the queue converts low precision into labels, which is the only thing that raises it. Read every threshold as a routing decision between the enforcement path and the labelling path, and the low-precision categories stop looking like failures.
Two consequences follow. Evaluate on a set stratified by category, since a pooled set is dominated by your highest-prevalence category and will hide a total regression in a rare, severe one. And re-derive every threshold after every retrain: score distributions shift even when accuracy improves, so an unchanged numeric threshold is a changed behaviour. Pin thresholds to a target quantile of recent benign traffic rather than a raw score and the shift becomes self-correcting.
The review queue: intake, prioritisation, agreement
The queue is not one stream. It has four intakes with different purposes, and if they share one budget the loudest one consumes the whole reviewer capacity within a month.
Uncertainty sampling pulls items near the decision boundary. It buys the most classifier improvement per label, and it has a structural blind spot: it only ever surfaces items the classifier already found interesting, so it can never discover a category the model is silently ignoring.
A random sample of allowed traffic is the only thing that can estimate your false-negative rate. Without it you know exactly what you caught and nothing about what you missed. It is small, it feels wasteful because most items are clean, and it is non-negotiable. For rare categories a uniform sample produces no positives at any affordable volume, so stratify or importance-sample and record the weights - an unweighted estimate off a stratified sample is a bias you will eventually report to someone as fact.
User reports and appeals are the users' view of your false positives, the error direction your own metrics see worst. Targeted sweeps are scoped, time-boxed pulls after an incident or policy change.
Within the queue, order by severity times calibrated confidence times exposure - how many people have already seen the item - with an age term so nothing starves behind a permanent stream of high-severity work.
Measure reviewer agreement continuously, not at onboarding. Inject gold items silently at around 5% of every reviewer's stream, and track both agreement-with-gold per reviewer and pairwise agreement across reviewers using a chance-corrected statistic - Cohen's kappa for pairs, Krippendorff's alpha once you have more than two raters or missing labels. When a category's alpha sits below roughly 0.7, the finding is about the category and not the reviewers: the definition is underspecified, and the fix is to rewrite it, add boundary examples drawn from the disagreements, and re-measure. Retraining people against a broken definition only produces confident inconsistency.
Double-review the S3 and S4 tiers, adjudicate disagreements, and keep them - disagreed items are simultaneously your best training data and your best evidence about which definitions are decaying. Appeals need a documented path and a stated turnaround, because a reversal is a confirmed, free false-positive label: route it into the training set and into the next threshold review for that category. Note the distinction from human-in-the-loop approval gates, which hold an agent's action before it executes; this queue reviews content decisions after the fact and its output is labels and policy changes, not per-request approvals.
Feedback loops, drift, and re-calibration
Review decisions become labels, labels become a retrain, the retrain moves the score distribution, thresholds are re-derived, and the new model generates the next batch of decisions. That loop is the only thing keeping a moderation system alive against a changing world, and it is also the main way teams poison their own.
Sampling bias is the big one. Train only on what the classifier flagged and you train on the classifier's own worldview; its blind spots stop being errors and become permanent structure, invisible to every metric you compute. The random allowed-traffic sample is the only antidote, which is why it has to be defended when someone proposes reallocating that reviewer capacity to the flagged backlog.
Label drift follows every policy change. Stamp each label with the taxonomy version, the rubric revision, and the reviewer id. When a definition changes, re-label its boundary examples first and decide explicitly whether old labels map forward; mixing taxonomy versions in one training set without a mapping teaches the model the average of two policies, which is a third policy nobody wrote.
Traffic drift arrives with a new surface, a new locale, a news cycle, or a new user cohort. Monitor the score distribution per category, not just the fire rate. A histogram sliding upward under a flat fire rate means a fixed threshold is quietly climbing the distribution and your effective policy is tightening without anyone deciding to tighten it - and the reverse is how systems silently stop enforcing.
A workable cadence: re-calibrate thresholds against fresh benign traffic monthly and immediately after any distribution alarm, retrain on a slower beat driven by label volume, and pin a regression corpus of past incidents so a retrain that reintroduces a closed gap fails the build rather than production. Before promoting a new model or threshold set, shadow it against live traffic and diff its decisions against the incumbent; that disagreement set, reviewed by humans, is the cheapest and most predictive pre-launch evaluation available, because it is exactly the population where the change will be felt.
Failure modes: language, context, and the audit obligation
Multilingual coverage degrades unevenly, which is what makes it dangerous. A classifier trained predominantly on English does not lose a uniform slice of accuracy elsewhere; it collapses in specific categories in specific languages while global recall barely moves. Report every metric per language-by-category cell and set thresholds per cell, because one aggregate number will happily hide a category at near-zero recall in a language that is 8% of your traffic. Translating to English before classification is a tempting shortcut that adds the translator's errors to yours and destroys the evidence a reviewer needs. Code-switching within a sentence and romanised transliteration are the hardest cases, so put them in the eval set deliberately rather than waiting for production to supply them.
Context dependence has an architectural fix, not a modelling one. The same string is a violation or not depending on who wrote it, to whom, and after what: a reclaimed slur among peers, a clinical discussion between practitioners, a quotation being criticised, an attack described by a security engineer. A stateless per-message classifier is structurally incapable of these calls, and no amount of training fixes a model that cannot see the deciding evidence. Score over a window with explicit role markers, pass product context - surface, user tier, declared purpose - as features, and accept that some categories are simply not decidable at the message level and must be scoped to the conversation in the taxonomy itself.
Every decision needs a record. Persist the input hash, the classifier ids and versions that voted, the per-category calibrated scores, the taxonomy version, the threshold set in force, the action taken, and the reviewer and rubric version if a human touched it. That record answers "why was this blocked" six months later, and it is the raw material for both an appeal response and a transparency report - neither of which can be reconstructed from aggregate counters. It is also a concentrated archive of the worst content on your platform, so redact at ingest, keep retention short and explicit per category, restrict access, and log the access itself; audit logging covers the tamper-evidence and WORM mechanics, DLP the redaction pipeline.