CONTENT FILTER · THRESHOLD AND CONTEXT
Content filter threshold: a quoted phrase versus a harmful request
Score twelve messages with a fictional harm classifier and see why one threshold blocks reporters and novelists while still missing a politely worded harmful request.
Your experiment
Seven allowed messages, five of which quote or report risky words, and five harmful requests, one of them politely worded with no risky words at all. A single classifier threshold of 0.5 blocks all five quoting messages and still lets the polite request through. Move the threshold both ways, then try the quote discount, a review band and the intent check.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Messages are summarised, never quoted, and their scores come from a fictional classifier. The quote discount applies only to messages that quote or report words without asking for a harmful act; the intent check is modeled as perfect at spotting explicit requests to harm a person, which a real classifier is not. Review capacity is 2 of 12 messages.
Twelve messages, two kinds of mistake
A content filter can wrongly block an allowed message or let a harmful request through. The fixture has seven allowed messages, five of which quote or report risky words (a news report, a moderator’s question, a novelist’s draft, a harassment victim asking how to report, a history question), and five harmful requests. Filter rule chooses a block-list keyword match, a fictional classifier’s harm score compared with Block threshold, or the score with Quote discount (context rule) subtracted for messages that quote words without asking for harm.
The opening run: aggressive and still leaky
With the score rule at 0.5, all five quoting messages score between 0.55 and 0.82 and are blocked, while the politely worded request to track an ex-partner’s location (M9) scores 0.38 and is allowed: 5 false blocks and 1 missed harm. Raise Block threshold to 0.8 and only the news report is still blocked, but all five harmful requests pass. Lower it to 0.35 and M9 is caught at the price of 7 false blocks. No threshold separates M9 from the quoting messages, because it scores lower than every one of them.
Keywords and context
Set Filter rule to Block-list keyword: messages with listed words are blocked whatever their purpose, including the recipe tip about killing the heat, so 5 allowed messages are blocked and 3 harmful requests without listed words pass. Set it to Score with quote discount: with the default 0.3 discount only the news report (0.82 − 0.3 = 0.52) is still blocked; a discount of 0.35 clears it. Requests are never discounted, so the film-line request M10 is still blocked at 0.77, but M9 is still missed.
Failure boundary: review band and intent
Add a Human review band below the threshold of 0.15 with the 0.35 discount: M9 lands in review, but so do five allowed messages, 6 of 12 against a review capacity of 2, so the verdict stays ERRORS. Tick Separate intent check for harmful requests instead: every explicit request to harm a person is blocked regardless of its score, and the context rule with a 0.35 discount gives 0 false blocks and 0 misses. With the intent check on, lowering the threshold to 0.45 blocks two quoting messages again; a score threshold should be tuned for content, and intent handled by its own check.
Reading the run and its limits
Play reveals the decisions in four batches and the outcome over sixty inspection seconds; the table lists every message with its score, the score used after any discount, the decision and the error type. Next event, Restart, Speed and Inspection time control the replay, and any input change recomputes it. Real classifiers are noisy, an intent check also makes mistakes, and attackers adapt to whatever rule is published, so a fixture like this shows the shape of the trade-off, not the right production threshold.