RELEASE GATE · WHICH ERROR MATTERS
Moderation release gate: block on the wrong error type
Gate a new moderation policy on its eval results and see an accuracy rule ship a candidate that misses four times as many severe cases.
Your experiment
Candidate B raises the blocking threshold: it blocks far fewer benign messages but catches 36 of 40 severe cases where the current policy A catches 39. The gate ships any candidate whose accuracy is not below A’s. Compare the decision with the production consequence, then gate on severe recall, change the eval set and try candidate C.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Fictional per-class results for three policies; production prevalence is a fictional 0.5% severe, 4.5% mild, 95% benign. Production consequences use each policy’s rates on the balanced set. The right decision is defined as shipping only a candidate that does not increase severe misses. No classifier or model is run.
Three policies and two eval sets
The current moderation policy A and two candidates have fixed results per class. On the balanced eval set (40 severe, 40 mild, 40 benign) A catches 39 severe and wrongly blocks 8 benign messages; candidate B, a higher threshold, catches 36 severe and blocks only 2 benign; candidate C adds a severe-harm rule to B and catches all 40 severe while blocking 3 benign. Candidate policy picks B or C, Eval set picks the balanced set or a 200-message production-mix sample, and Gate rule decides what has to improve.
The opening run: accuracy ships the wrong policy
With Gate rule set to Accuracy not below current, B scores 90% against A’s 84.2% and the gate ships it. Per 100,000 production messages, B falsely blocks 4,750 instead of 19,000 benign messages, but misses 50 severe cases instead of 12.5: four times as many. Accuracy counts a missed severe case and a wrongly blocked benign one as the same error, so the large drop in false blocks hides the loss that matters most.
Gate on the error that matters
Set Gate rule to Severe recall above a floor: with Minimum severe recall at 95%, B’s 90% fails and the gate correctly blocks it. Lower the floor to 90% or 85% and B ships again, so the floor must be at least the current policy’s 97.5% if severe misses may not rise. Choose candidate C: its severe recall is 100% and its accuracy 91.7%, so both rules ship it, and production severe misses fall to 0 while false blocks drop from 19,000 to 7,125.
Failure boundary: a set too small to measure
Set Eval set to Production mix with the severe-recall rule. The 200-message sample contains one severe case, which B happens to catch, so its measured severe recall is 100% and the gate ships B again: the wrong decision, with a correct-looking number. Accuracy on that set also favours B (94.5% against 80%), because 190 of 200 cases are benign. A gate for a rare, severe class needs enough examples of that class, so oversample it rather than mirroring production frequencies.
Reading the run and its limits
Play replays the eval set, A’s and the candidate’s scores, the gate verdict, the production consequence and the outcome over sixty inspection seconds; the table shows per-class counts, accuracy and severe recall. Next event, Restart, Speed and Inspection time control the replay, and any input change recomputes it. The numbers are fictional, the production rates are taken from the balanced set, and a real gate would add confidence intervals and per-category floors rather than one severe class.