SCORING · THRESHOLD AND ERROR COST
Agent scoring: threshold and error cost
An agent auto-approves refund requests below a risk score; sweep the threshold and compare the most accurate setting with the cheapest one.
Your experiment
The evaluation set has 13 legitimate and 7 fraudulent refund requests with risk scores. Approving fraud costs $200 and a manual review of a legitimate request costs $15; production fraud runs at 10%. Start at 0.65, the most accurate threshold, and look for the cheapest one; then change the costs and the fraud rate.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, Speed changes the pace and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Expected cost per 1,000 production requests = 1,000 × (fraud rate × share of fraud approved × fraud cost + (1 − fraud rate) × share of legitimate reviewed × review cost), with shares measured on the 20 labelled cases. Twenty cases give a coarse estimate; the cost structure, not the exact numbers, is the lesson. Scores and costs are fictional.
The most accurate threshold is not the cheapest
An agent approves a refund request automatically when its risk score is below a threshold and sends the rest to manual review. On 13 legitimate and 7 fraudulent labelled requests, 0.65 is the most accurate threshold, 90%: it approves 2 fraudulent requests and reviews no legitimate ones. With fraud costing $200, a review $15 and production fraud at 10%, that costs $5,714 per 1,000 requests. The cheapest threshold is 0.5 at $4,934, which approves 1 fraudulent request and reviews 2 legitimate ones at 85% accuracy: COSTLY THRESHOLD by $780.
Read the curve
The chart draws expected cost against every threshold from 0.05 to 1 as the inspection runs, with dashed lines at the chosen and the cheapest threshold. Approve when the risk score is below 1 approves everything, $20,000 per 1,000; 0.05 reviews everything, $13,500. Move the threshold to 0.5 and the result reads NEAR-OPTIMAL THRESHOLD; 0.35 reviews five legitimate requests and costs $5,192.
Costs move the optimum
Raise Cost of approving a fraudulent request to $500 and the cheapest threshold drops to 0.35, while 0.65 now costs $14,286. Lower it to $50 and 0.65 becomes the cheapest. Cost of reviewing a legitimate request pulls the other way: at $50 per review 0.65 is cheapest, at $5 the optimum is 0.35 at $1,731. Accuracy treats both mistakes as equal; the business rarely does.
Production is not the evaluation set
Fraud rate in production reweights the two error types. At 40% fraud the cheapest threshold is 0.35 at $3,462 and 0.65 costs $22,857; at 5% fraud 0.65 is the cheapest at $2,857. A threshold tuned on an evaluation set with a different fraud rate, or before fraudsters adapt, has to be re-derived from current rates and costs.
Reading the run and its limits
Play, Next event, Restart, Speed and Inspection time sweep the thresholds, then report the chosen, most accurate and cheapest settings across sixty seconds; the table lists every threshold with its errors, accuracy and cost. Twenty labelled cases give a coarse estimate with wide uncertainty, and costs are assumed constant per decision; the structure of the trade-off is the lesson, not the exact numbers.