EVAL DATASETS · COVERAGE
Evaluation datasets: expose coverage gaps
Build an evaluation set from logs and see whether it has enough cases in the intents where a candidate regresses.
Your experiment
The candidate regresses on three rare intents: refund disputes (3% of traffic), Spanish address changes (5%) and subscription pause (4%, new this month). Start with 100 cases sampled from last quarter’s logs in proportion to traffic, then refresh the logs, stratify, add per-intent minimums and change the size.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, Speed changes the pace and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Simplified rule: a regression is counted as visible when its intent has at least 10 cases; real detection depends on the size of the drop and the variance. Case counts are rounded traffic shares. Intents, shares and regressions are fictional.
Proportional sampling hides rare intents
The candidate regresses on three rare intents: refund disputes (3% of traffic, -35 points), Spanish address changes (5%, -30) and subscription pause (4%, -25), which only appeared this month. A regression counts as visible when its intent has at least 10 cases. The opening dataset samples 100 cases from last quarter’s logs in proportion to traffic: 3 refund disputes, 5 Spanish address changes and no subscription pause at all: BLIND SPOTS, 0 of 3 visible.
Fresh logs, then stratify
Check Sample from last week’s logs: subscription pause appears, but with only 4 cases, so still 0 of 3. Choose Evenly across intents: each of the eight intents gets 12 cases and all three regressions are visible in 96 cases: GAPS EXPOSED. Stratifying without refreshing gives 14 cases per intent but still misses the new intent: 2 of 3.
Minimums instead of bigger datasets
Keep proportional sampling from fresh logs and set Minimum cases per intent to 10: rare intents are topped up and all three regressions show in 129 cases. Raising Dataset size under proportional sampling works too, but slowly: 300 cases still give refund disputes only 9 cases and miss it, while 350 or 400 cases cover all three. With only 50 cases, stratified sampling gives 6 per intent and sees nothing.
Coverage is a property of the dataset
The overall score of a dataset dominated by common intents moves very little when rare intents collapse. Check per-intent counts before trusting any aggregate, keep minimums for intents that matter more than their traffic share, and refresh from recent logs so new behaviour is represented.
Reading the run and its limits
Play, Next event, Restart, Speed and Inspection time step through each intent and the verdict across sixty seconds; the table lists traffic share, case count, the candidate’s change and whether the regression is visible. The 10-case rule stands in for a statistical test, and counts are rounded traffic shares. Intents and regressions are fictional.