EVALUATION · CASE INSPECTION BENCH
Regression testing: inspect changed predictions
Inspect each recorded score, compare its decision with the reference label and watch the confusion matrix fill.
Scroll the diagram and fixture table horizontally to inspect every cell and score. The metrics below report the same counts as text.
Your experiment
Compare baseline and candidate predictions at thresholds 0.5 and 0.7. Find improvements and regressions per case rather than inferring all cases improved from one aggregate score.
The complete-fixture preview and animation use exactly the same decisions. Playback inspects one case every four seconds; Next case allows manual review.
How the metrics are computed
Positive means “accept as correct.” Precision = TP / (TP + FP). Recall = TP / (TP + FN). F1 = 2TP / (2TP + FP + FN). A zero denominator is displayed as undefined.
Weighted error cost = false acceptances × their configured cost + false rejections × their configured cost. Costs are arbitrary teaching units, not money.
Complete fixture and decisions
| Case | Scenario | Reference | Baseline score | Candidate score | Outcome |
|---|
Offline metric exercise, not an ADK evaluator implementation or a model benchmark. Sixteen scenario descriptions, reference labels and both score sets are handcrafted fixtures; there are no actual generated answers, model calls or measured judge confidence. The permissive rubric relabels C4 and C12 only. Scores are not calibrated probabilities. This small curated set cannot establish generalization, statistical significance, fairness or production readiness. No training, threshold fitting on held-out data, or independent test split is performed. Metric definitions reference.
Compare decisions case by case
A regression test asks which recorded cases changed decision, not only whether an aggregate moved. Sixteen fictional cases carry a reference label, a recorded baseline score and a recorded candidate score; a case is accepted when its score reaches Acceptance threshold. The page opens on the candidate at 0.5: TP 8, FP 3, FN 0, TN 5, F1 84.2%, weighted error cost 15, a pass. Choose Baseline fixture under Recorded score set: TP 6, FP 4, FN 2, TN 4, F1 66.7%, cost 22, a failure.
Find the three changed cases
Compare the Outcome column of the two runs; the table also lists both scores. At 0.5 exactly three outcomes differ. C6, an invented citation, drops from 0.61 to 0.30 and moves from FP to TN. C11 rises from 0.42 to 0.57 and C13 from 0.48 to 0.63, both from FN to TP. All three are fixes and all are hard cases: under Dataset slice Easy both variants reach F1 100.0%, while Hard shows 25.0% for the baseline and 66.7% for the candidate.
Equal F1, different behavior
Set Acceptance threshold to 0.65. Both variants report TP 6, FP 2, FN 2, TN 6 and F1 75.0%, yet two cases swapped. C4, missing a qualification, rises from 0.64 to 0.73 and becomes a new false acceptance; C14, an unsupported confident claim, falls from 0.77 to 0.62 and stops being one. An aggregate-only check reports no change. At 0.7 the swap flips the verdict: the baseline passes with F1 80.0% and cost 7, the candidate fails with 75.0% and cost 12, because C4 and C12 (0.68 to 0.78) are new false acceptances and only C14 improved.
Whether a flip is a regression depends on the label
Keep 0.7 and choose Reference rubric Permissive. C4 and C12 now count as correct, so the same increases become fixes: the baseline falls to F1 70.6% and fails, while the candidate reaches 88.9% with no false acceptances. Cost per false acceptance prices regressions: under the strict rubric at 0.7, setting it to 20 costs the candidate 42 against 22 for the baseline. Cost per false rejection acts only when false rejections exist; the opening candidate has none, so select the baseline first. Lowering Minimum F1 for fixture gate to 0.75 at 0.7 lets the candidate pass with both regressions intact.
Playback
Nothing plays on load. Play reveals one case every 4 seconds and Pause stops it: 64 seconds for all cases, 36 for the easy slice and 28 for the hard one. C6 appears at 24 seconds, C11 at 44 and C13 at 52. Next case adds one case, Restart returns to zero, Speed changes pace and the Inspection slider seeks. The caption flags cases whose decision disagrees with the reference. Changing an input rebuilds the run.
Assumptions and limits
The scores are recorded fixtures; no agent, model or ADK evaluator runs, and there is no diff view, so changed cases are found by comparing runs. A real suite pins model, prompt and tool versions and handles nondeterministic reruns. Sixteen cases cannot show significance: one changed case moves F1 by several points, and no paired test is computed.