EVALUATION · CASE INSPECTION BENCH
Evaluation integration: scores into decisions
Inspect each recorded score, compare its decision with the reference label and watch the confusion matrix fill.
Scroll the diagram and fixture table horizontally to inspect every cell and score. The metrics below report the same counts as text.
Your experiment
The candidate’s recorded scores are cut at 0.65: two wrong answers are accepted and two correct ones rejected, costing 12 but failing the 0.8 F1 gate. Sweep the threshold from 0.5 to 0.85, then change the two error costs and find the cut each objective prefers.
The complete-fixture preview and animation use exactly the same decisions. Playback inspects one case every four seconds; Next case allows manual review.
How the metrics are computed
Positive means “accept as correct.” Precision = TP / (TP + FP). Recall = TP / (TP + FN). F1 = 2TP / (2TP + FP + FN). A zero denominator is displayed as undefined.
Weighted error cost = false acceptances × their configured cost + false rejections × their configured cost. Costs are arbitrary teaching units, not money.
Complete fixture and decisions
| Case | Scenario | Reference | Baseline score | Candidate score | Outcome |
|---|
Offline metric exercise, not an ADK evaluator implementation or a model benchmark. Sixteen scenario descriptions, reference labels and both score sets are handcrafted fixtures; there are no actual generated answers, model calls or measured judge confidence. The permissive rubric relabels C4 and C12 only. Scores are not calibrated probabilities. This small curated set cannot establish generalization, statistical significance, fairness or production readiness. No training, threshold fitting on held-out data, or independent test split is performed. Metric definitions reference.
A score is not yet a decision
Each of the sixteen fixture cases has a reference label (should this answer be accepted?) and two recorded scores, one from a baseline and one from a candidate. An integration that consumes those scores still has to decide: every case scoring at or above the Acceptance threshold is accepted, everything below is rejected. That single cut turns scores into four counts: correct acceptances (TP), false acceptances (FP), false rejections (FN) and correct rejections (TN), and from them precision, recall, F1 and a weighted error cost. The release question is which of those numbers the decision should optimise.
The opening cut at 0.65
With Recorded score set on the candidate scores, all sixteen cases, the strict rubric and threshold 0.65, the run accepts two wrong answers, C4 (0.73) and C12 (0.78), and rejects two correct ones, C11 (0.57) and C13 (0.63): TP 6, FP 2, FN 2, TN 6. Precision and recall are both 0.75, so F1 is 0.750 and the fixture gate (minimum F1 0.8) fails. With a false acceptance costing 5 and a false rejection 1, the weighted cost is 2 × 5 + 2 × 1 = 12.
Failure boundary: cost and the gate disagree
Lower the threshold to 0.5: C11 and C13 become correct acceptances but C14 (0.62) joins the false acceptances, so FP 3, FN 0, F1 0.842 and the gate passes, yet the cost rises to 15. The gate prefers 0.5, the cost prefers 0.65. Raise it instead: 0.75 leaves only C12 falsely accepted (cost 7, F1 0.800, pass), 0.80 accepts no wrong answer (FP 0, FN 2, cost 2, F1 0.857, pass), and 0.85 also rejects C5 (0.82), so FN 3, F1 0.769 and the gate fails although the cost is only 3. One threshold can pass one criterion and fail the other.
Costs choose the cut; the baseline shows the trap
Set Cost per false acceptance to 1 and Cost per false rejection to 5: rejecting a correct answer is now the expensive mistake, and every threshold from 0.35 to 0.55 reaches the minimum cost of 3 (three false acceptances, no false rejections). With the original 5 : 1 costs the minimum is 2 at 0.80. Now set Recorded score set to Baseline fixture: its cheapest cut under 5 : 1 is also 0.80, cost 3, but there F1 is only 0.769 and the gate fails, while its gate-passing cut at 0.70 costs 7. Setting Reference rubric to Permissive relabels C4 and C12 as acceptable; at 0.65 that removes both false acceptances (F1 0.889, cost 2). Change the label set and every metric moves without any score changing.
Reading the run and its limits
Play reveals one case every four inspection seconds, sixteen cases in 64 seconds at 1×; Next case, Restart, Speed and the inspection slider step, rewind and scrub, and the confusion matrix fills as cases are revealed. Dataset slice restricts the run to the nine easy or seven hard cases: at 0.65 the easy slice has no errors (F1 1.000) while the hard slice has all four (F1 0.333). Minimum F1 for fixture gate only changes the gate verdict. The scores are fixed fixture numbers, not outputs of a model or a judge; sixteen cases are far too few for a confident release decision, and real costs are rarely a fixed pair of constants.