EVALUATION · CASE INSPECTION BENCH
Dataset-driven evaluation: choose the test slice
Inspect each recorded score, compare its decision with the reference label and watch the confusion matrix fill.
Your experiment
Compare easy, hard and all fixtures at the same threshold. Record their sample counts and error types; these curated examples do not estimate production capacity.
The complete-fixture preview and animation use exactly the same decisions. Playback inspects one case every four seconds; Next case allows manual review.
How the metrics are computed
Positive means “accept as correct.” Precision = TP / (TP + FP). Recall = TP / (TP + FN). F1 = 2TP / (2TP + FP + FN). A zero denominator is displayed as undefined.
Weighted error cost = false acceptances × their configured cost + false rejections × their configured cost. Costs are arbitrary teaching units, not money.
Complete fixture and decisions
| Case | Scenario | Reference | Baseline score | Candidate score | Outcome |
|---|
Offline metric exercise, not an ADK evaluator implementation or a model benchmark. Sixteen scenario descriptions, reference labels and both score sets are handcrafted fixtures; there are no actual generated answers, model calls or measured judge confidence. The permissive rubric relabels C4 and C12 only. Scores are not calibrated probabilities. This small curated set cannot establish generalization, statistical significance, fairness or production readiness. No training, threshold fitting on held-out data, or independent test split is performed. Metric definitions reference.
Check which dataset your result describes
Compare the nine-case easy slice with the seven-case hard slice and reconcile both with the sixteen-case confusion counts. Then change only the reference rubric and locate C4 and C12 in the table. Record the slice and rubric with every result; changing either changes the question being evaluated.
Keep the comparison fixed
Recorded score set chooses baseline or candidate scores for the same sixteen fictional cases. Scores and reference labels are printed in the complete fixture; no agent, judge or remote evaluator runs. Acceptance threshold makes a positive decision when score is greater than or equal to the threshold. With Candidate, All sixteen cases, Strict rubric and threshold 0.50, there are eight true positives, three false positives, zero false negatives and five true negatives. Precision is 8/11, recall is 8/8 and F1 is 16/19, displayed as 84.2 percent. Baseline at the same settings has six true positives, four false positives, two false negatives and four true negatives: F1 is 12/18, or 66.7 percent. These are fixture arithmetic results, not measured model rankings.
Separate the slice from the rubric
Dataset slice selects nine easy cases or seven hard cases. At the default candidate threshold, the easy slice has five true positives and four true negatives, so F1 is 100 percent. The hard slice has three true positives, three false positives and one true negative; its F1 is 6/9, or 66.7 percent. The all-case result is computed from pooled counts, not an unweighted average of the slice percentages. Reference rubric changes only C4 and C12: Permissive accepts their missing qualifications and exceptions, while Strict rejects them. Their recorded scores stay unchanged. Under Permissive, the complete candidate run has ten true positives and one false positive, giving F1 20/21. A changed grading policy cannot be presented as a model improvement.
Inspect the failure boundaries
Set Acceptance threshold to 1.00. Neither score set reaches one, so no answer is accepted: precision has a zero denominator and remains undefined, recall is zero and F1 is zero. Set threshold to 0.00 and every answer is accepted; with Strict and All cases, eight are false acceptances. At candidate threshold 0.95 only C1 and C15 pass, yielding precision one, recall 2/8 and F1 4/10. C15 scores exactly 0.95, demonstrating the inclusive threshold. High precision alone does not establish useful coverage. Empty-prefix metrics before playback are also undefined, because no case has yet been inspected.
Choose costs and a release condition deliberately
Cost per false acceptance multiplies FP, and Cost per false rejection multiplies FN. The units are hypothetical error penalties, not currency. On Baseline with threshold 0.50, set these costs to twenty and ten: four false acceptances and two false rejections produce cost one hundred. The confusion counts and F1 do not change when costs change. Minimum F1 for fixture gate compares the unrounded complete-fixture F1 against your requirement. Candidate passes 0.80 but fails 0.85 under the initial strict settings; rounding the displayed 84.2 percent does not change that decision. A passing F1 gate ignores per-slice weakness and error cost, so it is only one explicitly limited check.
Control and inspect the run
Play reveals one case per four inspection seconds: the complete set takes sixty-four seconds, Easy takes thirty-six and Hard takes twenty-eight. This timeline is for reading, not a latency or throughput measurement. Pause freezes it; Next case pauses and advances to the next case boundary; Restart clears the processed prefix while retaining policy and speed. Inspection scrubs the timeline. Speed offers quarter, half, normal and double rates, and changing it settles elapsed time at the previous rate first. Editing any policy resets without autoplay. Hiding the tab pauses. The complete-fixture preview and table always show the selected full run; Processed metrics show only the revealed prefix. On a phone, focus and horizontally scroll the diagram or table to read their full width. The caption, metrics and table provide text alternatives for the colored cells.
Record evidence and respect the limits
Capture baseline preserves a visible comparison; Replace baseline updates it and Clear baseline removes it. The notebook keeps observations locally, and Export notes downloads them. Expand experiment and Exit full screen change only the view. These curated cases do not include actual answers, sampling variation, timeouts or a held-out test split. Scores are not calibrated probabilities. Do not tune thresholds on this fixture and claim independent validation. A production evaluation needs representative data, stable reference rubrics, explicit failure accounting and a separate test set. The readings discuss reference-label governance and versioned, stratified datasets; this lab implements only deterministic binary decision accounting.