EVALUATION · CASE INSPECTION BENCH
CI evaluation gate: accept or reject a fixture run
Inspect each recorded score, compare its decision with the reference label and watch the confusion matrix fill.
Scroll the diagram and fixture table horizontally to inspect every cell and score. The metrics below report the same counts as text.
Your experiment
Increase the required F1 until the fixture gate fails. Explain why passing this configured gate alone is not sufficient evidence for a production release.
The complete-fixture preview and animation use exactly the same decisions. Playback inspects one case every four seconds; Next case allows manual review.
How the metrics are computed
Positive means “accept as correct.” Precision = TP / (TP + FP). Recall = TP / (TP + FN). F1 = 2TP / (2TP + FP + FN). A zero denominator is displayed as undefined.
Weighted error cost = false acceptances × their configured cost + false rejections × their configured cost. Costs are arbitrary teaching units, not money.
Complete fixture and decisions
| Case | Scenario | Reference | Baseline score | Candidate score | Outcome |
|---|
Offline metric exercise, not an ADK evaluator implementation or a model benchmark. Sixteen scenario descriptions, reference labels and both score sets are handcrafted fixtures; there are no actual generated answers, model calls or measured judge confidence. The permissive rubric relabels C4 and C12 only. Scores are not calibrated probabilities. This small curated set cannot establish generalization, statistical significance, fairness or production readiness. No training, threshold fitting on held-out data, or independent test split is performed. Metric definitions reference.
A CI gate over recorded scores
Sixteen fictional cases have a reference label, a recorded baseline score and a recorded candidate score. A case is accepted when its score reaches Acceptance threshold. Comparing acceptance with the reference fills four cells: true positives, false positives, false negatives and true negatives. The fixture gate passes when F1 reaches Minimum F1 for fixture gate. No model is called; the scores are fixed, which is what makes the gate reproducible in CI.
Baseline fails and candidate passes
With the strict rubric and threshold 0.5, the baseline score set yields six true positives, four false positives, two false negatives and four true negatives: precision 0.600, recall 0.750, F1 0.667, so the gate fails. Choose the candidate: eight true positives, three false positives, no false negatives and five true negatives give F1 0.842 and a pass. The three accepted-but-wrong cases are C4, C12 and C14, and they still cost fifteen at five units per false acceptance.
An aggregate pass can hide a failing slice
Keep the candidate and set Dataset slice to hard. Only seven cases remain: three true positives, three false positives and one true negative, F1 0.667, a failure. The easy slice passes with F1 1.000. The aggregate gate passed because the easy cases dominate the full set. A CI evaluation that reports one number can approve a change that regresses exactly the cases a release worried about; gating per slice exposes it.
The threshold is not monotonic
Move Acceptance threshold to 0.6: F1 falls to 0.778 and the gate fails. At 0.7 it is 0.750 and fails. At 0.8 precision becomes 1.000, recall 0.750 and F1 0.857, so the gate passes again with cost two, because only C11 and C13 are wrongly rejected. Cost per false acceptance and Cost per false rejection weight the same counts into a cost figure without changing the gate. Reference rubric switches two contested labels; it is explored in detail on the human-evaluation lab.
Inspection and limits
Play, Next case, Restart, Speed and the Inspection slider reveal the cases one by one so the matrix can be checked as it fills. Sixteen cases cannot support a confident release decision: one changed case moves F1 by several points and there is no confidence interval. Recorded scores also go stale when prompts, tools or model versions change, so a real pipeline must record those versions with the scores.