EVALUATION · CASE INSPECTION BENCH
Human evaluation: resolve grading disagreement
Inspect each recorded score, compare its decision with the reference label and watch the confusion matrix fill.
Scroll the diagram and fixture table horizontally to inspect every cell and score. The metrics below report the same counts as text.
Your experiment
Compare strict and permissive rubrics for answers missing qualifications or exceptions. Inspect the affected confusion-matrix cells before deciding which rubric fits the task.
The complete-fixture preview and animation use exactly the same decisions. Playback inspects one case every four seconds; Next case allows manual review.
How the metrics are computed
Positive means “accept as correct.” Precision = TP / (TP + FP). Recall = TP / (TP + FN). F1 = 2TP / (2TP + FP + FN). A zero denominator is displayed as undefined.
Weighted error cost = false acceptances × their configured cost + false rejections × their configured cost. Costs are arbitrary teaching units, not money.
Complete fixture and decisions
| Case | Scenario | Reference | Baseline score | Candidate score | Outcome |
|---|
Offline metric exercise, not an ADK evaluator implementation or a model benchmark. Sixteen scenario descriptions, reference labels and both score sets are handcrafted fixtures; there are no actual generated answers, model calls or measured judge confidence. The permissive rubric relabels C4 and C12 only. Scores are not calibrated probabilities. This small curated set cannot establish generalization, statistical significance, fairness or production readiness. No training, threshold fitting on held-out data, or independent test split is performed. Metric definitions reference.
Two graders disagree on two cases
Human evaluation produces the reference labels that every automatic score is checked against. In this fixture two rubrics disagree on exactly two of sixteen cases. C4, a mostly correct answer missing a qualification, and C12, a helpful answer that omitted an exception, are failures under the strict rubric and passes under the permissive one. Every other label is shared. Reference rubric selects which label set the gate trusts; the recorded scores do not change.
The release verdict follows the disagreement
Select the baseline score set with threshold 0.5. Under the strict rubric the baseline has six true positives, four false positives and two false negatives, F1 0.667, and the gate fails. Switch only Reference rubric to permissive: C4 and C12 become true positives, F1 rises to exactly 0.800 and the gate passes. Nothing about the answers changed. Before a disputed label decides a release, the disagreement has to be adjudicated with a written rule, not resolved by choosing whichever rubric passes.
Measure how much depends on the dispute
Repeat with the candidate. Strict gives F1 0.842 and permissive gives 0.952; both pass, so for this candidate the disagreement does not change the decision. On the hard slice the candidate fails strict with F1 0.667 and passes permissive with 0.909, because both disputed cases are hard. Report the gate under both label sets whenever graders disagree. If the decision flips, the evaluation has not yet established which outcome is correct.
Remaining controls
Recorded score set chooses baseline or candidate. Dataset slice restricts the cases to easy or hard. Acceptance threshold moves the decision boundary; at 0.7 with the permissive rubric the candidate has no false positives and F1 0.889. Cost per false acceptance and Cost per false rejection convert errors into a cost figure: candidate strict costs fifteen, permissive five. Minimum F1 for fixture gate sets the pass line. Play, Next case, Restart, Speed and Inspection reveal cases in order.
What is not modeled
The fixture has two fixed rubrics rather than individual graders, so it cannot compute Cohen kappa, Krippendorff alpha or any agreement statistic, and it has no pairwise or blind rating, rater training, calibration set or adjudication workflow. The scores are invented. Use it to see why disputed labels must be resolved and reported before a gate, not as a method for measuring agreement.