GROUND TRUTH · AGREEMENT AND EXPIRY
Ground truth: make the rubric explicit
Build a truth set from two annotators’ labels, measure their agreement with Cohen’s kappa, and see how a policy change makes old labels wrong.
Your experiment
Twelve refund decisions were labelled by annotators A and B under the old 30-day policy, using a rubric that never says how to treat partial refunds. The truth set is A’s labels alone. Compare their agreement, the agent’s measured accuracy and its true accuracy, then make the rubric explicit, adjudicate and relabel.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Twelve fictional cases with binary labels (approve or deny). Annotator A makes one error (G7); under the vague rubric B labels the four partial-refund cases the other way; the candidate agent is wrong only on G4 under the current policy. Kappa = (observed − chance) / (1 − chance), with chance from each annotator’s approve rate. No model or labelling tool is used.
Truth is data with a version
An evaluation scores an agent against a truth set, so the truth set’s errors become the score’s errors. Twelve refund cases were labelled approve or deny by annotators A and B while policy v1 (a 30-day window) applied. Policy v2 (14 days) now applies and turns G6, G8 and G12 into denials. Four cases (G2, G5, G9, G11) are partial refunds. The candidate agent follows the current policy and is wrong only on G4, so its true accuracy is 11 of 12, 91.7%.
The opening run: kappa 0.118 and a 58.3% score
Under the vague rubric (approve if the customer deserves a refund), B labels the four partial cases the opposite way to A, and A mislabels G7. They agree on 7 of 12: observed agreement 0.583, chance agreement 0.528 from their approve rates, Cohen’s kappa (0.583 − 0.528) / (1 − 0.528) = 0.118, far below the 0.6 gate. The truth set is A’s labels, still under v1: the candidate disagrees with it on G4, G6, G7, G8 and G12 and measures 58.3%, although four of those five truth labels are wrong.
Failure boundary: explicit rubric, then current labels
Set Labelling rubric to Explicit rules with worked examples: the annotators now label the partial cases the same way and disagree only on G7, so kappa is 0.824 and the gate passes. The score does not move, because the truth set is still A’s v1 labels. Turn on Relabel policy-dependent cases under policy v2 and the measured accuracy rises to 83.3%; switch Truth set built from to Agreed labels, disagreements adjudicated and G7 is settled correctly, giving 91.7%, the true value. Agreement says the task is well defined; it does not say the labels are current.
Choosing the truth set
Only cases both annotators agreed on drops the disputed cases: under the explicit rubric that leaves 11 cases and 90.9% after relabelling, 63.6% without it. Under the vague rubric it leaves 7 cases and 85.7% after relabelling, a smaller and easier set. Adjudication keeps every case but needs a senior reviewer. Minimum kappa to accept the labels sets the gate: with the vague rubric every reachable setting from 0.4 to 0.9 fails at kappa 0.118, even for a perfectly adjudicated, relabelled truth set, because low agreement means a second labelling round would not reproduce it. With the explicit rubric (kappa 0.824) the gate passes at 0.8 and fails at 0.85.
Reading the run and its limits
Play replays the cases, the rubric, agreement, truth-set construction, scoring and the outcome over sixty inspection seconds; Next event, Restart, Speed and Inspection time control the replay, and any input change recomputes it. The table lists both annotators’ labels, the truth label used, the label under the current policy and the candidate’s decision for every case. Twelve binary cases are far too few for a stable kappa or accuracy; real truth sets record rubric and policy versions per label, as the linked articles describe.