LLM-AS-JUDGE · AGREEMENT AUDIT
Judge evaluation: audit recorded quality scores
Compare twelve recorded judge scores with human reference labels, fill the confusion matrix and compute Cohen’s kappa before trusting the judge.
Your experiment
Twelve responses were scored by a judge model and labelled pass or fail by a human reviewer. Start with the holistic v1 scores at threshold 7, then switch to the rubric scores, move the threshold, correct the length preference and change the agreement gate.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, Speed changes the pace and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
The scores are a fixed, fictional record of two judge prompts; no judge model is called and no response is re-scored. The human labels are treated as ground truth. Cohen’s kappa is computed from the confusion matrix as (observed agreement minus chance agreement) divided by (1 minus chance agreement). Twelve items are far too few for a production audit; the arithmetic is the point.
Twelve recorded scores, a coin flip of agreement
Twelve responses were scored from 1 to 10 by a judge model and labelled pass or fail by a human reviewer. With the holistic v1 scores and Pass when the score is at least 7 the judge agrees with the reviewer on only six: 3 true passes, 3 false passes, 3 false fails and 3 true fails. Accuracy is 50% and Cohen kappa is 0, exactly chance agreement. The false passes are R2, verbose but answering a different question, R4, which cites a paper that does not exist, and R9, polished but off-topic: FALSE PASSES.
A rubric with the reference answer
Switch Judge prompt that produced the scores to v2 rubric with the reference answer. R4 drops to 3 and R2 to 4, the false passes disappear, and only R11, a terse correct answer scored 6, is failed wrongly: kappa 0.83 and JUDGE TRUSTED. Lower the threshold to 6 and every decision matches the reviewer, kappa 1. Raise it to 8 and four correct responses fail: LOW AGREEMENT at kappa 0.33. A threshold is part of the judge, and it has to be calibrated against labels, not picked as a round number.
Correcting a length preference
The v1 judge rewards length: the four responses over 300 words all score 7 or more although three of them are wrong. Check Subtract 2 points from responses over 300 words: R2 and R9 now fail correctly, but R4 still scores 7 and passes, so one false pass remains and kappa rises only to 0.33. A blanket correction treats a symptom; the fabricated citation needs a judge that checks claims against the reference.
The agreement gate
Minimum Cohen kappa to trust the judge applies once false passes are gone, so try it with the v2 scores: their kappa of 0.83 passes the default 0.6 gate and fails a 0.9 gate. Kappa corrects for agreement expected by chance from the pass rates, which raw accuracy does not: at threshold 1 every response passes, accuracy is 50% and kappa is 0, while at threshold 10 every response fails with the same kappa of 0. A gate on false passes alone would accept the all-fail judge.
Reading the run and its limits
Play, Next event, Restart, Speed and Inspection time step through every recorded score and the final audit across sixty seconds; the matrix lane fills as the audit completes, and the table lists word counts, labels, scores and cells. The scores are a fixed record, nothing is re-judged, and the human labels are assumed correct. Twelve items give very wide uncertainty; a real audit needs a larger, stratified sample and more than one reviewer.