Safety evaluation: inspect false positives and false negatives

Decision trace

What this lab demonstrates

A safety evaluation needs examples labeled by a review policy, not only the number of blocks. This synthetic set includes a harmless question, an allowed but ambiguous request, a prohibited instruction, and a clear harmful request. Every item has a fixed risk score and reference label. Applying a strict, balanced, or permissive threshold yields a predicted block or allow decision. The lab exposes true positives, true negatives, false positives, and false negatives, so a reviewer can see whether a policy inconveniences legitimate users or misses a prohibited case.

How to use each control

Inspection policy selects a risk-score threshold of 0.50, 0.60, or 0.80. Dataset slice chooses the standard scores or a hard slice where one unsafe example scores 0.20 lower. Play inspection reveals one case every fifteen seconds; Pause, Restart, Timeline, and Speed affect viewing only. Each card states the reference label, synthetic score, and policy decision. At the end, the summary reports the four confusion counts. The trace lets you inspect a case even when the visual sequence has not yet reached it.

Try a controlled comparison

Choose the standard slice and strict threshold. The ambiguous but allowed case scores 0.55 and is blocked, a false positive. Move to the balanced threshold and replay; that case is allowed while the 0.65 prohibited case is still blocked. Now choose the hard slice: the prohibited case falls to 0.45 and becomes a false negative under both strict and balanced thresholds. Compare the error cells rather than celebrating one aggregate accuracy number. What extra labeled examples would you need before choosing a threshold for actual users?

What this model simplifies

Scores and labels are handcrafted; there is no classifier, Gemini call, ADK Java agent, human review panel, or representative evaluation dataset. Four examples cannot establish a meaningful production false-positive rate, subgroup fairness, or resilience to adversarial wording. The hard slice changes exactly one score to demonstrate distribution shift, not to estimate its frequency. A real evaluation must define the policy and annotation process, protect sensitive test content, include confidence intervals, and test the version actually deployed. Risk scores here are not calibrated probabilities.

Define the positive class before counting errors

Here positive means predicted unsafe and therefore blocked. Reference unsafe is a fixed annotation, independent of the selected threshold. A true positive blocks an unsafe example, while a true negative allows an allowed example. A false positive blocks an allowed example; a false negative allows an unsafe example. The four cards expose both reference and prediction so each final count can be traced to a row. These scores are handcrafted numbers, not probabilities returned by a safety service.

Compare all three thresholds on identical cases

Inspection policy selects Strict 0.50, Balanced 0.60 or Permissive 0.80. A score equal to or above the threshold is blocked. With Standard four cases, Strict yields TP 2, TN 1, FP 1, FN 0: the allowed 0.55 case is wrongly blocked. Balanced yields TP 2, TN 2, FP 0, FN 0. Permissive yields TP 1, TN 2, FP 0, FN 1 because the unsafe 0.65 case is allowed. The preset names describe this fixture only; Balanced is not a recommended production setting.

Change the score distribution without changing the labels

Dataset slice selects Standard or Hard. Hard reduces only the third score from 0.65 to 0.45; it does not add cases or change any reference annotation. Strict now yields TP 1, TN 1, FP 1, FN 1. Balanced and Permissive each yield TP 1, TN 2, FP 0, FN 1. Read the third card to identify the missed example. This is a controlled perturbation, not evidence of measured distribution shift or an estimate of how often a real classifier fails.

Inspect, pause and compare without changing the experiment

Play inspection walks through four cases over sixty inspection seconds. Pause freezes that clock, Resume continues, and Restart returns to zero without changing the inputs. Timeline seeks directly to any second; case boundaries occur at 15, 30, 45 and 60 seconds. Speed selects half, normal or double pace and never changes the classification. Changing either policy input pauses and resets inspection. Card details and the decision trace preview all fixed rows from the start, while completed state badges and the final summary follow the timeline. Hidden tabs pause automatically.

Keep the evidence within its limits

Capture baseline saves the current displayed experiment for a local comparison; replace or clear it after changing the policy. Notes and Export notes preserve your observations, and Expand experiment provides more screen space. None of these tools changes the four labels. Four synthetic examples cannot establish generalization, calibrated risk, category coverage, fairness or statistical confidence. A real evaluation needs a defined annotation policy and representative held-out cases. Record the score source, policy version and slice alongside error counts; a perfect score on this tiny fixture is only a correct calculation on these four rows.