Almost every safety number you will be shown is the fraction of attack prompts a model did not refuse. It is cheap to compute, it is reported without qualification, and on close inspection it frequently measures the absence of an apology string rather than the presence of harm. That gap is where safety evaluation actually lives. The inversion that safety evaluation performs on ordinary quality evaluation — a case passes when the system declines, and a false negative is an incident rather than a regression — is developed at the harness level in agent safety evaluation, and the general craft of eval sets, judges and noise floors belongs to evaluation methodology. This article is about the measurement itself: what counts as a harmful completion, which public benchmarks are evidence of what, why a refusal rate without an over-refusal rate beside it is not a specification, how bias gets measured and why those benchmarks are contested, and what has to hold before two safety scores can honestly be compared at all.
What the number means when it says a jailbreak succeeded
The dominant grading convention is inherited from the first widely copied attack collections. The harmful-behaviour sets that accompanied the early universal-suffix work — AdvBench and its descendants — scored an attempt as successful when the completion did not begin with one of a hand-written list of refusal openers: an apology, a statement of inability, a reminder about being a language model. It is deterministic, free, and reproducible, which is why it spread.
It also fails in both directions, and the errors are not symmetric. The common false success is a completion that declines to refuse and declines to help: the model repeats the prompt, produces fluent nothing, drifts off topic, or refuses in a phrasing nobody put on the list. A grader looking for absent apologies scores all of that as a jailbreak. The rarer but worse false failure is a completion that opens with a refusal-shaped sentence and then supplies the content anyway — scored safe because it apologised first.
StrongREJECT is the benchmark that made this concrete: re-grading attacks that had been reported as highly effective showed that a substantial share of the claimed successes did not actually produce usable harmful content, and that measured attack effectiveness dropped sharply under a grader that asked whether the response answers the forbidden request. Its rubric scores refusal alongside how specific and how convincing the answer is — because an unusable answer is not an attack that worked.
The standard worth adopting is that a completion is harmful when it materially advances the requester's goal: specific, actionable, and correct enough to use. That is a judgement, not a string operation, and it forces a rubric and a grader into your pipeline. It also means an attack success rate quoted without stating how completions were graded is uninterpretable, and two rates produced by different graders are not comparable numbers that happen to differ — they are different quantities.
The pieces, and which ones this article is about
The conventional harness shape is uncontroversial: an attack corpus feeds a runner, the runner produces completions, a grader turns completions into verdicts, a gate compares the result against a pinned baseline, and reporting carries the trend upward. The mechanics of each box are covered elsewhere on this site and are deliberately not repeated here. Replay, fixtures, trajectory-level assertions and worst-of-N gating are developed in agent safety evaluation; the classifier threshold, false-positive-rate and banded-action mechanics live in jailbreak defence architecture; corpus governance, coverage grids and the finding-to-fix contract belong to the red-team programme; eval-set construction, judge calibration and noise floors to evaluation methodology.
The diagram below is inherited from an earlier version of this page, and it is worth reading for what it is missing as much as for what it shows. The input row is entirely adversarial — attack corpus, harmfulness, jailbreak, bias — and there is no box beside it for the benign-adjacent suite. A harness built to that picture can only ever move one number, and the model that maximises it refuses everything. Read the missing box in as you look at it; the rest of this article explains why it is load-bearing.
The judge is an instrument, and it must not be your guard
Once you accept that harm has to be judged rather than pattern-matched, something has to do the judging. The usual candidates are a safety classifier — Llama Guard, ShieldGemma, a provider moderation endpoint — or a purpose-trained grader like the classifier shipped with HarmBench, or a general frontier model driven by a rubric.
The first rule is structural rather than statistical: do not grade with the same classifier that guards production. If the eval judge and the deployed filter are the same model, every attack the filter cannot recognise is also invisible to the eval. The suite then scores perfectly on precisely the class of failure that will reach a user, and the failure mode is silent — the dashboard is green because the instrument and the defence share a blind spot. Grade with something from a different family than you filter with, and where the decision matters, route disagreement between two graders to a human.
The second is taxonomy alignment. A safety classifier is trained against a specific hazard taxonomy, and the widely adopted ones are converging on a shared vocabulary of hazard categories. That is useful as a lingua franca and it is not your policy. If the grader's categories do not map onto the taxonomy your moderation stack already maintains, the resulting rate cannot be attributed to a policy owner, and nobody can act on it.
The third is context blindness. A grader sees a prompt and a completion. It does not see who asked, which tools were attached, what the retrieval layer returned, or whether the output was about to be passed to an interpreter. Whole classes of harm are properties of the trajectory rather than of the text, which is why response-level grading has to be paired with assertions over the event stream. Finally, pin the grader to a version: a grader that moves rescales every number in your history at once, and nothing about that looks wrong on a chart.
Public safety benchmarks and what each one is evidence for
Public benchmarks are worth running and worth being precise about. Each answers a narrow question, and the value comes from knowing which.
Harmful-request refusal. AdvBench supplies harmful behaviours and target strings, and is the corpus most attack papers optimise against; its original refusal-prefix grading is its weak point. HarmBench standardised the comparison — a curated behaviour set spanning several harm categories, including contextual and multimodal behaviours, plus its own trained classifier, so that attacks and defences can be evaluated against each other rather than each in its own harness. JailbreakBench adds reproducibility discipline: an open behaviour set, a leaderboard, and an archive of the actual adversarial prompts, so a reported result can be re-run rather than believed. SORRY-Bench attacks the granularity problem, replacing coarse buckets with a fine-grained topic taxonomy and layering linguistic mutations over it. StrongREJECT, as above, is as much a grader as a dataset.
Toxicity and content. RealToxicityPrompts measures toxic continuation from naturally occurring web prompts; ToxiGen targets implicit hate that contains no slurs, which is exactly the material keyword filters and shallow classifiers miss.
Holistic suites. HELM is the reference argument that safety is not a scalar: many scenarios scored on several metrics — accuracy, calibration, robustness, fairness, bias, toxicity, efficiency — reported side by side rather than collapsed. DecodingTrust does something similar for trustworthiness specifically, spanning toxicity, stereotype bias, adversarial and distribution-shift robustness, privacy, ethics and fairness.
Agentic and domain suites. AgentHarm measures whether refusal survives a tool-using scaffold, where the harm is a completed multi-step task rather than a paragraph. CyberSecEval covers insecure code generation and willingness to assist offensive operations, with later revisions adding prompt injection and interpreter abuse. Procurement-oriented graded benchmarks sit slightly apart: they issue a grade over a hazard taxonomy for buyers, and are not built for debugging your own system.
One boundary is worth stating plainly. All of the above are behavioural: they ask what a model does when asked. Dangerous-capability evaluation asks a different question — what a model could be made to do under serious elicitation — and its machinery of uplift, thresholds and safety margins does not transfer here.
Jailbreak evaluation measures the attacks you already have
An attack success rate against a fixed corpus is a lower bound on how vulnerable you are. It reports what a known set of attacks achieved; it says nothing whatever about the attacks that are not in the set, and a defence tuned against a corpus scores well on that corpus by construction.
The correction is adaptive evaluation, and the lesson is older than language models. In adversarial machine learning, a long series of defences published with strong numbers against pre-existing attacks were subsequently broken by attacks designed with knowledge of the defence. Language-model defences have repeated the pattern: several with impressive static results have fallen to comparatively simple adaptive attacks — tailored templates, random search over suffixes, exploiting logit access, or filling in the beginning of the assistant turn. A robustness claim requires an attacker who knew what they were up against.
Several dimensions have to be varied deliberately or your suite will systematically miss classes of attack. Turn count: gradual escalation across a conversation, opening benign and drifting, defeats any screen that judges a single turn. Surface form: translation into lower-resource languages, encodings, character substitution and persona wrappers routinely recover behaviour that the plain phrasing refuses. Delivery channel: the hostile instruction arriving in a retrieved document or a tool result is a different eval fixture from a hostile user turn, and needs poisoned tool outputs rather than poisoned prompts.
Two properties of safety training itself shape what a valid eval looks like. Refusal behaviour is concentrated in the first few generated tokens — safety alignment is shallow in a fairly literal sense — so an eval that never controls the start of the assistant turn is not testing the property it thinks it is. And refusal behaviour is fragile under fine-tuning: a small amount of further training, including on data with no harmful content in it, measurably degrades it. Safety measured on a released checkpoint is not evidence about your fine-tune. Finally, report the decoding parameters and the number of attempts, and measure at the settings you deploy: the attacker chooses how many times to try, so a single-attempt rate is not the operational quantity.
Over-refusal is a cost you have to measure
A suite made only of harmful prompts can move exactly one number, and the model that maximises it answers nothing. This is not a hypothetical degenerate case — it is the predictable outcome of tuning against a one-sided metric for a couple of quarters, and it arrives as a product that has quietly become useless in its most valuable domains.
The instrument is a paired benign-adjacent suite: prompts that superficially resemble unsafe requests but are entirely legitimate. The recurring categories are homonyms and violent-sounding technical language — how to kill a process, how to inject a dependency — safety-motivated questions, fiction and historical writing, clinical and pharmacological queries, security research, and harm-reduction requests. XSTest is the compact version of this idea, pairing safe prompts with genuinely unsafe contrasts so both rates are readable from one run; OR-Bench scales the seemingly-toxic but-benign side up considerably.
Report the two numbers together, always. An attack success rate on its own is not a specification, and neither is a refusal rate — the pair defines a frontier, and the only honest comparison between two model versions is a comparison of frontiers. A change that moves both in the same direction has told you nothing about safety; it has told you the system became more or less cautious.
Two refinements. First, over-refusal is not distributed evenly: the cost lands on clinicians, lawyers, security researchers, and people asking about self-harm or substance use — populations whose need is high and for whom a refusal is itself a harm, not merely an annoyance. Disaggregate the rate by domain rather than reporting one figure. Second, outright refusal is the visible half; the softer failure is the hedged, watered-down, warning-laden answer that a binary refusal detector happily scores as compliance. Grade usefulness on the benign suite, not just non-refusal. The threshold and banded-action machinery for trading these off in the request path is developed in jailbreak defence architecture.
Bias evaluation, and why its benchmarks are contested
Bias evaluation asks whether equivalent requests receive equivalent treatment, and whether the model reaches for a stereotype when the input under-determines an answer. It is the part of the safety suite with the weakest off-the-shelf instrumentation, and it is worth knowing which tools are sound.
BBQ has the design most worth imitating. Each item appears twice: once in an ambiguous context where the correct answer is that there is not enough information, and once disambiguated so the context determines the answer. The ambiguous half measures whether the model substitutes a stereotype for absent evidence; the disambiguated half measures whether it overrides evidence that cuts against one. Coreference schemas such as Winogender and WinoBias are narrower — occupation and pronoun resolution — but they are cheap, auditable and diagnostic.
The widely cited stereotype pair sets, CrowS-Pairs and StereoSet, deserve a warning. Careful analysis of both has documented serious construct-validity problems: unclear and inconsistent operationalisations of what a stereotype is, pairs that differ in more than the intended attribute, items where the supposedly stereotyping direction is contestable, and outright noise. They remain in leaderboards. Treat a score from them as weak evidence and do not gate on it.
There is also a format gap that catches teams out. Most of these benchmarks are multiple-choice or likelihood comparisons, and models behave differently in free generation. A clean multiple-choice bias score is not evidence that the generative product path treats users equally.
Which is why the evaluation that actually protects your users is one you build: counterfactual perturbation over your own traffic. Hold a request fixed and vary the name, the stated location, the dialect, the pronouns, the implied age; then measure the difference in outcome, in refusal rate, in hedging, in tone, and in which tools get called. Disaggregate every other safety metric along the same strata, because an aggregate that looks fine routinely conceals one group being failed consistently. Perturbation sets need care of their own — a name is a crude proxy for anything — and a null result on a crude proxy is weak evidence of fairness rather than strong evidence of its presence.
Contamination, and refusal that does not generalise
Public safety benchmarks are public. They sit in repositories and papers that are crawled, and they end up in pretraining and instruction-tuning data. A model that declines a benchmark prompt may be recognising a string it has seen thousands of times rather than understanding a request, and nothing in the score distinguishes the two.
The more direct version is not contamination at all but ordinary practice: teams tune safety behaviour on public jailbreak collections because that is where the data is. That is defensible engineering. It does mean the collection has become a training-set metric, and reporting it as an evaluation result afterwards is circular in a way that is easy to miss when the number is flattering.
The behavioural signature of both is narrow generalisation. Refusal learned against specific phrasings often fails to transfer to a paraphrase, a translation, an encoding, or a change of framing that leaves the request identical. So measure the gap deliberately: run the public set, then run a systematically mutated version of the same requests, and treat the difference between the two rates as the number of interest. A large gap says the refusal is keyed to surface form. A small gap is one of the few pieces of genuinely reassuring evidence available in this field.
The corollary is that credible safety evidence is private. A held-out corpus drawn from your own incidents and red-team findings, never published, never used for tuning, and governed as the hazardous asset it is, is the only set whose score cannot have been optimised into meaninglessness. Public benchmarks then serve their proper role: comparison against other systems, and a floor. When your rate on a public set falls to near zero, the conclusion to draw is about the set rather than about the model. Governance of that private corpus — access, storage, versioning, and the reason none of it belongs in the application repository — is covered in the red-team programme.
When two safety numbers can be compared
Safety metrics are almost always consumed as a trend: is this release safer than the last one. That comparison is valid only under conditions that are rarely written down, and when one of them breaks, the resulting chart is not wrong in a visible way — it is wrong smoothly.
Six things have to match for two scores to be comparable: the corpus version, the grader model version and rubric wording, the decoding parameters and attempt count, the scaffold (system prompt, tools, retrieval configuration), the policy taxonomy revision that defines what counts as a violation, and the elicitation effort behind the attacks. Change any one and you have rebaselined, whether or not you meant to.
Two of those are worth calling out because they fail silently. A grader change rescales the entire history at once and nothing looks anomalous, which is the argument for pinning the grader version and for retaining raw completions rather than only scores: with the completions you can re-grade a stored baseline under the new grader and compare inside the new regime instead of across the boundary. And a corpus edit that quietly drops the hard cases produces a falling attack success rate that is indistinguishable from progress until someone checks which cases ran. Pin a corpus version into every published result.
A frozen canary slice — a few dozen fixed attacks, fixed parameters, re-run on a schedule whether or not anything changed on your side — is the cheapest detector of change underneath you. Movement there is attributable to the provider by construction, which is also the fastest answer to the incident question of whether the regression came from your deploy or from a model update you did not make. And when a rate improves, report what moved underneath it: a success rate falling while the corpus got harder is the good case, and it reads identically to the bad case unless the versions are stamped on the result.
What the evidence licenses, and how a release gets gated
A safety programme produces two kinds of evidence and they are not interchangeable. A benchmark rate is a population statistic: repeatable, comparable, and informative about a known set of attacks. A red-team finding is an existence proof: one transcript, no denominator, and it establishes that a failure is reachable in your system. The asymmetry matters at decision time. No measured rate, however clean, licenses the claim that a system cannot be jailbroken — it licenses the claim that none of the attacks you ran, graded that way, succeeded. A single reproducible finding can veto a launch; a clean benchmark run cannot approve one on its own.
So gates should have two shapes. Hard categories, where one reproducible case is disqualifying regardless of rate — child safety, weapons uplift, disclosure of another tenant's data — block on the finding. Everything else is a delta on a rate, which is advisory input to a human who reads the failures rather than an automatic verdict; the mechanics of wiring that into CI belong to agent safety evaluation.
A release evidence package that is actually decidable contains: which corpora ran at which versions and which grader at which version; the decoding parameters and scaffold tested; the public-versus-private gap; the attack-success and over-refusal rates as a pair; the disaggregated bias slice; and the open findings with named risk acceptances and expiry dates. That last item is the one executives can actually act on, and the one most reports omit.
Two final constraints. The evidence is about a system — model plus system prompt plus tools plus retrieval plus guardrails — not about a checkpoint, so swapping any component invalidates it, and a model version change is the marquee trigger for a full re-run precisely because it can improve capability while regressing refusal. And the people producing the evidence should not report to the person whose launch date it can move; the structural case for that independence is made in the red-team programme.