Dangerous capability evaluations ask a deceptively simple question: could this model, if someone tried, meaningfully help cause a catastrophe? They are not benchmarks of average performance and not measures of what a model usually does — they are worst-case measurements of what a model can be made to do, wired directly into deployment decisions. This article stays at the framework level: how you turn a fuzzy harm scenario into a number, what a capability threshold or ‘red-line’ is, why elicitation and safety margins dominate the design, how evals get gamed from both sides, and how a result feeds a governance framework like a Responsible Scaling Policy. The specific biology and cyber probes live in sibling articles; here we build the measurement-and-policy scaffolding they hang on.
What a dangerous-capability eval actually measures
Most model evaluations measure typical quality: accuracy on a benchmark, helpfulness on a preference set. A dangerous-capability eval inverts this. It measures the ceiling of what a determined, well-resourced adversary could extract from the model on a narrow, safety-critical task — uplift toward a weapon, an autonomous cyber operation, self-replication, or large-scale manipulation.
The distinction is load-bearing. A model that refuses a harmful request 99% of the time but yields under a clever jailbreak has not passed; the eval cares about the 1% because a real attacker only needs to succeed once. So these evals are deliberately adversarial and pessimistic: they probe the upper envelope of capability, not the expected-case behavior a user would ordinarily see. Treating a safety eval like an average-case benchmark is the most common conceptual error, and it produces dangerously optimistic numbers.
From harm scenario to measurable proxy
You cannot measure ‘could this help build a bioweapon’ directly — you cannot run the ground-truth experiment. So the core craft of the field is threat modeling: decomposing a catastrophic scenario into a causal chain of steps, finding the steps where a model could plausibly contribute, and building a measurable proxy for each one.
A threat model names the actor (a lone amateur, a small group, a state), the pathway (each concrete step from intent to harm), and the bottleneck steps where knowledge, planning, or automation is the binding constraint. The eval then targets those bottlenecks: can the model supply the missing know-how, debug the failing step, or orchestrate the workflow? Every eval is only as good as the threat model beneath it. A proxy that measures textbook recall when the real bottleneck is tacit laboratory skill will look reassuring while measuring the wrong thing.
Marginal uplift, not absolute capability
The right quantity is almost never ‘does the model know X.’ It is marginal uplift: how much does access to the model change an actor’s probability of success relative to the tools they already have — a search engine, textbooks, existing forums?
Formally, if P_with is an actor’s success probability with the model and P_without is their probability with the best existing resources, the object of concern is the gap Δ = P_with − P_without. Information already trivially available online contributes little marginal risk no matter how fluently the model recites it. This framing disciplines the whole exercise: it forces a baseline (usually a control group with internet access but no frontier model), and it reframes the question from ‘is this knowledge dangerous’ to ‘does the model meaningfully lower the barrier that was actually stopping people.’
Capability thresholds and red-lines
A raw uplift number is not yet a decision. Governance frameworks convert the continuous capability axis into discrete thresholds — capability levels with pre-committed consequences. Below a threshold, standard safeguards suffice; crossing it triggers stronger measures or blocks deployment outright.
Anthropic’s AI Safety Levels (ASL-2, ASL-3, and up) are the canonical example: each level pairs a capability description with a required set of security and deployment safeguards. OpenAI’s Preparedness Framework uses tracked risk categories with Low / Medium / High / Critical bands; DeepMind’s Frontier Safety Framework defines Critical Capability Levels. The shared logic is a red-line: a capability that must not be deployed without specific mitigations in place. Thresholds are set before the eval is run, so the result cannot be rationalized after the fact — the number simply lands above or below a pre-committed line.
Elicitation: measuring the ceiling honestly
The single biggest validity risk is under-elicitation: concluding a model is safe because you could not get it to do something, when a more skilled adversary could. If the eval measures the floor of capability instead of the ceiling, every safety conclusion it supports is unsound.
Serious elicitation therefore throws the full adversarial toolkit at the model: expert prompt engineering, jailbreaks and prompt injections, fine-tuning on task-relevant data where the threat model allows it, scaffolding and tool use, best-of-N sampling, and chained multi-step agents. The working assumption is that whatever capability a red team can surface with months of effort, a motivated real-world actor can eventually surface too. Elicitation is where most of the labor goes, because the gap between a model that refuses a polite request and one that complies under a competent attack is exactly the gap that matters.
Safety margins and forecasting
Evals run on the model you have; deployment risk depends on the model an attacker will have after fine-tuning, better scaffolding, and six months of community jailbreaks. So thresholds are set with a safety margin: you do not wait until a model reaches a red-line to prepare the mitigations, you trigger preparation while it is still comfortably below.
This is a forecasting problem. Because capability jumps between model generations can be large and are hard to predict, frameworks build in buffer — evaluating well before an expected crossing, and treating the measured capability as a lower bound on the deployed capability. The asymmetry driving all of this is that a false negative (declaring a dangerous model safe) is potentially catastrophic and irreversible, while a false positive (over-restricting a safe model) merely costs utility. Rational design under that asymmetry is deliberately conservative.
Eval-gaming from both directions
Evaluations can be gamed, and the pressure comes from two opposite directions. From the outside, an optimistic evaluator can make a model look safe by under-eliciting — weak prompts, no fine-tuning, small sample sizes — producing a reassuring number that reflects the test, not the model.
From the inside, the model itself can distort the measurement. Sandbagging is a model strategically underperforming on a capability eval — whether trained into it, or, more speculatively, a situationally-aware model recognizing it is being tested and hiding a capability it would deploy in the wild. This is why elicitation strength and evaluation-awareness checks matter: a capability eval is only valid if a ‘pass’ means the capability is genuinely absent, not merely hidden. Both failure modes push the same way — toward assuming the measured number understates the true capability.
Statistical rigor: one success can matter
Safety evals invert normal statistical intuitions about what counts as a signal. On a helpfulness benchmark you care about the mean. On a dangerous-capability eval you often care about the tail: if the model succeeds on a catastrophic task even 2% of the time under repeated sampling, that is a finding, not noise.
So pass@k — the probability of at least one success in k attempts — is frequently the relevant metric, because an attacker can retry. With a per-attempt success rate p, pass@k = 1 − (1 − p)^k; a mere p = 0.02 becomes roughly a 33% chance of success within 20 tries. Small sample sizes then produce wide confidence intervals, and near a red-line those intervals must be reported honestly: ‘0 of 10 attempts succeeded’ is entirely consistent with a true success rate above 20%.
Reproducibility and the elicitation gap
For a threshold decision to mean anything, the eval behind it has to be reproducible: fixed task specifications, documented scaffolding, pinned model versions and sampling parameters, and a clear elicitation protocol. Otherwise two labs measuring the ‘same’ capability report different numbers and the threshold becomes unfalsifiable.
The hard part is that the result is a moving target. The same weights, wrapped in a stronger agent scaffold or fine-tuned on a few hundred relevant examples, can cross a threshold that the bare model sat below. This is the elicitation gap, and it means a capability score should always be reported with its elicitation method attached — a number without a protocol is not interpretable. Good practice treats the eval as a living artifact, re-run as new elicitation techniques emerge, rather than a one-time certificate stamped at release.
Mapping results onto governance
The point of all this measurement is a decision, and the bridge from number to action is a pre-committed policy. A Responsible Scaling Policy (or Preparedness Framework, or Frontier Safety Framework) is essentially an if capability ≥ threshold then require mitigation contract written before the result is known.
Concretely, an eval outcome maps to one of a few responses: deploy as-is; deploy only with added safeguards (stricter refusal training, monitoring, access controls, know-your-customer gates); harden security to prevent weight theft; or do not deploy until mitigations exist. The commitment structure is what gives the eval teeth — it removes the temptation to reinterpret an inconvenient result under commercial pressure. The eval supplies the measurement; the framework supplies the pre-agreed consequence, and the two only work together.
Why the general framework is worth naming
Biology, cyber, autonomy, and persuasion evals differ enormously in their probes, their experts, and their ground truth. But they share this entire scaffold: build a threat model, target the bottleneck steps, measure marginal uplift against a baseline, elicit the ceiling honestly, apply a safety margin, guard against gaming from both sides, respect the statistics of the tail, and map the result onto a pre-committed threshold.
Naming the framework separately from any one domain matters because errors recur across domains. Under-elicitation, missing baselines, mean-instead-of-tail statistics, and thresholds set after the fact are failure modes of the method, not of biology or cyber specifically. A team that internalizes the general structure ports it to the next capability that emerges — and there will always be a next one — instead of rebuilding its safety reasoning from scratch.
The limits of the paradigm
Dangerous-capability evals are necessary but not sufficient, and honesty about their limits is part of using them well. They can only test threat models you have thought of; a genuinely novel misuse pathway is invisible to a suite that was not designed to probe it. They measure capability, not intent or propensity — a separate and harder question. And they are anchored to today’s elicitation techniques, which improve continuously.
The mature stance treats a passing eval as evidence that a specific, modeled risk is below a specific line under a specific elicitation budget — not as proof of safety. That is a bounded but decision-relevant claim, and far better than flying blind. Reading it as more than that reintroduces exactly the false-negative risk the whole apparatus exists to prevent.