LLM red team architecture — attack corpus, automation, human, reporting

Why architecture matters here

A red team is not a test suite, and treating the two as interchangeable is the most common way an AI security programme produces activity without assurance. A suite answers a question you have already asked: these four thousand attacks used to work, do they still work now? A red team exists to ask the questions nobody has written down yet, and its output is not a pass rate. It is a set of findings that changes what the suite contains next quarter.

That distinction decides what this article covers and what it hands off. The mechanics of the adversarial harness - mutating seed payloads into variants, running an automated attacker loop within a turn budget, scoring completions with a judge, pinning past incidents as a permanent regression wall and failing a build on them - are developed in jailbreak defense architecture and in agent safety evaluation. Judge validation, sampling variance and eval-set construction belong to evaluation methodology. What is left over, and what those articles do not touch, is the programme wrapped around the harness: who is permitted to attack what, how you know whether you have looked everywhere, who owns a finding once one exists, and what you put in front of a board that cannot read a confusion matrix.

Get the programme wrong and you get the failure mode that shows up in organisations whose tooling looks mature: an impressive count of attack attempts, an attack success rate trending reassuringly downward, and a launch that ships a whole harm class nobody thought to try.

The architecture: every piece explained

Read the diagram in two halves. The top rows - corpus, automated attacker, human testers, harness, judge - are machinery, and they are covered in depth in the articles linked above. Treat them here as a box that consumes attempts and emits scored attempts. Everything below that line is the programme, and it is where red teaming succeeds or fails as an organisational practice rather than as a piece of tooling.

Coverage is the accounting layer, and the interesting quantity is not how many attempts ran but which regions of a harm-by-surface grid were touched at all. Reporting converts scored attempts into findings, which are a different kind of object: a finding has a severity, an owner and a due date, and it outlives the run that produced it. Fix cycle is the contract that a finding becomes a code change and then a permanent test case. Coordination is the standing interface to the people building defences, and metrics is the trend line you carry upward.

The ops band underneath is the part teams bolt on last and regret. A red-team exercise generates artefacts that are themselves hazardous: payloads known to work, transcripts of your own product producing exactly the content your policy forbids, and occasionally real customer records pulled out of a staging index that turned out to be less synthetic than promised. Deciding where those artefacts live, who may read them, and when they are destroyed is a design decision taken before the exercise, not paperwork filed after it. It is also the first thing counsel asks about when an exercise goes sideways.

Rules of engagement: what an exercise may touch

Every exercise starts with a written scope, and a scope is more than a list of endpoints. Five things need deciding before anyone types a prompt.

Surface. A red team pointed at the base model is testing the vendor's alignment work, which you cannot fix and did not buy. Point it at your product instead: the system prompt, the retrieval corpus, the tool set, the permissions those tools hold, the human review queue behind them. The failures worth finding live at the seams between those pieces rather than inside the weights.

Environment. Staging is safe and it lies; production is honest and it is dangerous. The compromise most teams settle on is production code paths and production configuration against a production-shaped but synthetic data set, with tool side effects stubbed at the outermost boundary so a successful attack produces a recorded intent rather than a transfer.

Blast radius. State in advance what a tester may actually do when a payload works. Read but not write; write only to a fixture tenant; never to a real one. Without that line drawn beforehand, a successful exfiltration test is an exfiltration, and your finding becomes your incident.

Authorisation. A named person signs the scope, and the exercise carries a reference that an on-call engineer can look up at three in the morning when the anomaly detector lights up on the testers' traffic. Tripping your own detection is a good result. Paging six people who have no way to find out why is an expensive one.

Duration and a stopping rule. An exercise with no end date runs until the calendar interrupts it. Fix the window, and define what finishing means: either the grid is filled to the agreed depth, or the last discovery of a genuinely new technique is far enough back that the team has moved on to grinding variants of what it already has.

Coverage is a denominator, not a count

The number a red team most wants to put on a slide is the number of attempts, and it is the number that carries the least information. Four thousand attempts produced by parameterising three seed ideas across encodings, personas and carrier tools is three ideas wearing four thousand costumes, and one defence that recognises the shape retires all of them at once.

Coverage needs a denominator to be a percentage of anything. Build one as a grid. Down one axis, the harm categories - reuse the policy taxonomy your moderation stack already maintains rather than inventing a second one that will drift out of sync with it. Across the other, the delivery surfaces: a user turn, a retrieved document, a tool result, an uploaded file, state carried from an earlier conversation, a message from another agent. Every cell is a question with a yes-or-no answer: has anyone ever tried to cause this harm through this surface? An honest coverage report says how many cells were attempted, how many were attempted by a person rather than by a generator, and which ones are still empty.

The empty cells are the deliverable. They are the only part of a red-team report with predictive value, because everything else in it describes attacks you have now seen and will shortly defend. Two disciplines keep the count truthful. Cluster attempts by technique before counting, so a run that reads as thousands resolves to the few dozen distinct ideas it actually contains. And track human-attempted cells separately, because a generator only varies the dimensions somebody thought to parameterise - a new row on the grid has never once come from the generator that fills in the existing ones.

The attack corpus is a governed asset

Where attack cases come from - public collections, templated families, model-assisted mutation, your own incident history - is developed in the safety evaluation article. What that discussion tends to leave out is what the corpus becomes once it exists. A curated, deduplicated, indexed collection of payloads known to make your product misbehave, sorted by the harm each one produces, is close to the most useful document an attacker could hold about you. You built it for them and stored it in a repository.

Treat it accordingly. Access on request against a named business reason, rather than by default to anyone who can clone the monorepo. Storage separate from application code, so that an over-broad open-source release or a leaked build artefact does not carry the corpus along as a passenger - corpora have escaped exactly this way. Payloads held at rest in a form that cannot be executed by accident, encoded or fenced, so a nightly job, a documentation build or a crawler cannot lift them into a live prompt.

Classification inside the corpus matters as much as access to it. A payload that yields mildly off-policy marketing copy and one that yields genuinely dangerous instructions are not the same object and should not sit in the same directory with the same permissions. Some teams keep their highest-severity cases as hashes plus a private retrieval path, so the plain text never appears in a diff, a search index or a screenshot.

Versioning is the operational half, and it protects your own reporting from you. Pin a corpus version in every result you publish. A falling attack success rate produced by quietly dropping the hard cases looks identical to progress until somebody thinks to check which cases ran.

From finding to fix: triage, severity and ownership

A scored attempt is data. A finding is an object with an owner, a severity and a date, and the conversion between them is where most programmes leak. Attempts accumulate in a results store nobody reads, and the exercise's value decays to whatever the loudest tester remembers to raise in person.

Severity for an AI finding does not fit a vulnerability-scoring vector cleanly, because the impact is frequently reputational or regulatory rather than technical and because there is often no privilege boundary being crossed. A workable rubric runs on three axes. First, the harm if that output reached a real user in that context. Second, the plausibility of the path - does it require the attacker to plant a document in a corpus they have no way to write to? Third, reproducibility.

Reproducibility is where teams misuse a number. A payload that lands three times in a hundred is still a defect, and it should still fail the regression gate, which judges the worst run rather than the average precisely because an attacker gets to retry; the defence-side gates are built on that assumption. Reproducibility belongs in triage order and fix urgency, not in the decision about whether a finding is real. Deterministic first, because it is exploitable today and verifiable tomorrow.

Ownership is the harder half. A jailbreak has no natural owner: the fix might be one line of a system prompt, a classifier threshold, a tool permission, or a product decision that the capability should not have existed. A queue where everything routes to the safety team becomes a backlog nobody can burn down, and the team that files findings learns that filing them accomplishes nothing. Route by fix location, decided at triage by somebody senior enough to assign work across team boundaries, and give every severity an SLA plus an explicit accept-the-risk path with a named person, a written reason and an expiry date. The alternative is not fewer accepted risks; it is the same risks accepted silently as tickets that rot.

Finally, agree in advance who adjudicates when a defender disputes a finding on the grounds that no real user would ever do that. Disputed findings with no referee are the single most reliable way to make a red team stop reporting.

Independence: who the red team reports to

The red team should not report to the person whose launch date its findings can move. That sounds too obvious to state and is violated constantly, usually without anyone intending it: the engineer who built the classifier is genuinely the best-placed person to attack it, so they do both jobs, and over a few quarters the programme stops finding things that would be inconvenient to have found.

The mechanism is not dishonesty. It is attention. People do not probe their own design at the point where they privately suspect it is weakest and know the fix is a quarter of work; they probe where they are proud of it and expect to be reassured. Independence is what buys you the attacks nobody on the team wants to be true.

Practical placements, in ascending order of independence and cost: a rotating duty inside the product team, which is cheap, catches the obvious and is structurally blind to any assumption the team shares; a central function funded outside the product line and reporting into security or risk, which gives continuity and depth; and periodic external engagement, which is the only one that reliably surfaces assumptions your internal team holds in common with the builders. Most organisations of any size need two of the three.

Whatever the placement, the escalation path matters more than the box on the org chart. The test is simple: can the red team put a finding in front of somebody empowered to stop a launch, without that finding first passing through the person who wants the launch to happen?

External red teams, bounties and inbound reports

External testing buys the assumptions your own people cannot see and charges you a scoping problem in return. A vendor engagement needs everything an internal exercise needs plus three additions: what data the vendor may retain after the engagement ends, whether they may publish, and what becomes of the payloads they develop while working on you. A contract that lets a vendor keep and reuse the corpus they built against your product has, in practical terms, published it.

A bounty programme is a different animal, because you cannot scope the participants - only the invitation. Two things decide whether it produces value. The first is the scope wording. Without explicit exclusions you will receive a steady inbound stream of hallucination reports, off-policy jokes, and complaints that a refusal is censorship, all submitted in good faith and none of them actionable. Say what counts: a demonstrated bypass of a stated policy, an injection through a named surface, data belonging to another tenant. Say plainly what does not. The second is triage capacity. AI bounties generate a high volume of low-signal submissions with a thin tail of genuinely novel ones, and if your median time to first response drifts past a couple of weeks, the researchers whose submissions you wanted are the first ones to stop bothering.

Reports that arrive outside any programme still need a published address, an acknowledgement clock and a decision about disclosure timelines made before you need it rather than during. Those mechanics are their own subject - see bounty programme design and responsible disclosure.

Reporting to people who do not run the harness

The audience for a red-team report is usually an executive, a lawyer or an auditor, and all three will misread the same chart in the same way.

The trap is the trend line. Attack success rate is a ratio whose numerator is what you found and whose denominator is what you tried, and both of those move when the team gets better at its job. A quarter in which success rate rose is far more often a quarter in which the testers learned a new technique than one in which the product got worse. Report that ratio on its own and you teach leadership to reward the team for finding less, which is the one incentive an assurance function cannot survive. Always report it beside what changed underneath it: which corpus version ran, which techniques were introduced, which surfaces entered scope.

What non-engineers actually need is three things. Coverage, stated as what was tested and, with more emphasis, what has never been tested. Residual risk, meaning the findings you decided not to fix, who accepted each one and until when. And time-to-fix by severity, which is the only figure in the whole report that demonstrates findings turn into changes rather than into tickets.

Two presentation rules save arguments later. Never put a working payload in a document that will be forwarded; describe the technique and cite the case identifier, and let anyone who needs the string go and get it from the corpus under its own access controls. And keep a hard line between we found this and this happened. A red-team finding is not an incident, and a report that blurs the two will eventually be read as a breach notification by somebody whose entire job is escalating breach notifications.

Staffing, rotation and the cost of the work

Red teaming a language model means spending working days eliciting, reading and cataloguing precisely the material your policy exists to prevent. That is a moderation-adjacent job in everything except its title, and it carries the same occupational hazard - typically without any of the support structures that content moderation organisations spent a decade learning to build.

Plan for it explicitly. Cap the share of any individual's time spent inside the highest-harm categories, and rotate people between categories rather than letting one person become the permanent owner of the worst one, which is exactly what happens by default because they got good at it. Push volume onto the automation so that a human reads a sampled and clustered set rather than every generated transcript. Keep review of the most severe outputs opt-in, and staffed by people who chose it knowing what it involves.

Composition deserves as much thought as headcount. In most exercises the highest-value findings do not come from security engineers at all. They come from people who understand the domain the product operates in: a clinician, a compliance officer, a support agent who knows what customers genuinely ask for and where the awkward requests cluster. Security specialists find the technique. Domain specialists know which harms are worth a technique in the first place, and they are the ones who add rows to the grid.

End-to-end flow

Concretely, a two-week exercise opens with a signed scope naming the surfaces, the environment and the blast-radius limit. The harness replays the pinned corpus to establish a baseline, freeing the human testers to work the empty regions of the coverage grid rather than re-deriving attacks the tooling already runs nightly. Forty attempts score as bypasses. Clustering by technique resolves those forty to nine distinct ideas, of which two are genuinely new rows on the grid and seven are variants of families already tracked - a distinction that never survives to the summary slide unless someone does the clustering deliberately.

Triage then assigns each of the nine a severity and an owner chosen by where the fix lives: one system-prompt change, one tool-permission tightening, one classifier threshold with a retrain behind it, and a fourth that turns out not to be an engineering problem at all but a policy question, routed to the policy owner with a decision date attached. Each accepted finding becomes a permanent case so the fix stays testable after the people involved have moved on. The report that goes upward leads with the two empty regions of the grid rather than with the bypass count, and the residual-risk table names who accepted what, and until when.