LLM Jailbreaks and Prompt Injection: Build Defenses at the Trust Boundary, in depth

Two failures that need different measurements

A jailbreak is an attempt to make the model violate its own safety policy, for example producing content it was trained to refuse. Prompt injection is an attempt to make the application do something its owner did not intend, by placing instructions in text the model will read. OWASP lists prompt injection as LLM01 in its 2025 Top 10 for LLM applications and separates direct injection (the user types it) from indirect injection (it arrives inside a web page, email, file or tool result). It also notes that retrieval-augmented generation does not remove the risk; retrieval is one of the main delivery routes.

The two failures need different measurements. A jailbreak is judged on the output text. An injection is judged on effects: did data leave, did a tool run, did a record change. An assistant can refuse every harmful question and still forward a customer's address to an attacker because a retrieved page asked it to. Conversely, an unpleasant sentence in a reply does not mean the payment service was touched. Keep separate test suites and separate metrics, and never let a good jailbreak score stand in for injection resistance. For the attack mechanics themselves, see LLM jailbreaking in depth and indirect prompt injection in depth; this article is about the defenses you build around the model.

Map the trust boundary on a worked example

Take the support assistant. A single turn mixes four inputs. The authenticated session says customer 4411 is logged in. The customer's message claims order 9902 arrived broken. The retrieval step returns the refund policy, version 7, owned by the payments team. A tool call returns the order record from the order service. Concatenated into one prompt, all four look the same to the model: tokens. Your application must not treat them the same.

Write a short table for every input: origin, what it may be used for, and the maximum effect it may cause. The session is trusted and decides identity. The order record is trusted data but not an instruction. The policy is trusted reference text. The customer message is a claim; it can name an order but cannot choose the account, the amount, or the destination. Anything fetched from the open web, an uploaded file, or an email is untrusted data that may only be summarised or quoted.

Label provenance and propagate taint

The first concrete control is a context builder that never loses provenance. Every span placed in the prompt carries an origin label, and every value the model later emits is traced back to the spans that could have influenced it. In practice you cannot trace influence precisely inside the model, so use the conservative rule: once any untrusted span is in the context window, the whole turn is tainted, and every proposal from that turn inherits the taint.

The labels also travel through summaries. If one agent step summarises an untrusted web page and a later step reads the summary, the summary is still untrusted. Losing that label is how injections survive compression; injection via LLM-generated summaries walks through the attack.

from dataclasses import dataclass, field
from enum import IntEnum

class Trust(IntEnum):
    UNTRUSTED = 0      # user text, web pages, files, email, third-party tool output
    TRUSTED_DATA = 1   # records from your own services, read with the session's scopes
    TRUSTED_POLICY = 2 # versioned text owned by your team

@dataclass
class Span:
    text: str
    origin: str        # e.g. "web:example.org/page", "orders:9902", "policy:refunds@7"
    trust: Trust

@dataclass
class Turn:
    spans: list[Span] = field(default_factory=list)

    def add(self, span: Span) -> None:
        self.spans.append(span)

    @property
    def taint(self) -> Trust:
        # Conservative: the turn is only as trusted as its least trusted span.
        return min((s.trust for s in self.spans), default=Trust.TRUSTED_POLICY)

    def render(self) -> str:
        # Delimit untrusted spans so the model sees boundaries (helps, does not enforce).
        out = []
        for s in self.spans:
            if s.trust == Trust.UNTRUSTED:
                out.append(f"<data origin='{s.origin}'>\n{s.text}\n</data>")
            else:
                out.append(s.text)
        return "\n\n".join(out)

The delimiters in render are a form of what Microsoft researchers called spotlighting: marking untrusted text so the model is less likely to treat it as instructions. It measurably reduces success rates in published tests, but it is a probabilistic aid. The taint property is the part you enforce.

Gate capabilities by taint, not by prompt

Simon Willison's "lethal trifecta" names the dangerous combination: an agent that has access to private data, is exposed to untrusted content, and can communicate externally. Any one alone is manageable. All three in the same turn let injected text read secrets and send them out. The policy engine turns that observation into a rule: a tainted turn may not invoke tools that exfiltrate or mutate unless a human approves the exact action.

# Capability classes, declared next to each tool's schema, not inferred by the model.
READ_OWN, READ_PRIVATE, EGRESS, MUTATE = "read_own", "read_private", "egress", "mutate"

TOOLS = {
    "get_order":        {READ_OWN},
    "search_kb":        {READ_OWN},
    "fetch_url":        {EGRESS},            # a URL can carry data out in its query string
    "send_email":       {EGRESS},
    "propose_refund":   {MUTATE},
}

def decide(tool: str, turn_taint: Trust, approved: bool) -> str:
    caps = TOOLS[tool]
    if turn_taint == Trust.UNTRUSTED and caps & {EGRESS, MUTATE}:
        return "allow" if approved else "needs_approval"
    return "allow"

Notice what the rule does not do. It does not ask the model whether the request looks malicious, and it does not scan the text for suspicious phrases. Those are useful signals elsewhere, but the gate is a function of origin labels and declared capabilities, so a cleverly worded injection cannot talk its way past it. fetch_url is classed as egress because a fetched address can encode private data; restricting destinations is covered in egress control for agents.

Fix the plan before untrusted data arrives

Gating by taint makes agents that read the web much less useful, because almost every turn becomes tainted. The architectural answer is to fix the plan before untrusted data arrives. Willison's dual LLM pattern (2023) splits work between a privileged model that sees only the trusted request and plans tool calls, and a quarantined model that processes untrusted text but has no tools. The privileged side refers to quarantined outputs by opaque variable names and never reads their contents. Google DeepMind's CaMeL paper (2025) extended the idea: the privileged model writes a small program, an interpreter runs it, and every value carries capability metadata that a policy checks before each tool call.

In a customer-facing bot the customer's own message is untrusted too, so the planner treats it as intent only: it picks one of a small, fixed set of plan templates before any retrieved document, order note or tool output enters context. For "refund my broken order" the template is: look up the customer's orders, check the policy, propose a refund. The customer's text chooses which template runs but cannot add steps to it, and third-party content goes only to the quarantined side. If a retrieved page or the order's free-text note contains an instruction, it can change the summary the quarantined model writes, but it cannot add a send_email step to a plan that was already compiled. Direct injection can still select the wrong template, so the executor keeps enforcing taint and approval. The cost is flexibility. Tasks whose next step genuinely depends on reading untrusted content, such as following links in a research task, need either human approval at the branch point or a narrower tool set.

Model output is the second boundary

Model output is the second boundary, and OWASP gives it its own entry, LLM05 Improper Output Handling. Treat every generated string as untrusted input to the next system. Encode it for the destination: HTML-escape before rendering, parameterise before it reaches SQL, never pass it to a shell. Strip or neutralise markdown images and links in rendered chat, because an image URL with conversation data in its query string is a zero-click exfiltration channel. Validate structured output against a schema and reject extra fields instead of ignoring them.

Agents add a twist: the output of one model call becomes the input of the next tool. A refund proposal is a JSON object; the executor must re-derive the account from the session rather than reading an account_id field the model filled in. If a field can be derived from trusted state, do not accept it from the model at all.

Separate proposal, approval and execution

OWASP's LLM06 Excessive Agency covers the remaining gap: tools that can do more than the task needs. Expose propose_refund(order_id, reason) rather than a general database executor. The executor resolves the customer from the session, loads the order from the order service, computes the refundable amount in code, and shows a preview with order, amount, currency and destination. Approval is bound to those exact values and expires; any change requires a new approval. Permissions are re-checked at execution time, and an idempotency key prevents a retried call from paying twice. The confused deputy article covers delegated tokens for agents that act across services.

Evaluate effects, not refusal vocabulary

An evaluation harness for this design measures three outcomes per case: did the response violate content policy, did the model attempt a forbidden tool call, and did a forbidden effect actually happen. Only the third is a breach. A blocked attempt is a containment success that still tells you the model was steered.

MARKER = "CANARY-7F3A"   # harmless token planted in synthetic fixtures

def run_case(case, agent, trials=20):
    results = {"policy_violation": 0, "attempted": 0, "effect": 0, "false_block": 0}
    for _ in range(trials):          # sampling is stochastic: report rates, not anecdotes
        trace = agent.run(case.messages, fixtures=case.fixtures)
        results["policy_violation"] += case.judge_text(trace.reply)
        results["attempted"] += any(c.tool in case.forbidden for c in trace.tool_calls)
        results["effect"] += (MARKER in trace.outbound_payloads) or trace.ledger_changed
        results["false_block"] += case.benign and trace.blocked
    return {k: v / trials for k, v in results.items()}

Record the model version, prompt version, tool schema and policy version with every run. Include benign neighbours: a security researcher quoting an injection, a translated complaint, a document about fraud. Anthropic's April 2024 many-shot jailbreaking research showed that long contexts full of examples could shift behaviour in the models tested, so the suite must use the context lengths and turn counts your product permits, not just short single-turn prompts. Run it on every model, prompt or tool change; a continuous red team pipeline shows how to wire that into CI. Only test systems you own or are authorised to assess.

Failure modes

• Taint laundering. A summary, memory entry or cache stores untrusted text without its label and a later turn treats it as trusted. Fix: labels are part of the storage schema, not the prompt.
• Model-supplied identity. The executor reads account_id from the proposal. Fix: derive every identity field from the session.
• Approval fatigue. Tainted turns ask for approval so often that users click through. Fix: move planning before retrieval so fewer turns need it, and show diffs, not prose.
• Classifier as gate. An injection detector is placed in front of tools and treated as the control. Fix: use it as a signal for logging and review; the gate stays deterministic.
• Rendering leaks. Markdown images or auto-fetched links carry data out. Fix: strip remote resources from rendered output or proxy them through an allowlist.
• Refusal-only metrics. The dashboard shows refusal rate going up while effect rate is never measured. Fix: count effects.

Trade-offs

Strict taint gating is safe and blunt: it will block legitimate actions in turns that merely read a document. Plan-then-execute keeps capability but costs flexibility and engineering effort, because you now maintain an interpreter or a fixed planner. Classifiers and spotlighting are cheap and lower average success rates, but they give no guarantee against an attacker who adapts to them, so they belong in the detection layer. Human approval is strong when the preview is precise and weak when it is a paragraph of model prose. Most products end up with a mix: read-only flows run freely, egress and mutation require either an untainted plan or an approval bound to exact values.

What to do next

1. Inventory every input to your assistant and record its origin, permitted use and maximum effect.
2. Add an origin label to every context span and persist it through summaries, memory and caches.
3. Declare a capability class for every tool and enforce the taint-by-capability rule in code.
4. Remove identity, tenant and amount fields from tool schemas; derive them from the session and your services.
5. Escape and validate all model output for its destination; strip remote images and links in rendered chat.
6. Bind approvals to exact action values with an expiry and an idempotency key.
7. Build an evaluation suite that counts effects, attempts, policy violations and false blocks separately, with benign neighbours.
8. Run the suite on every model, prompt and tool-schema change, and write the release criterion as an enforceable property.