A guardrail is not a paragraph in a system prompt. It is a policy decision, written down, enforced in code, and testable — and the most useful reframing when you start designing one is that the model driving your ADK agent is untrusted input, not a colleague you can brief. Every string it reads is a candidate instruction: the user’s message, a retrieved document, an email body, the JSON your own tool returned. Prompt hardening raises the cost of an attack; only policy bounds the damage — deciding which tools may run, with which arguments, on whose behalf, in which trust context, and what happens when the answer is no. This piece designs that layer: capability envelopes, trust boundaries, tool-use authorization, the confused-deputy trap, human approval, honest refusal, and the adversarial tests that turn a claimed guardrail into a demonstrated one. The callback mechanism that enforces it is a sibling topic; here we decide what the callbacks should say.
Capability, not compliance — screens buy detection, walls buy guarantees
The threat model is hostile to single-point defenses. The attack surface is the input space itself, so indirect injection arriving in a tool result bypasses any defense aimed at the user’s messages. The failure mode is action, not speech: a jailbroken chatbot embarrasses you; a jailbroken agent with tools moves money or encodes secrets into the arguments of an innocent-looking request. And a screen that catches 99% of injections meets thousands of attempts, so the 1% needs a wall behind it.
Hence the organizing principle: capability, not compliance, defines safety. The design question is never ‘will the model refuse?’ but ‘if the model is fully compromised this turn, what is the worst it can do?’ Read the stack in two materials. Probabilistic layers — screening, instruction hardening, model judgment — buy detection and telemetry, and are built from the same kind of system the attacker is manipulating. Deterministic layers — allow-lists, thresholds, IAM roles, egress policy, approval gates — are ordinary code and cannot be persuaded. Both belong in the design; only the second belongs in the risk register.
Write the capability envelope before you write the policy
Guardrail work starts with a one-page artifact, not with code: the capability envelope for this deployment. It enumerates every tool the agent can reach, the identity each acts under, the data each can read or mutate, the egress destinations reachable from the runtime, and the worst case stated as a sentence a non-engineer can check. For a support agent: full compromise can at most read orders belonging to the authenticated user, refund up to Rs. 3,000 per day against those orders, and reach two allow-listed internal hosts.
That sentence does three jobs. It makes review possible — a reviewer can argue with it, which they cannot do with ‘we have guardrails.’ It makes the policy derivable: each clause becomes a specific check at a specific boundary, and any check with no clause behind it is cargo cult. And it makes drift visible — adding a tool changes the sentence, so the sentence becomes what change review actually reviews. Agents rot toward more capability, and an envelope that must be re-signed is the cheapest brake on that drift.
Trust boundaries: label every string by where it came from
Everything downstream depends on a question the runtime must be able to answer at any moment: for this text, who wrote it? By the time content is tokens in a prompt, your database row and a stranger’s web page look identical. So attach provenance where content enters — at the input and tool boundaries — and carry it in session state where later gates can read it.
| Tier | Typical source | Policy stance |
|---|---|---|
| Trusted | System prompt, config, your code | May instruct the agent |
| Semi-trusted | The authenticated user’s own message | May request; may not redefine policy |
| Untrusted | Web pages, email, uploads, third-party APIs | Data only — never instructions |
The payoff is that policy can be written against the tier rather than against the wording of an attack. ‘No outbound web tool in a turn dominated by untrusted content’ is enforceable and testable; ‘block texts that say ignore previous instructions’ is rewritten around in one sentence. Spotlighting — explicitly marking quoted untrusted content as data — is the probabilistic companion to the same idea.
Input screening — cheap rules first, judgment second
Screening belongs at the front door, built as a cascade. Layer one is structural and deterministic: length caps, encoding normalization, stripping zero-width and bidirectional control characters, rejecting content types you do not handle. Layer two is pattern heuristics for known attack shapes. Layer three is a classifier or small-model judge scoring injection likelihood, and separately policy-relevant intent.
What matters in the design is the thresholds. Resist a single block/allow flip: two thresholds are far more usable — hard block above the high mark, and below it annotate rather than reject (tag the turn, spotlight the content, narrow which tools are reachable). False positives are expensive in a support product, so the middle band is where most real traffic should live. Write the verdict into state as a structured value rather than silently mutating text, so every later gate and audit query sees the score that was in force. And decide fail-open versus fail-closed per screen on purpose: if the classifier times out, a sensible default is to keep serving but drop to read-only tools.
Tool-use authorization: subject, verb, object, arguments
This is the heart of guardrail policy, and it is a plain authorization problem in unfamiliar clothes. Every tool call is a request by a subject (which end user, in which session, under which agent identity) to perform a verb (the tool) on an object (the record named in the arguments) under constraints (amount, scope, trust tier). The model chooses the verb and proposes arguments; it must never supply the subject. Read identity from the session, not from a parameter the model fills in.
Permitting a tool is only half the decision — the arguments carry most of the risk. issue_refund is safe at 200 and a fraud vector at 200,000; send_email is safe to a verified customer address and an exfiltration channel to an attacker’s domain. Argument policy therefore needs range checks, allow-lists on destinations and identifiers, ownership checks binding every object to the subject, and shape validation. Prefer making bad calls ungenerable — an enum, a narrower tool, an argument the tool derives itself — over catching them at a gate. Express the whole thing as data rather than scattered if statements: a table keyed by tool name is reviewable by non-programmers, diffable in review, enumerable in an audit, and gives you default deny for the day someone adds a tool and forgets the policy row. The enforcement point in ADK is the tool callback — see the callbacks article — but the callback should be a thin interpreter of this table, not where policy lives.
# Policy is DATA, not prose. One entry per tool; the model never sees it.
TOOL_POLICY = {
'lookup_order': dict(roles={'customer', 'support'}, owner_scoped=True),
'issue_refund': dict(roles={'support'}, owner_scoped=True,
max_amount=3000, approve_above=500),
'export_ledger': dict(roles={'finance'}, allow_untrusted_turn=False),
}
def guard_tool(tool: BaseTool, args: dict, ctx: ToolContext) -> dict | None:
"""Registered as before_tool_callback. Returning a dict short-circuits
the call; returning None lets the real tool run."""
p = TOOL_POLICY.get(tool.name)
if p is None: # default deny
return {'status': 'denied', 'reason': 'not_in_policy'}
role = ctx.state.get('user:role') # from the session, never
owned = ctx.state.get('user:order_ids', []) # from a model argument
if role not in p['roles']:
return {'status': 'denied', 'reason': 'role_not_permitted'}
if p.get('owner_scoped') and args.get('order_id') not in owned:
return {'status': 'denied', 'reason': 'not_your_record'}
if args.get('amount', 0) > p.get('max_amount', float('inf')):
return {'status': 'denied', 'reason': 'over_policy_limit',
'message': 'Above the automatic limit; offer to escalate.'}
if not p.get('allow_untrusted_turn', True) and ctx.state.get('turn:untrusted'):
return {'status': 'denied', 'reason': 'untrusted_context'}
return NoneThe confused deputy — the agent's authority is not the user's
A confused deputy is a privileged component performing an action for a less-privileged requester using its own authority. An ADK agent is a near-perfect deputy: it holds a service account with the union of everything its tools need, and it takes instructions from strings. If the refund tool authenticates as that service account and the ownership check lives only in the model’s reasoning, then anyone who talks the model into refunding someone else’s order gets it — the backend sees a fully authorized request and complies.
Three mitigations, in decreasing strength. Best is delegated identity: the tool acts as the end user with a user-scoped token, so the downstream system enforces its own authorization and the audit log names a human. Next is request-scoped narrowing: the tool derives the object from session state instead of accepting it as an argument, so ‘refund order X’ can only mean an order this session owns. Weakest but still necessary is the gate check comparing argument against state. All three share one property — none trusts the model to have got the subject right, because the model is the component the attacker controls.
When tool output re-enters the prompt
The most underrated boundary in an agent is the return path. A tool result is not an answer; it is attacker-influenceable text about to be concatenated into the next model call. The fetched page, the summarized email, the ticket description a stranger typed — all re-enter with the same syntactic status as your system instructions. That is indirect prompt injection, and it is what makes tool-using agents categorically riskier than chatbots.
Policy has three moves. Sanitize on return: strip markup that hides text, normalize encodings, truncate to what the task needs, and delimit the payload as quoted data. Tag provenance so the turn is flagged untrusted for every downstream gate. Restrict capability in tainted turns — the dual-LLM idea in practical form: reason freely over quarantined data, but do not let that turn reach mutating or outbound tools. A summarization turn that suddenly wants http_get is the signature you are looking for, and the rule blocking it is deterministic and testable. Cap how much untrusted text one result may contribute, too: a 200 KB page in context is both an injection surface and a cost incident.
Output filtering and the egress question
Output policy runs in two directions and teams usually build only one. Outbound-to-the-user filtering is familiar: scrub credential patterns, redact PII the recipient is not entitled to, catch policy-violating content, and check that consequential claims are grounded in something actually retrieved. The harder half is egress: every tool that reaches the network is an exfiltration channel, and the payload need not look like data. A URL path, a query parameter, a filename, an image reference a chat client auto-fetches — all carry bytes outward.
So write egress policy at the level of destinations rather than content: an allow-list of hosts per tool, no arbitrary URLs assembled from model-chosen strings, no rendering of remote references the client will fetch. Pair it with the rule that secrets never enter context at all — injected at the tool boundary from a secret manager, never in prompts, state, or logs — because a filter that has to recognize your API key in base64 has already lost. The strongest output guarantee is the one where the sensitive value was never available to leak.
Gating the irreversible: approval, budgets, and a switch you can flip
Some actions should never be fully autonomous, and the honest criterion is reversibility times blast radius, not a vague feeling that something is sensitive. Deleting a row you can restore in five minutes is cheap; emailing 40,000 customers is not, because there is no undo. Tier actions on that axis: auto-execute, auto-execute-with-notification, execute-after-approval, never-automate. Mechanically the third tier is what long-running tools are for — the call returns a pending handle, the invocation parks, a human decides. The design work is the approval surface, because a rubber stamp is worse than no gate: it adds latency and manufactures the paperwork of oversight without the substance. Show the reviewer concrete argument values, the originating user and session, the conversation excerpt, the injection score for the turn, and what changes on approval. Default to reject on timeout, make rejection one click, and require a reason. Then watch the approval rate: a gate approved 100% of the time is either mis-tiered or unread.
Other attacks are ordinary usage repeated — a per-call-billed tool invoked 400 times, a transfer loop that never terminates — where no single step violates a policy. That needs aggregate limits: cost ceilings per session and per user per day, per-tool call caps, loop-iteration and transfer caps, each with a soft threshold that tags and a hard one that stops. The operational counterpart is the kill switch, and the requirement is that it works without a deploy: gate every risky capability behind a config flag read at call time, so an on-call engineer can disable one tool, agent, or tenant in seconds. Back it with a containment runbook — freeze the session, revoke the tokens, snapshot the event log — and an audit trail where every gate decision is a structured event carrying tool, arguments, rule fired, and identity in force, so forensics and compliance evidence become the same query.
Refusal and escalation: what the agent says when policy says no
A blocked call still has to become a sentence, and this is where good policies are commonly let down. The failure mode is an agent that receives a denial, does not understand it, and either invents a plausible success (‘I’ve processed your refund’ — it has not) or retries variations until something slips through. Both come from an illegible denial. Return structured refusals — machine-readable reason, an explicit statement that the action did not happen, and guidance on what to do next — and the model relays them honestly.
Then design the escalation path, because refusal without a route is a dead end for a legitimate user. Over-limit refunds should offer a human handoff; out-of-scope questions should name the right channel; an ownership failure should say which identity the agent is acting for, so the user can notice they are in the wrong account. Be deliberate about disclosure: enough to be actionable, not enough to be an oracle for enumerating thresholds. ‘That amount needs a supervisor’ is helpful; ‘the limit is 3,000 and you asked for 3,050’ is a tuning signal for the adversary.
Proving it holds: adversarial tests, not assertions
A guardrail you have not attacked is a guardrail you are guessing about, and unit tests on the gate function prove it works when reached, not that it is reached. Test at three levels. Unit: the policy function denies the over-limit argument. Integration: with a scripted or stubbed model that deliberately emits the forbidden call, assert that no side effect occurred — the assertion is that the fake payment gateway was never touched, not that the reply reads politely. Adversarial: replay a corpus of real attacks — direct jailbreaks, injections embedded in fixture documents and emails, multi-turn escalation, encoding tricks — asserting block-or-safe-handling.
Keep that corpus alive: every incident and near miss becomes a permanent case, and the suite runs on every prompt, tool, model, and policy change, because a model swap can silently undo a behavioural defense. Track two numbers together — attack success rate and false-refusal rate on benign traffic — since any guardrail reaches zero on the first by wrecking the second. A policy with no executable test of its own claim is documentation, not a control.