Agent Prompt Architecture in Depth

A system prompt for an agent is a specification, not an instruction

A single-turn prompt is judged against one input. You can hold that input in your head while you write, and if the output is wrong you can see it. An agent's system prompt is a standing contract that gets re-evaluated against every state the loop can reach, including the states you did not design. That difference changes what "good" means. Underspecification in a one-shot prompt shows up as a wrong answer. Underspecification in an agent prompt shows up as inconsistency - the same task handled three different ways in three runs, each defensible, none reproducible.

The most common source is a pronoun or a bare noun that was unambiguous when you wrote it. "Summarise the results" is perfectly clear at turn one. At turn twelve the transcript holds four sets of results, and the model picks whichever is nearest. Every noun in an agent prompt should survive the question which one, at turn thirty? Naming the referent - "summarise the rows returned by the most recent query_orders call" - costs six words and removes a class of run-to-run variance.

The second property a specification needs is coverage of the degenerate branches. A one-shot prompt can stay silent about the empty case because a human reads the output. An agent that receives an empty list with no guidance will invent a policy: widen the date range, retry with looser filters, or answer from memory. Write those branches down explicitly - empty result, ambiguous match, partial success, permission denied, and the request that no available tool covers. Each of them is a decision the model will otherwise make freshly, and differently, every time.

A cheap test: hand the prompt and the tool list to a colleague and ask them to play the agent against a real transcript. Everywhere they stop to ask you a question, the model is guessing.

Tool descriptions are the prompt surface nobody edits

The system prompt is versioned, reviewed and argued over. The tool schemas are written once, by whoever wrote the tool, from the implementation's point of view, and then never touched. Yet they are concatenated into the same context and read by the same model, and in a well-equipped agent they are usually the larger half of the prompt. Start from an artifact rather than a principle. Here is a search tool as it typically ships, and as it should have shipped:

// as written by the person who wrote the endpoint
{
  "name": "search",
  "description": "Search the catalog.",
  "parameters": {
    "q":      {"type": "string"},
    "limit":  {"type": "integer"},
    "filter": {"type": "string"},
    "since":  {"type": "string"}
  }
}

// as written by someone who knows the model reads this
{
  "name": "search_products",
  "description": "Full-text search over the product catalog. Returns matching
                  products, newest first. Use search_orders instead if the user
                  is asking about something they already bought.",
  "parameters": {
    "query":             {"type": "string",
                          "description": "Free-text search terms, not a question."},
    "max_results_total": {"type": "integer",
                          "description": "Hard cap on products returned across the
                                          whole search. Default 20, maximum 100."},
    "status_equals":     {"type": "string", "enum": ["active", "discontinued"]},
    "listed_after":      {"type": "string",
                          "description": "ISO-8601 date, e.g. 2026-03-14."}
  }
}

Nothing in that rewrite is an instruction. Every difference is a name or a type. limit does not say whether it is a page size, a hard ceiling, or a per-document cap, so a model that wants forty results will set it to 40 and paginate forty times; max_results_total is self-describing and the pagination loop never starts. filter is a string, which is an open invitation to write natural language into it; an enum makes the illegal value unrepresentable. since has no format, so the model picks one, and it picks a different one on Tuesdays.

The worst offender is the shared bare name. If your order tool takes id and your customer tool takes id, the model will eventually cross them, and no amount of instruction stops it reliably. Rename to order_id and customer_id and the confusion largely evaporates, because the argument now carries its own type in its name. This is the general shape of the trade: a renamed parameter changes behaviour on every future turn for zero recurring tokens, while a sentence in the system prompt telling the model to be careful with that parameter costs tokens forever and works less well. Descriptions answer when should I call this and when should I not; parameter names and their descriptions answer what exactly goes here. Evolving these safely once agents depend on them is covered in tool schema versioning, and the separate problem of a catalog too large to bind at all is covered in tool selection.

Give the model enough to choose, not just enough to call

Tool descriptions are written one at a time, but selection is a property of the set. Two tools that are each individually well documented can still be undecidable together, and that is the failure you actually see in traces: not a malformed call, but a confident call to the wrong tool.

Say you have search_docs and search_tickets, and both descriptions begin "search the knowledge base for...". Everything about each one is accurate. Nothing in either says which to reach for. The fix is a discriminator written in both directions: use search_docs for published product documentation; if the user is describing a symptom rather than asking how something works, search_tickets first, and the mirror clause in the other tool. Overlapping pairs are the cheapest thing to audit in an agent - list the tools, find the pairs whose descriptions could be swapped without either becoming false, and write one sentence into each.

Choice also means the non-tool options have to be nameable. If the prompt never says that asking the user is a legal move, or that answering directly without a tool is a legal move, then every turn looks like a tool-selection problem, and the model will pick the least-bad tool rather than admit the set does not cover the request. Give both of those an explicit affordance, ideally as real tools (ask_user, answer_directly) so they appear in the same list the model is choosing from.

Two more things worth stating because models will honour them. Relative cost: if one route is slow, metered, or rate-limited, say so and say what to prefer. Preconditions: "call get_order first; issue_refund requires an order in state SHIPPED" converts a failed call and a recovery round trip into a correct first call, which is the cheapest kind of prompt engineering there is.

Stating what the agent must not do

Negative instructions are the weakest instrument available, and they get weaker as the run gets longer. A prohibition has to be recalled and applied at every step, while competing with a user who is pushing, with the model's own earlier reasoning, and with a context window in which the prohibition is now thirty thousand tokens behind. A positive instruction only has to be followed once, when the moment arrives.

Rank the options by strength before writing anything. Strongest: the capability does not exist - there is no tool, no credential, no route. Next: the capability exists but something outside the model refuses the call. Weakest: the prompt says don't. If you do not want an agent deleting rows in production, the durable version is a read-only connection; the prompt version is a sentence it will follow most of the time. Removing a tool from the bound set is not a lesser measure than describing its misuse, it is a stronger one, and it is also cheaper - the tokens that described the tool stop being billed.

Where you genuinely cannot remove the capability, three things make a negative work better. State the positive alternative in the same breath: "do not guess an ID; call lookup_customer" is followed far more reliably than "do not guess an ID", because the second leaves the model with a problem and no move. Attach the rule to the moment rather than the preamble - a caution that lives in issue_refund's description sits directly beside the decision it governs, whereas the same caution in a rules list at the top competes with everything that arrived since. And keep the list short: ten prohibitions read as a specification, forty read as a wish, dilute each other, and conveniently enumerate the interesting attack surface for anyone reading injected content.

The enforcement side of this - action policies, approval gates, capability isolation - belongs to agent guardrails and sandboxing. What the prompt contributes is making the permitted path obvious enough that the enforcement rarely has to fire.

Error strings are prompt input

Look at what your tools return when they fail, then remember that the model reads it as its next observation. This is prompt text delivered at the exact moment the model most needs help, and it is almost always written for a human reading a log. Compare:

{"error": "Bad Request"}

{"error": "invalid input"}

{"error": "start_date: expected an ISO-8601 date such as 2026-03-14,
           received '03/14/2026'. Reformat the argument and call again."}

{"error": "No order with id ORD-99213 exists. This will not change on retry.
           Ask the user to confirm the order number, or call
           search_orders with the customer's email."}

The first two cost you between one and four extra turns each, and sometimes a loop. The last two cost one turn, and the fourth costs none if the model takes the suggested branch. Four things separate them. Name the offending input by parameter, so the model knows which of five arguments to change. State what was expected and include one literal valid value - a concrete example in an error string outperforms three sentences of format instruction in the system prompt, because it arrives when it is relevant. Give a retry verdict, explicitly: a model facing an ambiguous failure retries by default, so "this will not change on retry" is the single highest-value field you can add. And when retry is pointless, name the next move.

Two failure modes to avoid on the way. Never suggest an action the agent cannot take - "check the admin console" earns you a hallucinated tool call or an apology loop, because the model has been told to do something and given no way to do it. And truncate: a four-thousand-token stack trace does not just waste budget, it displaces the actual task from the model's attention for the rest of the run. Return the first cause and a reference, not the trace.

One subtler property is stability. If the same underlying failure comes back worded differently each time, the model cannot tell that it is stuck, because nothing in its context repeats. Deterministic error text is what lets a model notice it is going in circles. The machinery around all this - classifying failures, backoff, budgets, and deciding which failures should never reach the model at all - is tool retry and recovery; this section is only about the words.

Output contracts at each step, not only at the end

Most teams put a schema on the final answer and leave every intermediate turn free-form. That is backwards for diagnosis. The final answer is the one place a human is likely to notice a problem anyway; the intermediate turns are where the run silently went wrong, and they are unstructured precisely where structure would have paid.

A per-step contract is small: alongside each action, the model emits the sub-goal it believes it is working on, why it chose this tool, and what it expects the call to return. Two or three enumerable fields, not a free-text reasoning blob - a field that accepts anything constrains nothing and is not a contract. The expectation field is the one that earns its keep twice: it makes traces greppable, and it gives the model something to compare the actual result against. A model that has stated what it expects and then receives something else has a hook to notice; without the statement, a surprising result is absorbed without comment and the run continues on a false premise.

The cost is real - tokens on every turn, and some models get stiffer when made to fill in fields - so the honest guidance is to contract the steps where drift is expensive (anything that mutates state, anything that hands off) and leave read-only exploration loose. When a step's output fails to parse, handle it in the scaffold rather than by asking the model more politely in the prompt. The schema mechanics, decoding constraints and validation strategy are covered in structured output; what belongs here is the decision about which steps get a contract at all.

Encourage the model to stop

Agents fail by looping far more often than by giving up too early. Giving up early is visible, cheap, and usually recoverable by a human. A loop burns the budget, fills a transcript nobody will read, and frequently ends in a confident summary of work that never happened. If you only harden one thing in an agent prompt, harden the stopping behaviour.

Failure to stop is usually a prompt gap rather than a reasoning failure. The prompt describes what success looks like and is silent about everything else, so every state that is not success reads to the model as "not finished yet, keep going". Four moves fix most of it.

Enumerate the terminal states, plural. Done, blocked-needs-user, blocked-needs-permission, no-such-thing, out-of-budget. Give each one a concrete shape to emit. When only one ending is described, the model will strain toward it past the point of usefulness.

Make failure respectable. Prompts written in an encouraging register - be thorough, be persistent, do not give up - produce agents that will not stop. Say plainly that returning "I could not do X because Y" is a correct outcome, and that it is preferred to a plausible guess. Tone is load-bearing here in a way it is not in single-turn prompting.

Define done as an artifact, not a feeling. "You are done when the comment is posted and you have its URL" is checkable by the model in a way that "when the task is complete" never is.

Say what a repeat means. Models re-issue tool calls because acting feels like progress. A single line - "if a call returns the same result as an identical earlier call, that is not new information: change approach or stop" - removes the most common loop shape there is. Pair it with a running budget signal in the context ("4 of 20 tool calls used"), which measurably shifts a model toward wrapping up; a limit stated once at the top of the prompt is forgotten by the time it matters.

All of that makes the agent stop gracefully. It does not make it stop. The hard turn cap, wall-clock limit and spend ceiling belong in the scaffold and must exist regardless of how good the prompt is. Loop shapes that come from a plan regenerating itself rather than from tool repetition are a different animal, handled in planner design; the reason-act cycle those loops run inside is described in the ReAct pattern.

Context growth quietly changes a prompt you already tuned

The prompt you validated at turn three is not the prompt in force at turn thirty. The bytes are identical; the neighbourhood is not. Two effects bite prompt authors specifically. Dilution: a rule stated once, early, now competes with thousands of tokens of tool output and loses on recency. Self-anchoring: a guess the model made at turn four, restated a few times since, is now indistinguishable from a retrieved fact, and your instruction to verify claims is being applied to everything except the model's own history.

Three practical responses. Re-assert the two or three load-bearing rules near the end of the assembled context rather than only at the top. Keep the immovable part of the prompt byte-stable so it stays cacheable and so its position does not shift under the model. And treat whatever summarises the history as a prompt you own and test - it decides which of your commitments survive, and a lossy summariser undoes good prompt work silently. The assembly mechanics themselves are covered in context packs and context compaction.

Prompt or scaffold - where a rule belongs

The general rule is short: anything correctness-critical is enforced in code. If a violation would be an incident - money moved, data deleted, a message sent to a customer, a limit with a number in it - the prompt is not where that rule lives, because a prompt is a strong prior and never a guarantee.

The part that gets missed is that this is not an argument for leaving the rule out of the prompt. An agent constrained only in code keeps proposing the forbidden action, gets refused, and tries a variation - the constraint is honoured and the run still fails, expensively. Written in the prompt and enforced in code, the agent takes the permitted path the first time and the check almost never fires. Prompt-only is appropriate for preferences, tone, ordering heuristics and when to ask a clarifying question, where being wrong costs a turn rather than an incident. For the enforcement architecture itself, see agent guardrails.

The layers of an agent prompt, and where each is covered

The diagram below is the full stack an agent prompt assembles on every turn. This article has argued the parts that are specific to acting - the system prompt as specification, tools, negatives, errors, contracts, stopping. The rest are real layers with their own articles.

System prompt. Role, standing rules, output conventions. Kept stable so it can be cached and so its position does not move.

Tools registered. Names, parameter schemas and descriptions - the largest and least-edited part of most agent prompts.

Memory injection. Retrieved facts and prior-session state, covered in agent memory layers.

Response format. Per-step and final contracts, covered in structured output.

Safety rules. Refusals and escalation triggers, with enforcement in guardrails.

Few-shot examples. For agents these are usually whole traces rather than input-output pairs; selection strategy is in dynamic few-shot.

Chain-of-thought hint. Increasingly redundant on reasoning models and occasionally harmful, discussed in chain-of-thought prompting.

Budget signals. A live remaining-steps count in context, which is the stopping lever from the previous section.

Versioning and eval. Registry and evaluation.

Multi-model portability. Worth a caution: agent prompts port considerably worse than single-turn ones, because they encode assumptions about tool-calling behaviour rather than about text. How eagerly a model calls a tool at all, whether it emits parallel calls, how it reacts to a tool error, how much unprompted reasoning it produces before acting, and how readily it decides it is finished all vary by model and by version. When you swap models, re-run the loop tests - call counts, stop rates, recovery rates - not only the answer-quality tests, because the answers can hold steady while the number of tool calls doubles.

A worked prompt, and where each rule shows up in it

Here is a support agent's system prompt with the ideas above applied. Nothing in it is long, and most of the work is in nouns rather than sentences.

You are the support agent for [Company]. You resolve customer issues using
the tools below. You do not speculate.

Definition of done. You are done when you have either (a) answered using a
value you retrieved from a tool, (b) created a ticket and have its ID, or
(c) reported that you cannot proceed and why. (c) is a correct outcome and
is always preferred to a guess.

State handling.
- No matching order: do not widen the search. Ask the user for the order ID.
- Ambiguous match: list the candidates and ask; do not pick one.
- A tool says an error will not change on retry: stop calling it.
- A call returns the same result as an identical earlier call: that is not
  new information. Change approach or finish with (c).

Never state a policy from memory. Every policy claim in your answer must
come from lookup_policy in this session.

Tools:
- lookup_policy(topic_slug: enum)         -> policy text. Use for refunds,
    returns, warranty. Not for order status.
- get_order(order_id: str)                -> order details, including status.
    Call before create_ticket so the ticket carries the real status.
- create_ticket(summary: str,
                priority: enum[low|med|high]) -> ticket_id.
- ask_user(question: str)                 -> the user's reply.

Each turn, emit: {step_goal, tool, expected} before the call.
Budget: you have {n} of 12 tool calls remaining.

Read it against the sections above. The terminal states are enumerated and failure is named as correct, so the model has somewhere to go other than "try again". The repeat rule is stated in one line. topic_slug is an enum rather than a free string, so a malformed topic is unrepresentable rather than discouraged. get_order carries its precondition in its own description instead of in a rules list. The one absolute - never state policy from memory - is short enough to survive a long context, and in production it would also be checked outside the model, because an unsourced policy claim to a customer is an incident rather than a style problem.

What is not in the prompt matters as much. There is no refund tool, because this agent is not permitted to issue refunds, and removing it is stronger and cheaper than a paragraph explaining when not to use it. There is no instruction about date formats, because the tools that take dates say so in their own descriptions and their errors quote a valid example. There is no "think step by step", because the per-step contract already forces the model to state a goal and an expectation before acting.

Iterating without regressing the cases you already fixed

Agent prompts accumulate scar tissue. Every incident adds a bullet, and after two quarters the prompt is three thousand tokens of narrow prohibitions, several of which contradict each other and none of which can be traced to the case that caused them. The habit that prevents this is to treat the trace, not the bullet, as the durable artifact: every change starts from a specific failed run, and that run is captured as a stored case before the prompt is edited. The prompt then becomes something you can re-derive rather than something you can only append to.

Change one thing per iteration, because agent prompts have long-range interactions that single-turn prompts do not. A sentence added to discourage redundant tool calls will also discourage the necessary first call; a firmer stopping rule will produce premature give-ups on genuinely long tasks. With two changes in flight you cannot attribute either effect.

Prefer the smallest surface that can fix the problem, in this order: the tool description, the error string, the parameter name, and only then the system prompt. The first three apply exactly at the decision point and cost nothing on turns where that tool is not in play, whereas system prompt text is paid on every call of every run forever. A surprising share of behaviours that look like they need instruction turn out to need a better noun.

Finally, consolidate on a schedule. Periodically rewrite the prompt from the case set instead of editing it, then replay the cases. A prompt that only grows will eventually contradict itself, and a self-contradictory agent prompt does not fail loudly - it produces inconsistency across turns, which arrives in your inbox as "the agent is flaky". Be suspicious of a change that fixes exactly one trace and moves nothing else; that is usually a memorised patch rather than a fix. How to build and score the case set - metrics, judges, sampling, regression gates - is prompt evaluation, and the machinery for versioning and staged rollout is the prompt registry.