Three caches, three completely different risk profiles
‘Caching’ in an agent stack is three unrelated mechanisms wearing one word, and conflating them is how teams end up with an exciting outage. Sort them by how a hit is decided, because that determines the blast radius.
| Layer | What is cached | Hit rule | If it goes wrong |
|---|---|---|---|
| Prefix / context cache | The tokenised stable head of the request | Byte-identical prefix | Nothing — you silently pay full price |
| Tool-result cache | The return value of one tool call | Exact match on a key you build | Stale data, or one user’s data served to another |
| Semantic cache | A whole final answer | Embedding similarity above a threshold | A confidently wrong answer to a question nobody asked |
The ordering is not accidental. Prefix caching is exact: the provider matches the bytes or it does not, so a bug costs money and never correctness. Tool caching is exact on a key you chose, so its failures are key-design failures — forget a dimension and you serve the wrong row. Semantic caching decides a hit by judgment, so it can be wrong even when every line of your code is right. Do them in that order, and be honest that most agents should stop after the first two.
What actually makes a prefix cacheable
Every major provider now bills a repeated request prefix at a steep discount — the mechanics differ, the constraint does not. Gemini offers both implicit caching (automatic, best-effort) and explicit context caching, where you register a block of content and reference it by handle for a TTL you choose. Newer ADK releases have been growing first-class configuration for Gemini context caching at the app level; if your version exposes it, use it rather than hand-rolling, and if it does not, everything below still applies because it is all about what you put in the request.
Three rules survive every provider. First, the match is on a prefix, not a set: the cached span starts at token zero and runs forward, so one changed byte at position 12 invalidates everything after it. Second, there is a minimum length — typically on the order of a thousand tokens — below which nothing is cached at all, which is why a fat, stable instruction is an asset rather than a liability. Third, entries expire — implicit caches within minutes, kept alive by use; explicit ones on the TTL you set. An agent relying on implicit caching at low traffic will miss constantly no matter how clean its prefix is.
Assembling an ADK request so the prefix survives
The practical implication is an ordering discipline: stable content first, volatile content last. In ADK terms the head of every request is the agent’s instruction plus the declarations ADK derives from tools; the tail is the conversation ADK assembles from session events. The head is yours to keep still.
from google.adk.agents import LlmAgent
# Cacheable: a module-level constant, and a tool list whose order is fixed
# at import time rather than computed per request.
agent = LlmAgent(
name="support_agent",
model="gemini-2.5-flash",
instruction=SYSTEM_PROMPT,
tools=[lookup_order, refund_policy, escalate],
)
# Not cacheable: the instruction differs on every single request.
# instruction=f"You are a support agent. It is {datetime.now()}."
One ADK-specific trap deserves naming. Instruction strings support {state_key} templating, which is genuinely useful and quietly catastrophic for the prefix: any templated value that varies per user or per turn re-renders the system instruction and kills the cache. The same goes for an InstructionProvider callable reading mutable state. Per-user context belongs in the first user message — below the cacheable head, not inside it.
A volatility audit for the cached head
Prefix cache failures are invisible: the request succeeds, the answer is correct, and the bill quietly doubles. Audit the head of your request explicitly for anything that moves.
| Volatile thing in the head | Why it appears | Fix |
|---|---|---|
| Current date or time | ‘So the model knows today’s date’ | Move it into the user turn, or round it to the day |
| Session or request ID | Copied in for log correlation | Correlate in traces, not in the prompt |
| User name or tenant in the instruction | Personalisation via templating | Put it in the first user message |
| Tool list built from a dict or a set | Dynamic toolsets, filtered per user | Sort deterministically; prefer a fixed catalog |
| Retrieved documents pasted above the instruction | RAG glued on at the top | Retrieved chunks go last, they are the volatile part |
Each was added for a reasonable local reason by someone not thinking about the prefix, so the defence is a review rule rather than a test: nothing enters the system instruction or the tool declarations without a justification for why it cannot live one layer lower. Where a dynamic toolset is genuinely required, accept that you have several distinct prefixes and keep them few — three tool catalogs is three cache entries, one per user is none.
Tool-result caching: what you are actually storing
Tool caching has the best ratio of savings to risk for most agents, because agent tool calls are pathologically repetitive: a re-planning loop calls the same lookup twice, a follow-up needs the same order record, two sub-agents independently enrich the same customer. The interception points are the tool-boundary callbacks — a before-hook that returns a value to short-circuit the call, an after-hook that records the result. The callback mechanics are covered in the callbacks article; what matters here is what you put in the box.
Cache the tool’s return value, not a rendered string, and store it with metadata you will need later: the key material, the write time, the schema version, and the source system’s own freshness marker if it has one (an ETag, a last_modified, a row version). That last one turns a blind TTL into cheap revalidation: on a stale-ish hit you ask the upstream ‘changed since X?’ and pay for a conditional request rather than a full one. Never cache a streaming fragment, and never cache an object holding a live handle — a file, a connection, an expiring signed URL. Serialise to something inert.
Key construction is the whole game
A tool cache is only as correct as its key, and the failure mode is always the same: a dimension that affects the result was left out, so two genuinely different calls collide. Build the key from every input that can change the answer — including inputs the model never supplied.
import hashlib, json
KEY_VERSION = "v3" # bump when the tool's logic or output shape changes
def tool_cache_key(tool_name: str, args: dict, tool_context) -> str:
canonical = json.dumps(args, sort_keys=True,
separators=(",", ":"), default=str)
st = tool_context.state
tenant = st.get("user:tenant_id", "anon")
scopes = ",".join(sorted(st.get("user:scopes", [])))
locale = st.get("user:locale", "en-US")
material = f"{KEY_VERSION}|{tool_name}|{canonical}|{tenant}|{scopes}|{locale}"
return hashlib.sha256(material.encode()).hexdigest()
Four things that look pedantic and are not. Canonicalise: json.dumps with sort_keys=True and fixed separators, or {"a":1,"b":2} and {"b":2,"a":1} become different keys for the same call, halving your hit rate for nothing. Version the key: when you change what the tool returns, old entries are not stale, they are wrong shape, and bumping KEY_VERSION retires them atomically at deploy time. Normalise arguments the model is sloppy about — case, whitespace, 1 versus 1.0. And include identity, which the scoping section below takes up.
TTL from change rate, and invalidation that actually invalidates
A TTL is an assertion about how fast the underlying data changes and how much staleness the answer tolerates. Most teams pick 300 seconds because it is a round number. Derive it instead, from two questions: how quickly can this datum change, and what happens to the user if they see the previous value?
| Data | Change rate | Reasonable TTL |
|---|---|---|
| Country and currency reference tables | Yearly | Hours to days |
| Product catalog and policy text | Daily | Tens of minutes, plus event invalidation |
| Order status mid-fulfilment | Minutes | Seconds, or do not cache |
| Account balance, inventory count | Continuous | Do not cache |
A TTL is not invalidation; it is a bound on how long you are willing to be wrong. Real invalidation is a write path: whoever mutates the record deletes the affected keys, or bumps a generation counter that participates in the key. The generation trick is the one that scales — embed gen:{tenant}:{entity} in the key material and increment it on write, and every dependent entry dies in one operation with no key enumeration. Where the mutation happens in a system you do not control, a short TTL is your only lever; be honest about which situation you are in rather than pretending a TTL is a correctness guarantee.
Errors, empties, and the things you must never cache
The default reflex is to cache successes and ignore everything else, which leaves the two cases that hurt most. Negative results are worth caching — a lookup that legitimately returns ‘no such customer’ is a real answer, and without one a hallucinated ID in a retry loop will hammer your upstream with the same doomed query. Give empties a shorter TTL than positives, because absence flips to presence more readily than a value changes.
Errors are the opposite. Never cache a timeout, a 5xx, a rate-limit rejection, or a transport failure: that converts a transient outage into a persistent one and defeats the retry that would have succeeded. The never-cache list is worth writing down: authorisation checks (the answer changes the moment someone is offboarded), anything with a nonce or one-time token, anything the user is about to act on financially, and anything whose value derives from being read now. A tool named get_current_* is a strong hint that it should not have a cache in front of it.
Semantic caching: the mechanism
Semantic caching skips the model entirely. Embed the incoming query, search a vector store of previously answered queries, and if the nearest neighbour sits above a similarity threshold, return its stored answer. Where a thousand users ask forty distinct questions in a hundred phrasings the arithmetic is striking: a sub-50 ms lookup replaces a multi-second generation, and the hit rate can exceed half of all traffic.
The implementation sits at the model boundary or, better, in front of the agent entirely — there is no point running a planner whose output you intend to discard. Three decisions matter more than the code. The threshold is the risk dial and must be tuned against a labelled set of query pairs rather than guessed; you are choosing a false-hit rate, so choose it with data. The embedding model must be pinned and versioned into the store, because swapping it silently rotates the geometry of your cache. And the write policy should be selective: store only answers produced without user-specific tool calls, that the agent did not flag as uncertain, and that no user subsequently corrected.
Why semantic caching is riskier than it looks
Embedding proximity is not semantic equivalence, and the places where they diverge are exactly the places where being wrong is expensive. Embeddings are notoriously weak on the small tokens that carry the most meaning: not, except, numbers, dates, and named entities. ‘Can I cancel my order?’ and ‘Can I cancel my order after it ships?’ sit very close in vector space and have opposite answers. ‘What is the fee for a wire under $10,000?’ and ‘over $10,000?’ are nearly identical strings.
Worse, the failure is silent and confident. A stale tool result at least produces an answer that was true; a false semantic hit produces a fluent, authoritative answer to a question the user did not ask, with no signal in the response that anything happened. The mitigations are unglamorous: a conservative threshold accepting a lower hit rate; a hard exclusion list for queries containing negation, comparatives, currency amounts, dates, or entity IDs; restriction to a curated FAQ domain rather than open traffic; and offline sampling where a stronger model judges whether cached answers actually answered the queries that hit them. Without that last loop you have no idea what your false-hit rate is.