An agent is a machine for doing the same work over and over. The same system instruction and tool declarations are re-sent and re-billed on every turn of every session; the same five questions arrive from a thousand users; the same slow enrichment API is called with the same account ID by four agents in an hour. Caching is the single largest cost lever an ADK agent has, and unlike model choice or prompt compression it costs nothing in output quality — provided you cache the right thing at the right layer. This piece works through all three layers in ascending order of danger: the provider-side prefix cache (exact, free, mostly a matter of not breaking it), the tool-result cache (exact, but the key is where correctness lives), and the semantic cache (approximate, seductive, and the one that will eventually serve somebody else’s answer to your user). Along the way: TTLs derived from change rate rather than habit, invalidation that actually invalidates, and how to tell whether any of it saved money.
Three caches, three completely different risk profiles
‘Caching’ in an agent stack is three unrelated mechanisms wearing one word, and conflating them is how teams end up with an exciting outage. Sort them by how a hit is decided, because that determines the blast radius.
| Layer | What is cached | Hit rule | If it goes wrong |
|---|---|---|---|
| Prefix / context cache | The tokenised stable head of the request | Byte-identical prefix | Nothing — you silently pay full price |
| Tool-result cache | The return value of one tool call | Exact match on a key you build | Stale data, or one user’s data served to another |
| Semantic cache | A whole final answer | Embedding similarity above a threshold | A confidently wrong answer to a question nobody asked |
The ordering is not accidental. Prefix caching is exact: the provider matches the bytes or it does not, so a bug costs money and never correctness. Tool caching is exact on a key you chose, so its failures are key-design failures — forget a dimension and you serve the wrong row. Semantic caching decides a hit by judgment, so it can be wrong even when every line of your code is right. Do them in that order, and be honest that most agents should stop after the first two.
What actually makes a prefix cacheable
Every major provider now bills a repeated request prefix at a steep discount — the mechanics differ, the constraint does not. Gemini offers both implicit caching (automatic, best-effort) and explicit context caching, where you register a block of content and reference it by handle for a TTL you choose. Newer ADK releases have been growing first-class configuration for Gemini context caching at the app level; if your version exposes it, use it rather than hand-rolling, and if it does not, everything below still applies because it is all about what you put in the request.
Three rules survive every provider. First, the match is on a prefix, not a set: the cached span starts at token zero and runs forward, so one changed byte at position 12 invalidates everything after it. Second, there is a minimum length — typically on the order of a thousand tokens — below which nothing is cached at all, which is why a fat, stable instruction is an asset rather than a liability. Third, entries expire — implicit caches within minutes, kept alive by use; explicit ones on the TTL you set. An agent relying on implicit caching at low traffic will miss constantly no matter how clean its prefix is.
Assembling an ADK request so the prefix survives
The practical implication is an ordering discipline: stable content first, volatile content last. In ADK terms the head of every request is the agent’s instruction plus the declarations ADK derives from tools; the tail is the conversation ADK assembles from session events. The head is yours to keep still.
from google.adk.agents import LlmAgent
# Cacheable: a module-level constant, and a tool list whose order is fixed
# at import time rather than computed per request.
agent = LlmAgent(
name="support_agent",
model="gemini-2.5-flash",
instruction=SYSTEM_PROMPT,
tools=[lookup_order, refund_policy, escalate],
)
# Not cacheable: the instruction differs on every single request.
# instruction=f"You are a support agent. It is {datetime.now()}."
One ADK-specific trap deserves naming. Instruction strings support {state_key} templating, which is genuinely useful and quietly catastrophic for the prefix: any templated value that varies per user or per turn re-renders the system instruction and kills the cache. The same goes for an InstructionProvider callable reading mutable state. Per-user context belongs in the first user message — below the cacheable head, not inside it.
A volatility audit for the cached head
Prefix cache failures are invisible: the request succeeds, the answer is correct, and the bill quietly doubles. Audit the head of your request explicitly for anything that moves.
| Volatile thing in the head | Why it appears | Fix |
|---|---|---|
| Current date or time | ‘So the model knows today’s date’ | Move it into the user turn, or round it to the day |
| Session or request ID | Copied in for log correlation | Correlate in traces, not in the prompt |
| User name or tenant in the instruction | Personalisation via templating | Put it in the first user message |
| Tool list built from a dict or a set | Dynamic toolsets, filtered per user | Sort deterministically; prefer a fixed catalog |
| Retrieved documents pasted above the instruction | RAG glued on at the top | Retrieved chunks go last, they are the volatile part |
Each was added for a reasonable local reason by someone not thinking about the prefix, so the defence is a review rule rather than a test: nothing enters the system instruction or the tool declarations without a justification for why it cannot live one layer lower. Where a dynamic toolset is genuinely required, accept that you have several distinct prefixes and keep them few — three tool catalogs is three cache entries, one per user is none.
Tool-result caching: what you are actually storing
Tool caching has the best ratio of savings to risk for most agents, because agent tool calls are pathologically repetitive: a re-planning loop calls the same lookup twice, a follow-up needs the same order record, two sub-agents independently enrich the same customer. The interception points are the tool-boundary callbacks — a before-hook that returns a value to short-circuit the call, an after-hook that records the result. The callback mechanics are covered in the callbacks article; what matters here is what you put in the box.
Cache the tool’s return value, not a rendered string, and store it with metadata you will need later: the key material, the write time, the schema version, and the source system’s own freshness marker if it has one (an ETag, a last_modified, a row version). That last one turns a blind TTL into cheap revalidation: on a stale-ish hit you ask the upstream ‘changed since X?’ and pay for a conditional request rather than a full one. Never cache a streaming fragment, and never cache an object holding a live handle — a file, a connection, an expiring signed URL. Serialise to something inert.
Key construction is the whole game
A tool cache is only as correct as its key, and the failure mode is always the same: a dimension that affects the result was left out, so two genuinely different calls collide. Build the key from every input that can change the answer — including inputs the model never supplied.
import hashlib, json
KEY_VERSION = "v3" # bump when the tool's logic or output shape changes
def tool_cache_key(tool_name: str, args: dict, tool_context) -> str:
canonical = json.dumps(args, sort_keys=True,
separators=(",", ":"), default=str)
st = tool_context.state
tenant = st.get("user:tenant_id", "anon")
scopes = ",".join(sorted(st.get("user:scopes", [])))
locale = st.get("user:locale", "en-US")
material = f"{KEY_VERSION}|{tool_name}|{canonical}|{tenant}|{scopes}|{locale}"
return hashlib.sha256(material.encode()).hexdigest()
Four things that look pedantic and are not. Canonicalise: json.dumps with sort_keys=True and fixed separators, or {"a":1,"b":2} and {"b":2,"a":1} become different keys for the same call, halving your hit rate for nothing. Version the key: when you change what the tool returns, old entries are not stale, they are wrong shape, and bumping KEY_VERSION retires them atomically at deploy time. Normalise arguments the model is sloppy about — case, whitespace, 1 versus 1.0. And include identity, which the scoping section below takes up.
TTL from change rate, and invalidation that actually invalidates
A TTL is an assertion about how fast the underlying data changes and how much staleness the answer tolerates. Most teams pick 300 seconds because it is a round number. Derive it instead, from two questions: how quickly can this datum change, and what happens to the user if they see the previous value?
| Data | Change rate | Reasonable TTL |
|---|---|---|
| Country and currency reference tables | Yearly | Hours to days |
| Product catalog and policy text | Daily | Tens of minutes, plus event invalidation |
| Order status mid-fulfilment | Minutes | Seconds, or do not cache |
| Account balance, inventory count | Continuous | Do not cache |
A TTL is not invalidation; it is a bound on how long you are willing to be wrong. Real invalidation is a write path: whoever mutates the record deletes the affected keys, or bumps a generation counter that participates in the key. The generation trick is the one that scales — embed gen:{tenant}:{entity} in the key material and increment it on write, and every dependent entry dies in one operation with no key enumeration. Where the mutation happens in a system you do not control, a short TTL is your only lever; be honest about which situation you are in rather than pretending a TTL is a correctness guarantee.
Errors, empties, and the things you must never cache
The default reflex is to cache successes and ignore everything else, which leaves the two cases that hurt most. Negative results are worth caching — a lookup that legitimately returns ‘no such customer’ is a real answer, and without one a hallucinated ID in a retry loop will hammer your upstream with the same doomed query. Give empties a shorter TTL than positives, because absence flips to presence more readily than a value changes.
Errors are the opposite. Never cache a timeout, a 5xx, a rate-limit rejection, or a transport failure: that converts a transient outage into a persistent one and defeats the retry that would have succeeded. The never-cache list is worth writing down: authorisation checks (the answer changes the moment someone is offboarded), anything with a nonce or one-time token, anything the user is about to act on financially, and anything whose value derives from being read now. A tool named get_current_* is a strong hint that it should not have a cache in front of it.
Semantic caching: the mechanism
Semantic caching skips the model entirely. Embed the incoming query, search a vector store of previously answered queries, and if the nearest neighbour sits above a similarity threshold, return its stored answer. Where a thousand users ask forty distinct questions in a hundred phrasings the arithmetic is striking: a sub-50 ms lookup replaces a multi-second generation, and the hit rate can exceed half of all traffic.
The implementation sits at the model boundary or, better, in front of the agent entirely — there is no point running a planner whose output you intend to discard. Three decisions matter more than the code. The threshold is the risk dial and must be tuned against a labelled set of query pairs rather than guessed; you are choosing a false-hit rate, so choose it with data. The embedding model must be pinned and versioned into the store, because swapping it silently rotates the geometry of your cache. And the write policy should be selective: store only answers produced without user-specific tool calls, that the agent did not flag as uncertain, and that no user subsequently corrected.
Why semantic caching is riskier than it looks
Embedding proximity is not semantic equivalence, and the places where they diverge are exactly the places where being wrong is expensive. Embeddings are notoriously weak on the small tokens that carry the most meaning: not, except, numbers, dates, and named entities. ‘Can I cancel my order?’ and ‘Can I cancel my order after it ships?’ sit very close in vector space and have opposite answers. ‘What is the fee for a wire under $10,000?’ and ‘over $10,000?’ are nearly identical strings.
Worse, the failure is silent and confident. A stale tool result at least produces an answer that was true; a false semantic hit produces a fluent, authoritative answer to a question the user did not ask, with no signal in the response that anything happened. The mitigations are unglamorous: a conservative threshold accepting a lower hit rate; a hard exclusion list for queries containing negation, comparatives, currency amounts, dates, or entity IDs; restriction to a curated FAQ domain rather than open traffic; and offline sampling where a stronger model judges whether cached answers actually answered the queries that hit them. Without that last loop you have no idea what your false-hit rate is.
Scoping: the key mistake that becomes a data breach
Every cache in an agent is a shared surface, and the canonical incident is always the same shape. A team caches a tool result keyed on ("get_account_summary", {}) — empty arguments, because the tool reads the current user from session state. The first user warms the cache; every subsequent user gets that user’s balance. The code is correct, the tool is correct, the key is wrong, and it is a data breach.
The rule is that the cache key must include every input to the authorisation decision, not just every input to the function signature. Concretely: the tenant, the principal where the result is personal, the permission scopes, any row-level security context, plus locale where it changes the content. That is why the key builder above reads user:-prefixed state and not only the model-supplied arguments. Then decide scope deliberately per cache: app-scoped for genuinely universal data, tenant-scoped for organisational data, user-scoped for anything personal. When in doubt scope narrower — a lower hit rate costs money, a scoping bug costs customers.
Stampede, single-flight, and the in-process dictionary lie
Two operational problems arrive together at scale. The first is the stampede: a popular entry expires and the fifty requests in flight at that instant all miss and all call the expensive upstream at once — frequently the exact pattern that trips the rate limit you were caching to avoid. Take the fixes from the web-caching world wholesale: single-flight (the first miss takes a lock and computes, the rest wait on the result), stale-while-revalidate (serve the expired value and refresh in the background), and jittered expiry so entries written together do not expire together.
The second is quieter. A module-level dict works beautifully in development and lies to you in production, because your agent runs as N replicas behind a load balancer and autoscaling recycles them. Your hit rate drops by roughly a factor of N, deploys start cold, and single-flight protects only one process. Anything that matters belongs in a shared store with real TTL support and an atomic set-if-absent for the lock. An in-process layer in front of it is a fine optimisation — treat it as an L1 with a very short TTL, not as the cache.
Measuring: hit rate is the vanity metric, cost saved is the real one
Hit rate is easy to measure and easy to misread. An 80% hit rate on a cheap, fast tool saves nothing worth having; a 12% hit rate on a two-second, dollar-per-thousand-calls enrichment API is transformative. Instrument value directly: on every hit, record the cost and latency that would have been incurred, and sum it. That gives you a savings number to weigh against the complexity and the risk, and it usually reveals that one of your three caches is doing all the work and another should be deleted.
For the prefix cache the instrumentation already exists: the model response carries usage metadata separating cached from uncached input tokens, so log the cached fraction per request and alert when it falls. That alert catches the timestamp somebody added to the system instruction on the day it ships, rather than at the end of the billing month. Alongside it, track key cardinality (an explosion means a volatile dimension leaked into a key), hit-age distribution, and for the semantic cache a sampled false-hit rate. A cache with no staleness metric is a cache nobody is allowed to trust.