A cache is a bet that the same bytes will be asked for again before they change. When the bet pays, a request that would have cost a database seek, a template render and a transcontinental round trip costs a hash lookup instead. When it loses, you are serving something that is no longer true. Everything hard about caching lives in that second sentence: the mechanics of storing a copy are trivial, and the mechanics of knowing when the copy stopped being valid are a distributed-systems problem you have now taken on once per tier. This article is the map — what each tier can and cannot do, which read and write patterns exist, how invalidation actually goes wrong, and what to measure — with links to the deep dives on the individual mechanisms.
What a cache actually buys, in arithmetic
Mean latency through a cache is h·t_hit + (1−h)·t_miss, and the term that matters is the one with the small coefficient. With a 0.5 ms hit and a 20 ms miss, a 90% hit ratio yields about 2.5 ms average; pushing to 99% yields about 0.7 ms. The user-visible gain is real but bounded, because t_miss still sets the tail — the p99 of a cached endpoint is usually just the uncached path.
The larger effect is on the thing behind the cache. Going from 90% to 99% does not improve the hit ratio by 10%; it divides origin traffic by ten. That asymmetry is why the number worth tracking is the miss rate, not the hit rate: 99% and 99.9% look identical on a dashboard and differ by a factor of ten in load on your database. It also runs backwards, which is the part teams discover during an incident. If a tier at 99% hit ratio empties, the layer behind it receives one hundred times its normal read volume, instantly and with no ramp. A cache does not merely accelerate the origin; it silently re-sizes it, and the origin you provisioned is the one that only ever saw 1% of reads.
The cost is uniform across tiers: each cached value is a second copy of a fact that lives elsewhere, so each tier adds a staleness window you are responsible for naming.
The tiers: where a byte can live
Roughly five places, ordered from the user inward. The client — browser HTTP cache, mobile app store, service worker — is the cheapest hit and the only tier you cannot invalidate. You can only wait out the freshness lifetime you already granted, which is why long-lived assets are addressed by content hash in the filename rather than purged.
The CDN edge is a shared HTTP cache keyed on URL plus whatever the response declares in Vary. It has a purge API, so it is invalidatable, but not atomically. Its internals — anycast catchment, cache-key normalization, TTL layering with s-maxage and stale-while-revalidate, origin shielding — are a subject of their own; see Content Delivery Network Architecture in Depth and the provider treatments of CloudFront and Cloud CDN.
The reverse proxy (NGINX, Varnish, an API gateway) applies the same HTTP semantics inside your perimeter, where it can see authenticated context the edge cannot. The application cache — Redis, Memcached, or an in-process map — is keyed on anything you can compute: entity ids, query fingerprints, rendered fragments, permission decisions. Maximum freedom, therefore maximum invalidation risk. The database caches whether you ask it to or not: the buffer pool holds hot pages (buffer pool architecture), plan caches hold compiled statements, and materialized views are a cache you declare in SQL and refresh on a schedule.
In-process versus remote: opposite failure modes
An in-process cache — a Caffeine or Guava map on the heap — turns a hit into a lookup on the order of a hundred nanoseconds with no serialization and no network. It also gives you one independent copy per instance. A hundred application instances means a hundred caches, each stale in its own way, with no protocol between them; an invalidation now has to be broadcast rather than issued. It costs heap, so a large local cache buys hit ratio with GC pause time, which is why serious implementations move the bytes off-heap (HBase BucketCache is the worked example). And it is cold on every deploy: a rolling restart of a hundred pods is a hundred cold caches sliding across the fleet.
A remote cache inverts every one of those. One logical copy, so invalidation is a single DEL. Capacity independent of application memory, survives deploys, shared by every instance. The price is a network round trip on the hit path — sub-millisecond within a datacenter, plus serialization on both ends — and a new hard dependency. Client timeouts must be aggressive, because a cache slower than the origin it fronts is worse than no cache at all, and the fall-through path must actually work under full miss traffic. Sharding, replication and hash-slot routing for that tier are their own subject — see designing a distributed cache.
The common compromise is two tiers: a small near-cache with a few seconds of TTL in front of Redis. It absorbs hot keys and removes the round trip, at the cost of a bounded incoherence window you should be able to state out loud in seconds.
Read patterns: cache-aside and read-through
Cache-aside (lazy loading) is what most systems run. The application reads the cache; on a miss it reads the store, writes the value back with a TTL, and returns it. Nothing is cached until someone asks, so the cache holds exactly the working set, and a cache outage degrades to slow rather than broken. Its real cost is that correctness is spread across every call site: any code path that writes the store without invalidating — a backfill job, a migration, an operator running UPDATE by hand — produces stale reads that no code review will catch.
Read-through moves the loader inside the cache. You configure a loading function once (Caffeine.newBuilder().build(loader), or a transparent proxy like DAX in front of DynamoDB) and the cache calls it on a miss. One place to change, and request coalescing comes for free — concurrent loads of the same key collapse into a single call. The trade is that store failures now surface as cache failures and the abstraction hides which requests paid for a load.
Both patterns share a blind spot: the negative result. If "not found" is not cached, a single hot missing key — a deleted product still linked from an email blast, a scanner probing ids — sends every request straight to the store. Cache the absence under a distinguishable sentinel with a short TTL, short enough that a real creation becomes visible quickly.
Write patterns: through, behind, and around
Write-through writes the cache and the store synchronously in one logical operation. The cache is never stale relative to the store, which is the strongest guarantee on offer without coordination, and the price is that every write pays both latencies and every write populates the cache — including keys nobody will read, which burns memory on write-heavy, read-cold workloads.
Write-behind (write-back) acknowledges after the cache write and flushes to the store asynchronously. Writes are fast and repeated updates to the same key coalesce into one store write, which is a genuine throughput win for counters and last-seen timestamps. The loss window is equally genuine: if the cache node dies before the flush, acknowledged writes are gone. This is only honest when the data is reconstructible, or when the cache itself is durable — which is exactly the deal a database buffer pool makes, and the write-ahead log is what makes it safe.
Write-around writes the store and leaves the cache alone; the next read fills it. It keeps bulk imports and one-shot writes from evicting the working set, and it accepts that a read immediately following a write will miss. Most real systems mix all three by key family rather than picking one. The dedicated comparison lives in cache-aside, write-through and write-behind.
Invalidation: delete, do not update
Four mechanisms exist, in increasing precision and cost. TTL is the only one that bounds staleness without anyone cooperating, and it is the reason every entry should carry an expiry even when you also invalidate explicitly: the TTL is your recovery from the invalidation bugs you have not found yet. Staleness is then bounded by the TTL, which is a number you can put in a design document.
Explicit invalidation on write should delete the key, not overwrite it. Two writers that each update the cache can commit to the database in one order and reach the cache in the other, leaving the loser's value cached indefinitely with no error anywhere. Deletion is idempotent and order-insensitive: whoever reads next refills from the source of truth. The rule is worth enforcing in review — invalidate, never update.
Versioned keys avoid invalidation entirely by making the key change when the content changes: user:812:v37, or a content hash in an asset filename. Nothing is ever purged; obsolete entries age out under eviction. The cost is distributing the version bump. Event-driven invalidation derives deletes from the database change log via CDC or a transactional outbox. It is the only approach that covers writes bypassing the application, which is the largest source of impossible-looking stale data. More strategies in cache invalidation strategies.
The fill race that leaves a value stale forever
This is the bug that survives correct-looking code. Reader R misses key k and reads v1 from the database. Writer W then commits v2 and deletes k — correctly, in that order. R, still holding its in-flight result, now completes its fill and writes v1 into the cache. The delete happened before the write it was meant to cancel. The cache holds v1, the database holds v2, no event is pending, and without a TTL that state is permanent.
Four mitigations, in ascending cost. A TTL on every entry converts permanent to bounded — necessary, not sufficient. A delayed second delete (delete, write, then delete again after longer than a typical fill) closes most of the window and needs a durable retry queue to survive a process dying mid-sequence. A conditional fill stores the source version alongside the value and refuses to overwrite a newer one — a compare-and-set, in Redis usually a small Lua script so the check and the write are atomic. Leases are the complete answer: on a miss the cache issues a token, only a fill carrying the current token is accepted, and any invalidation voids outstanding tokens. That is the mechanism described in Facebook's Scaling Memcache at Facebook, and it solves stampede in the same move because only the token holder goes to the database.
If the fill reads a replica rather than the primary, replication lag widens the window from milliseconds to seconds — see read-replica routing.
Stampede: three herd shapes, three different fixes
Hot-key expiry. One popular entry expires and every concurrent request misses in the same instant, so a thousand identical database queries launch together. The fix is to make one loader win: request coalescing per key inside the process, plus a distributed lock or a lease across processes so only one instance recomputes while the others wait or serve the previous value. The lock-free alternative is probabilistic early expiration — refresh with a probability that rises as the entry nears expiry, the rule from Optimal Probabilistic Cache Stampede Prevention — which staggers refreshes without coordination.
Synchronized expiry. Entries created together expire together: a warm-up script, a bulk import, or a deploy that populates a namespace all share one TTL and therefore one expiry instant. The whole cohort falls off a cliff at once, on a period equal to the TTL, which makes it look like a mysterious recurring spike. The fix is jitter — a random fraction, commonly ±10%, added to every TTL at write time.
Cold cache. A restart, a failover, or an over-broad purge empties the tier and 100% of traffic reaches the origin simultaneously. No per-key technique helps; you need load shedding at the origin, a warm-up before the instance takes traffic, and capacity planned for the miss rate rather than the hit rate. Serving stale while refreshing in the background converts all three shapes from an outage into a latency footnote. Details in cache stampede prevention.
Eviction: LRU is a default, not an answer
Entries leave for two unrelated reasons, and conflating them hides a real problem. Expiry means the TTL elapsed — intended. Eviction means capacity forced the entry out before its TTL — the cache is undersized, and the hit ratio you are measuring is capacity-bound rather than workload-bound. Track the two counters separately.
LRU is the default because it is cheap and recency is a decent predictor, and it fails predictably on scans: one analytics sweep or one crawler walking cold pages touches every key once and evicts the entire working set on the way through. The repairs are segmentation and admission. Segmented designs split the cache into a probation queue and a protected queue so a single-touch entry can never displace a repeatedly-used one. Admission policies go further: W-TinyLFU, the policy in Caffeine, keeps a compact frequency sketch and admits a candidate only when it is estimated to be more popular than the victim it would evict, which makes a one-shot scan structurally incapable of polluting the cache.
In Redis, this is maxmemory plus maxmemory-policy, and the default matters: noeviction turns a full instance into write errors, which is right for a datastore and wrong for a cache. Choose among allkeys-lru, allkeys-lfu, volatile-ttl and friends deliberately. On sizing, popularity is usually Zipf-like, so hit ratio rises roughly logarithmically with capacity: past the knee, the next doubling of memory buys almost nothing.
Hot keys, layered staleness, and what caching cannot fix
A single key is served by exactly one shard, so once one key's traffic exceeds one node's capacity, sharding has nothing left to offer. Detection is per-key sampling; mitigation is a short-TTL near-cache in front of the shared tier, or splitting the key into k replicas that readers choose among at random. The full treatment is hot-key mitigation.
Staleness composes down the chain rather than being capped by the largest tier. A browser granted ten minutes of freshness in front of an edge granted ten minutes can show content roughly twenty minutes old, and an application cache underneath adds its own window on top. Budget staleness end to end and derive per-tier TTLs from it, not the reverse.
Two rules with no exceptions. Per-user data must never enter a shared tier unless the identity is part of the key — a personalized response cached at a proxy under a bare URL is how one user's account page gets served to another. And a cache is not a fix for an unbounded write rate, for a query whose result is unique per request and therefore has nothing to reuse, or for a read that genuinely requires the latest committed value. What caching does reliably is hide the cost of the uncached path until the day you need it, which is why that path deserves a load test of its own. Where exact keys are too strict, semantic caching widens the match.
Operating a cache: the numbers worth alerting on
Measure hit ratio per key family, never as one global number. A global 95% comfortably hides a family sitting at 20% that is generating most of the origin load, and the aggregate moves too slowly to alert on. Report hit latency and miss latency separately for the same reason: hits dominate the mean and bury the tail that users actually feel.
On a Redis tier the standing four are keyspace_hits versus keyspace_misses, evicted_keys versus expired_keys, used_memory against maxmemory, and mem_fragmentation_ratio. An eviction rate rising from zero is the earliest honest signal of undersizing. Watch key cardinality too — unbounded key growth from an unnormalized parameter is the usual cause of an eviction storm that appears overnight with no traffic change.
Alert on origin QPS rather than on cache health, because origin QPS is the number that hurts. Also alert on a hit-ratio cliff after deploys: changing a key format is a full flush, and it ships silently in a refactor. Treat the cache as a dependency — short timeouts, a circuit breaker, and a fall-through path exercised deliberately. The only honest test of that path is to fail the cache during a load test and confirm the origin sheds load instead of collapsing. General metric hygiene is covered in metrics, counters and histograms.