Why architecture matters here
The naive mental model — spot is on-demand with occasional random failures — underestimates the problem in two specific ways. First, interruptions are not independent events. A capacity pool is a market for one instance type in one availability zone; when the provider needs that pool back, it reclaims broadly. A fleet of 400 m5.2xlarge nodes in a single AZ is not 400 independent risks, it is one risk taken 400 times, and the day it materializes the entire fleet gets its two-minute notice inside the same five-minute window. Diversification across instance types, sizes, families, and zones is not an optimization; it is the core mechanism that converts a correlated shock into a background trickle.
Second, the two-minute window is a budget the whole termination path must fit inside: notice detection, load balancer deregistration, connection draining, pod eviction, checkpoint upload, and whatever the application does in its SIGTERM handler. A service whose graceful shutdown takes four minutes does not have a graceful shutdown on spot; it has a crash with extra steps. Architecting for spot means measuring and shrinking that path until it fits with margin.
The payoff justifies the effort arithmetic: at 70% savings, a platform spending $500k/month on compute recovers $350k — several engineers of budget — every month, forever. But the savings compound only if interruptions stay non-events. One data-loss incident traced to a preemption converts the organization back to on-demand faster than any spreadsheet converts it forward, which is why the architecture below spends most of its complexity on making reclaims boring.
Fit matters as much as machinery. Spot is a spectrum, not a binary: CI runners and render farms are nearly free wins; stateless serving tiers are excellent once drains are fast; stream processors and training jobs are good with checkpoint discipline; quorum systems and primary databases are poor fits regardless of engineering, because their failure cost dwarfs the discount. The honest first step of a spot program is classifying the fleet along that spectrum and taking the easy 60% of spend before litigating the hard 10% — the savings curve is steep at the start and flat at the end, and the incidents live at the flat end.
The architecture: every piece explained
Capacity pools and allocation strategy. The unit of spot supply is the pool: one instance type in one zone. A resilient request spans many pools — a dozen instance types across three zones gives thirty-plus pools — and lets an allocation strategy choose among them. Price-capacity-optimized (the sane default on AWS) weights toward pools with deep spare capacity and acceptable price, minimizing near-term interruption probability rather than chasing the absolute cheapest pool, whose price is often low precisely because it is about to be reclaimed.
The fleet orchestrator. Auto Scaling groups with mixed-instances policies, GCP MIGs, or Kubernetes-native provisioners like Karpenter own the launch decisions. They express workload needs as constraints — vCPU, memory, architecture — rather than instance types, so the scheduler can substitute freely across pools. A mixed policy also carries the on-demand base: a floor of guaranteed capacity sized to the minimum viable service, with spot layered above it as opportunistic headroom.
Interruption plumbing. Every node runs a listener — a daemonset like the node termination handler, or the orchestrator native integration — polling the instance metadata endpoint for spot interruption notices and rebalance recommendations. On notice, the drain controller cordons the node, deregisters it from load balancers, and evicts pods in order, respecting PodDisruptionBudgets but with a hard deadline that overrides them near the end of the window: a PDB is a preference, the reclaim is a fact.
The workload layer. Stateless services need only fast startup and connection draining. Everything else needs a checkpoint discipline: batch jobs persist progress markers to object storage at intervals sized so a reclaim wastes at most a few minutes of work; stream processors commit offsets and rebuild state from the log or a snapshot; training jobs checkpoint model state on a cadence balancing checkpoint cost against redo cost. The replacement instance — launched by the orchestrator, preferentially from a different pool — resumes from the checkpoint rather than from zero.
Scheduling tiers inside the cluster. Capacity type must be visible to the scheduler, not just to finance. Spot nodes carry a label and often a taint; workloads declare their tier by tolerating it. Critical singletons and quorum members schedule only to the on-demand base; stateless and checkpointed work prefers spot but can run anywhere; best-effort batch tolerates spot exclusively. Priority classes complete the picture: when a reclaim shrinks the cluster faster than replacements arrive, the scheduler evicts best-effort work first and protects the serving tier, turning a capacity dip into delayed batch jobs instead of dropped requests. The same tiering keeps a naive Helm chart from accidentally landing a database on a node that is contractually allowed to disappear at lunch.
End-to-end flow
A CI-and-batch platform runs a Kubernetes cluster with a Karpenter provisioner constrained to 4-16 vCPU nodes across twelve instance types in three zones, spot preferred, on-demand fallback enabled. Monday 14:02, the provider needs c6i.4xlarge capacity in zone A and issues interruption notices to nine of the platform nodes within ninety seconds of each other. Each node termination handler sees the notice within its two-second poll interval.
On every affected node the same choreography runs: cordon at t+2s so the scheduler stops placing work; deregister from the ingress load balancer at t+3s, with in-flight requests draining behind the deregistration delay; evict pods starting at t+5s. A batch pod running a 40-minute integration-test shard traps SIGTERM, writes its progress marker — shard 7, test 214 of 380 — to object storage in 800 ms, and exits cleanly at t+9s. Kafka consumer pods commit offsets and leave their groups, triggering rebalances that complete in seconds because cooperative rebalancing is enabled. By t+45s the nodes are empty shells; the reclaim at t+120s takes nothing of value.
Meanwhile Karpenter has been provisioning replacements since the first eviction made pods unschedulable. It skips c6i.4xlarge in zone A — the telemetry loop has already marked that pool hot — and lands equivalent capacity on m6a.4xlarge in zone B at a price 4% higher, still 68% below on-demand. The test shard restarts, reads its marker, and resumes from test 215. Total cost of the event: about three minutes of redone work and zero human attention. The interruption-rate dashboard ticks up for one pool; nobody is paged. Had no spot pool had capacity, the provisioner fallback would have launched on-demand and the cost-anomaly alert, not an outage, would have been the signal to investigate.
Two weeks later the harder scenario arrives: a regional capacity crunch dries up most spot pools at once, and replacement launches start failing with insufficient-capacity errors across nine of twelve pools. The provisioner walks its preference list, lands what it can on the three surviving pools, and fills the remainder with on-demand capacity under the fallback policy — the fleet stays whole, the savings rate for the week drops from 68% to 41%, and the finance dashboard, not the uptime dashboard, records the event. A reclaim-back loop keeps retrying spot in the background; three days later the pools recover and the on-demand nodes drain away one by one during low-traffic hours. The program's promise was never that spot is always available — it is that spot's unavailability costs money for a while instead of costing an outage.