Why architecture matters here
Shutdown architecture matters because it is the difference between a deployment being a routine event and a deployment being an incident. In a modern environment your agent runtime is terminated constantly and for entirely mundane reasons: every deploy, every autoscale-down, every node drain, every spot reclamation, every OOM-adjacent eviction. If each of those loses work, then your reliability is bounded not by your code's correctness but by your deployment frequency — and deployment frequency is something you presumably want to increase.
The reason it is harder for agents than for web services comes down to task duration and cost. A dropped HTTP request is retried by the client and costs milliseconds. A dropped agent task has already burned several model calls' worth of tokens, possibly executed tool calls with real side effects, and left a user staring at a spinner that will now spin forever. Retrying is not free and it is not always safe — if the task already sent an email before it was killed, retrying sends a second one. The economics and the semantics both push toward finishing or checkpointing rather than dropping.
This creates a real design fork based on task duration. Short tasks — a few seconds — are best simply finished; drain until they complete, which fits comfortably inside any reasonable grace period. Long tasks cannot be finished within the window, and no amount of asking nicely changes that. For those, the only viable strategy is to checkpoint: persist enough state that another instance can resume, hand the task back to the queue, and let it be picked up elsewhere. This means long-running agent tasks need to be checkpointable by design, which is an architectural requirement that reaches all the way back into how you structure agent execution — you cannot bolt it on during shutdown.
There is also an ordering problem that is easy to get backwards, and getting it backwards is worse than doing nothing. Between SIGTERM arriving and the load balancer noticing you are gone there is a propagation delay — readiness probes have intervals, endpoint updates take time to propagate through the control plane. If you stop accepting requests the instant SIGTERM arrives, the load balancer keeps sending traffic to a socket that now refuses it, and users get errors during every single deploy. The correct sequence flips readiness to false first, then waits out the propagation delay while still serving, and only then stops accepting. Shutdown starts with a deliberate pause during which you continue doing exactly what you were doing.
Finally, shutdown is where resource ordering bugs surface, because it is the only time the dependency graph runs in reverse. Dependencies are acquired in a natural order during operation — the connection pool exists before the agent using it — and during shutdown that order must be inverted, with nothing enforcing it. Close the pool while a draining task still needs a connection and the task fails having nearly succeeded. Flush telemetry before the last spans are written and you lose exactly the data needed to debug this. The bugs only manifest under termination, and your integration tests almost certainly never terminate anything.
The architecture: every piece explained
The signal handler is the entry point, and its job is to be fast and to delegate. A JVM shutdown hook registered via Runtime.getRuntime().addShutdownHook() catches SIGTERM, but hooks are unordered with respect to each other, which means a design with several independent hooks has a race with itself. The robust pattern is exactly one hook that invokes a single shutdown coordinator, which then sequences everything explicitly. One hook, one sequence, deterministic order. Registering three hooks and hoping is how you get a connection pool closed underneath a draining task.
The readiness gate is the first step in that sequence and the one most often skipped. Flipping the readiness probe to false signals the orchestrator to remove this instance from the service endpoints, but propagation is not instant — the probe has a period, the endpoint controller has to observe it, and the change has to reach every proxy. Until that completes, traffic keeps arriving and must keep being served. The coordinator therefore flips readiness and then sleeps, serving normally, for long enough to cover propagation. Kubernetes offers a preStop hook for exactly this, which fires before SIGTERM and gives you a clean place to put the sleep. This deliberate delay feels wrong the first time you write it — the process was asked to stop and it is doing nothing — and it is what eliminates deploy-time errors.
The admission gate stops new work once propagation has completed. This is a flag the task-accepting path checks: new task submissions are rejected with a retryable status and a Retry-After, so callers route elsewhere rather than failing permanently. Critically, this gate is separate from the readiness flip, because the two happen at different times and mean different things — readiness is advisory to the infrastructure, admission is enforcement in the application. Conflating them is what produces the errors-during-deploy problem.
The drain coordinator is the core. It tracks in-flight tasks — which requires a registry that tasks enter on start and leave on completion, maintained accurately, because a leaked registry entry means shutdown waits forever for a task that finished an hour ago. It waits for the count to reach zero, bounded by a deadline. On expiry, it moves to checkpointing: each remaining task is asked to persist its state and yield. Checkpoint state goes to the same durable store the task's normal state lives in, and the task is requeued so another instance resumes it. Only after the checkpoint write is durably acknowledged is the task considered handled — a checkpoint that was written to a buffer and lost at exit is worse than no checkpoint, because the requeue already happened and now two instances disagree about the task's state.
Resource teardown runs last, in reverse dependency order, and this is where Java's specifics bite. Executors need shutdown() followed by awaitTermination() with a bounded timeout, then shutdownNow() if the timeout expires — the pattern exists precisely because neither call alone is correct. Virtual threads change the picture: they are always daemon threads, so they do not keep the JVM alive, which means a virtual thread doing important work will simply evaporate at exit with no warning. Structured concurrency helps by making the lifetime explicit and joinable, and it is the right tool here specifically because it turns an invisible lifetime into one the coordinator can wait on. Telemetry flushes absolutely last, after everything else, because everything else generates the spans and metrics you want to see.
End-to-end flow
Walk a real termination. The platform decides to roll a deployment. Kubernetes marks the pod for termination and fires the preStop hook, then sends SIGTERM. The grace period is 60 seconds; SIGKILL follows at 60 regardless of what is happening.
The preStop hook flips the readiness endpoint to return failure and sleeps for 5 seconds. During those 5 seconds the process is fully operational — it accepts new tasks, serves existing ones, behaves exactly as it did a moment ago. This is deliberate. The kubelet observes the failing readiness probe, the endpoint controller removes the pod from the service, and that removal propagates to every proxy in the mesh. By second 5, no new traffic is being routed here, and no client ever saw an error.
SIGTERM arrives and the single shutdown hook invokes the coordinator. The admission gate flips: any task submission that somehow still arrives gets a 503 with Retry-After, which is a clean, retryable answer rather than a connection reset. The coordinator reads the in-flight registry: eight tasks are running. It computes its drain deadline as 45 seconds — deliberately 15 short of the 60-second grace period, leaving margin for checkpointing and teardown.
Over the next 30 seconds, six tasks complete naturally. They were short: a couple of model calls each, and they finish and deregister from the registry. Two remain — long research tasks, each several minutes into a multi-step plan, with no chance of completing inside the window. At the 45-second deadline the coordinator stops waiting and switches to checkpointing. It signals each remaining task to yield at its next safe point — not immediately, because interrupting mid-tool-call could leave a side effect half-applied, but at the next boundary between steps where the state is coherent. Each task serializes its state: which plan steps completed, what results they produced, what the next step is. That state is written to the session store and fsync-acknowledged before the task is requeued, because a requeue whose checkpoint is still in a buffer is a task that will resume from a state that never existed.
With the registry empty at second 52, teardown runs in reverse dependency order. The task executor is shut down and awaited. The agent runtime closes. The HTTP client pool drains its keep-alive connections. The database pool closes, which is safe now because nothing needs a connection any more — had this run before the checkpoint writes, those writes would have failed at the worst possible moment. Finally the telemetry exporter flushes, pushing out every span and metric including the ones describing this shutdown, which is how you will know it worked. At second 56 the JVM exits 0, four seconds ahead of SIGKILL. No task was lost: six completed, two resumed elsewhere within seconds. From the users' perspective, nothing happened at all.