BULKHEADS · RESERVED CAPACITY
Bulkheads: reserved capacity versus borrowing
Share ten workers between a search tool and a payments tool, slow the search backend down for thirty seconds, and see whether payments keep working.
Your experiment
Search and payments share one pool of ten workers. From 10 to 40 seconds search calls take 8 seconds instead of 1. Watch search occupy every worker and payments fail, then reserve workers per tool and allow bounded borrowing.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
One-second model: search arrives at 3/s, payments at 4/s, each call holds one worker for its duration, and a request that finds no worker waits in its tool’s queue or is rejected. Borrowing takes idle workers from the other tool’s reservation and returns them only when the borrowed call ends. No thread pool, circuit breaker or real dependency is used.
Two tools, ten workers, one slow backend
The agent calls a search tool (3 requests per second) and a payments tool (4 per second); each call holds a worker for 1 second, so normally 7 of the 10 workers are busy. From 10 to 40 seconds the search backend slows to Search latency during the incident, 8 seconds by default. Search alone would then need 3 × 8 = 24 workers. The question is whether the slow dependency takes payments down with it. Worker pools chooses one shared pool, fixed per-tool reservations, or reservations with bounded borrowing.
The opening run: payments starve
With one shared pool, search calls pile up and hold all 10 workers from early in the incident. Payment requests find no free worker, fill their 2-place queue and are rejected: 101 of 120 payments fail during the incident, so only 15.8% are served, although nothing is wrong with the payments backend. Search itself admits only 46.7% of its requests. A shared pool lets the slowest dependency decide everyone’s capacity.
Failure boundary: reservations and borrowing
Choose Fixed bulkheads with 4 workers reserved for search: search can hold only 4 workers, admits 20% during the incident, and payments keep their 6 workers and serve 100%. Reserve 7 for search and payments have only 3 workers for a load that needs 4, so only 75% are served: their own reservation is too small. 6 is the largest reservation that keeps payments whole. Choose Bulkheads with borrowing: with Most workers a tool may borrow at 2, search takes 2 of payments’ idle workers (admitting 28.9%) and payments still serve 100%; at 3, payments drop to 78.3%, and at 6 borrowing is as bad as sharing, because a borrowed worker stays held for 8 seconds.
Severity and queues
Search latency during the incident sets how many workers search tries to hold. At 2 seconds search needs 6 workers and even the shared pool copes; at 3 seconds it needs 9, and the shared pool already drops to 54.2% of payments. A longer Queue per tool does not create capacity: with 10 queue places the shared pool still serves only 16.7% of payments, and with none it serves 7.5%. Bulkheads trade some of the slow tool’s throughput for the healthy tool’s availability; they do not make search faster.
Reading the run and its limits
Play replays the incident, saturation, the first payment rejection, search admission, recovery and the outcome over sixty inspection seconds, while the chart draws the workers held by each tool. Next event, Restart, Speed and Inspection time control the replay, and any input change recomputes it. Arrivals and durations are fixed per second, there are no timeouts or circuit breakers, and borrowed workers cannot be preempted. Real bulkheads are usually combined with timeouts and breakers so slow calls release workers sooner.