RELIABILITY · REQUEST WORKSHOP
Rate limiting: burst budget versus sustained rate
Fill the waiting room, watch workers process real model requests, and inspect every admission decision.
Your experiment
Compare a bucket of 4 and 20 tokens with the same refill rate. Why does a bigger burst budget not increase long-run worker capacity?
- Play the default run and inspect the queue.
- Pause and capture a baseline.
- Change one parameter, then compare the same model time.
- Account for every offered unit: completed + pending + rejected + held back.
Admission trace
Teaching model, not an ADK/A2A runtime or measured infrastructure benchmark. Fixed service time, one FIFO, no retries, deadlines or network overhead; 50 ms ticks. Backpressure pauses a cooperative producer at 75% queue occupancy and resumes at 25%. Held-back demand is not transmitted or retained for later delivery. Early shedding uses half of queue capacity. Token bucket admission consumes one token per request, including requests subsequently rejected by the queue. The local Retry-After illustration is not a protocol-wide guarantee. Throttling reference · Load shedding reference.
Separate a burst allowance from capacity
Choose token-bucket limiting, twelve desired requests per second and refill six. Compare burst capacities four and twenty while retaining three workers and one-second service. The larger initial bucket admits more early traffic but cannot turn three workers into six. Reduce refill to two and observe more limiter refusals; increase workers without changing refill and notice that admission still has its own ceiling. Inspect waiting work as well as 429 events, because consuming a token is not a reservation of a processing slot.
Set offered work and service capacity
Desired requests per second sets deterministic arrivals, not measured customer demand. Workers sets the number of concurrent service slots. Service seconds is the fixed processing time for one request. Queue capacity bounds waiting requests; active work occupies separate slots. Compare desired rate with workers divided by service time before interpreting a growing queue. Completions are processed before admission in each fifty-millisecond model tick, so equal-time arrivals can reuse a newly freed slot.
Choose an admission policy
Policy selects bounded FIFO, producer backpressure, token-bucket rate limiting or early load shedding. FIFO rejects arrivals after the waiting queue fills. Shedding uses half the configured queue capacity, with a minimum of one waiting slot. Backpressure pauses the modeled producer at the high watermark and resumes below the low watermark: deferred requests are counted but are not retained for later replay. Token refill per second and Burst tokens apply only to the limiter. An initially full bucket permits a burst, but tokens do not add workers or increase service capacity.
Read outcomes and replay
Completed work, active requests, waiting requests, rejected work and deferred demand describe different outcomes. The limiter can spend a token before a later queue-capacity rejection. Inspect the event log for 429 versus overload responses instead of treating every rejection as the same fault. Play runs sixty modeled seconds; Pause holds the board, Step advances one second, Restart rewinds, Speed changes the inspection pace and the timeline reproduces the same state when scrubbed backward. Changes rebuild the scenario. Latency statistics concern this fixed-duration synthetic workload; there are no retries, variable service distributions, multiple tenants or production HTTP calls.