Agent batching: throughput grouping versus request wait

What this lab demonstrates

Batching can improve device utilization while adding time in a queue for the earliest request. This lab gives each of twelve numbered requests an arrival tick. In the steady fixture, one request arrives per tick; in the burst fixture, three arrive together at ticks zero, three, six, and nine. A batch-size rule groups consecutive requests in twos or fours and flushes when the group is complete. Each card reports its batch, arrival tick, flush tick, and computed wait. The summary counts distinct batches and maximum observed wait. A larger batch reduces the number of launches but can hold an early request longer.

How to use each control

Policy selects the validation, compatibility, vote, or batch-size rule shown in the settings. Fixture selects complete or failure-injected traffic. Play steps reveals one numbered step every five seconds over a sixty-second inspection. Pause steps freezes the current state, Restart returns to the beginning, and Timeline jumps to an exact point. Speed changes viewing pace to 0.5, 1, or 2 times without changing the deterministic result. Read the card state, the highlighted step detail, and the running count together: waiting steps do not enter the count. Changing either input restarts the same twelve-step sequence so you can compare policies on identical examples.

Try a controlled comparison

Use One request each tick and Batch up to two; inspect the first pair to see which request waits. Increase the limit to four and compare the maximum wait. Inject Four bursts of three requests and inspect requests S01 through S04: a burst can fill part of a batch immediately, but the fourth request arrives at a later tick. Seek to sixty seconds to compare total batch count. Pause at twenty seconds and note that the visible maximum wait can grow as a later batch completes. Record both throughput grouping and latency, not only one metric.

What this model simplifies

A tick is a teaching unit, and batch execution itself has zero modeled duration. This is a simple fixed-size grouping, not continuous batching, an adaptive timeout, GPU scheduling, or a real ADK Java service. Production batching must account for token length, memory pressure, queue deadlines, fairness across tenants, and model-specific execution cost.

Trade fewer dispatches for waiting

Twelve requests are grouped in arrival order into fixed batches of two or four. Each group flushes when its last member arrives, and each request’s wait is that flush tick minus its own arrival tick. The output records a batch identifier and wait for every request. This is fixed-size request batching, not continuous token batching, GPU scheduling or a measured throughput benchmark. No service time or capacity limit is included, so fewer batches cannot be translated into a claimed speedup.

Set group size and arrival shape

Policy chooses Batch up to two or Batch up to four. Fixture chooses one request per tick or four bursts of three requests. The bursts arrive at ticks zero, three, six and nine. Both controls recompute the complete row list and reset inspection. A tick is an abstract arrival interval, not a second of processing time. The timeline shows one request every five viewing seconds regardless of its modeled arrival tick; use the row details to inspect waiting.

Calculate the steady-arrival result

With one request per tick, batches of two produce six dispatches and maximum wait one tick. Each first request waits for the next request and each second request waits zero. Batches of four produce three dispatches and maximum wait three ticks; waits repeat three, two, one, zero. All twelve requests are accepted in both configurations. Compare dispatch count and waiting together instead of treating a smaller dispatch count as an unconditional improvement.

Expose a burst boundary

With four three-request bursts and batches of two, request three waits from tick zero to tick three for request four. Maximum wait is therefore three ticks despite the smaller batch size. Batches of four also have maximum wait three ticks in this fixture. Group boundaries and arrival boundaries do not align, which explains why the steady-arrival intuition does not transfer directly. The input size is divisible by both batch sizes; there is no partial final batch, timeout flush, cancellation or overload case. Without a timeout, a real partial batch could wait indefinitely, a failure that this complete twelve-request fixture cannot simulate.

Inspect the replay

Play starts the sixty-second inspection and becomes Pause while running. Pause holds the current stage; Restart returns to the beginning without changing the experiment settings. Speed changes viewing pace rather than model outcomes. Timeline seeks directly to a stage and pauses playback. Changing a scenario setting recomputes the deterministic fixture and rewinds it. The final decision appears at the end, so move Timeline to sixty to compare completed runs. Stage cards and the text trace describe the same selected fixture; their elapsed viewing time is not measured agent execution latency.