A/B capacity: sample size changes operational exposure

What this lab demonstrates

A candidate rollout consumes real serving capacity even before an experiment has enough evidence for a product decision. This fixed cohort routes twelve requests through A or B and counts both execution demand and accepted outputs. Changing the candidate fraction from a quarter to three quarters reallocates load. The answer-regression switch makes candidate quality worse but does not magically reduce its assigned traffic. This separation is useful when planning quota, cache warmth, and rollback headroom. The request cards expose the particular bucket and result rather than hiding a synthetic average behind a bar.

How to use each control

Policy selects the traffic split or decision rule shown beside the simulation. Fault fixture chooses a normal cohort or a scripted failure case. Play requests advances one request every five seconds across a sixty-second sequence; Pause requests freezes the current state. Restart returns to the start, Timeline seeks directly to a second, and Speed changes playback to half, normal, or double rate without changing the result. Each numbered request card gives its route, accepted status, latency, and execution count. The summary counts only requests already completed at the selected time; full-cohort totals appear at sixty seconds.

Try a controlled comparison

Run 25% candidate at normal quality and note the B count. Increase to 50% and then 75% without changing the fixture. The B share grows while A shrinks. Inject the candidate regression and ask how much traffic experiences the weaker answer fixture under each split. Seek to 30 seconds to compare only the first six requests with the full cohort at 60 seconds. Avoid treating an early partial count as a final capacity estimate.

What this model simplifies

One request equals one unit of assignment here; real agents have variable token usage, queue time, concurrency, cost, and retry amplification. Twelve predetermined requests cannot forecast peak traffic or statistically establish quality. Capacity planning needs real arrival distributions and service-time measurements.

Separate assignment from capacity claims

This local experiment assigns each of twelve fixed requests to exactly one arm. Candidate percentages control a threshold over twelve deterministic buckets; they do not generate a new random sample. Request index i starts at zero and receives bucket (7 times i) modulo 12. A request goes to B when its bucket is below three, six or nine for the 25, 50 or 75 percent setting. The threshold is strictly less than, so the boundary bucket belongs to A. The same buckets are retained across settings, making candidate exposure monotone as the threshold rises.

Verify an assignment by hand

Choose 25% candidate and Normal candidate. Finish the timeline and inspect B requests R01, R03 and R08, with buckets zero, two and one respectively. The remaining nine requests go to A. There are twelve executions, with no retries or duplicate calls. B accepts two of three and A accepts seven of nine. R01 is a candidate failure. These counts illustrate exposure and conservation of assignments, not a production capacity forecast. Real resource demand also depends on tokens, concurrency, service times, retries and shared quotas, which this model deliberately excludes.

Increase exposure without inventing a quality gain

Change Policy to 50% candidate: A and B each receive six requests. With the normal fixture, A accepts four and B accepts five. At 75%, B receives nine and accepts seven; A receives three and accepts one. The total accepted count falls from nine at 50% to eight at 75%, even though B has the better observed fraction in each split. Different requests entered each arm. Do not infer a causal treatment effect from these unrandomized, tiny cohorts or pool them as independent experiments. No confidence interval or significance claim is produced.

Find a fault that a small cohort misses

At 25%, switch Fault fixture to Candidate answer regression. The totals remain unchanged because the extra scripted B failures occur outside those three assigned B requests. At 50%, the same fault lowers candidate acceptance from five of six to four of six, while A stays four of six. At 75%, B falls from seven of nine to six of nine. The fault does not change assignments or execution demand. This is an exact example of why an unchanged aggregate in a small cohort cannot prove a candidate safe. Online experiments also require stable user or session assignment, exposure logging, guardrails and protection against interference through shared state; this request-only fixture does not implement those production systems.

Inspect the same cohort at your own pace

Policy and Fault fixture restart the twelve fixed requests at zero. Play requests advances the sixty-second inspection; Pause requests freezes it, and Resume requests continues. Restart retains the chosen policy and fault but rewinds the observations. Timeline pauses and seeks directly; a request completes at each five-second boundary. Speed selects half, normal or double inspection pace and does not change assignments, answer labels or modeled milliseconds. At sixty seconds Replay requests starts again. Keyboard users can activate buttons with Space and move the timeline with arrow keys. The observed-record table shows completed requests only; the current request text previews the active request before its completion. Export fixture JSON deliberately includes the full computed fixture and separately labels the current inspection position, so it must not be mistaken for a partial observation export.

Observed request records

The export includes the full computed fixture and separately identifies the inspection position.