Agent A/B assignment: expose the candidate deliberately

What this lab demonstrates

An A/B rollout is an assignment decision before it is a graph of quality scores. This experiment routes twelve numbered requests with a stable bucket function. The selected candidate share controls exposure, while the answer-regression switch changes the candidate fixture without changing the assignment. Watch each request enter A or B and compare accepted answers with the number of requests each arm actually received. A raw count of accepted answers can mislead when the arms have different sample sizes. The browser does not call an agent or judge live model output.

How to use each control

Policy selects the traffic split or decision rule shown beside the simulation. Fault fixture chooses a normal cohort or a scripted failure case. Play requests advances one request every five seconds across a sixty-second sequence; Pause requests freezes the current state. Restart returns to the start, Timeline seeks directly to a second, and Speed changes playback to half, normal, or double rate without changing the result. Each numbered request card gives its route, accepted status, latency, and execution count. The summary counts only requests already completed at the selected time; full-cohort totals appear at sixty seconds.

Try a controlled comparison

Start with 25% candidate and the normal fixture. Seek to the end and record the candidate request count and accepted count. Change to 75%; more requests now see B, and the observed counts change with exposure. Turn on the candidate answer regression and compare accepted fractions within each arm, not only total acceptances. Replay at 0.5 speed to see the exact bucket assigned to each request. A tiny twelve-request sample is intentionally too small for a significance claim.

What this model simplifies

The assignment sequence and answer outcomes are deterministic teaching fixtures. There is no randomization service, stratification, user stickiness, statistical confidence interval, or real ADK Java agent. The sample can illustrate exposure and arithmetic, not prove that one agent is better.

Keep assignment separate from answer quality

Twelve fictional requests are assigned by a fixed bucket rule. Request index i uses bucket (i times seven) modulo twelve, and a bucket below the selected candidate fraction goes to B. The remaining requests go to A. Policy selects twenty-five, fifty or seventy-five percent, yielding exactly three, six or nine candidate requests in this tiny fixture. This is request-level allocation, not sticky user or session assignment. Replaying identical inputs reuses identical assignments and outcomes.

Change one source of evidence

Fault fixture switches between a normal candidate and a candidate answer regression. It changes the predefined accepted-answer pattern for B without changing assignment buckets. Control acceptance is false at indices divisible by five. Normal candidate acceptance is false at indices divisible by six; the weak candidate fails at indices divisible by three. These authored patterns make cohort counts reproducible, but they are not measured responses from any model. Latencies are also deterministic fixture values, not timing measurements.

Observe the sample boundary

Choose fifty percent and Normal candidate, then inspect the completed trace. Six requests reach each variant; five candidate answers and four control answers are accepted. Switch only to Candidate answer regression: candidate acceptances fall to four while control acceptances stay at four. At twenty-five percent the candidate has only three observations and its accepted count does not change between these two patterns. That boundary demonstrates why a fault knob need not change a result when the exposed subset misses the affected cases.

Compare like denominators

Increasing exposure changes which requests are observed in each cohort and changes the denominator of its acceptance rate. Raw accepted counts alone do not rank the agents. This fixture has no confidence intervals, random sampling across users, multiple comparisons, early-stopping correction or causal attribution. A favorable outcome on twelve authored requests cannot justify production promotion. Examine the request identifiers and assignment bucket before making a comparison, and retain the distinction between known fixture truth and evidence an actual experiment would collect.

Inspect the replay

Play starts the sixty-second inspection and becomes Pause while running. Pause holds the current stage; Restart returns to the beginning without changing the experiment settings. Speed changes viewing pace rather than model outcomes. Timeline seeks directly to a stage and pauses playback. Changing a scenario setting recomputes the deterministic fixture and rewinds it. The final decision appears at the end, so move Timeline to sixty to compare completed runs. Stage cards and the text trace describe the same selected fixture; their elapsed viewing time is not measured agent execution latency.