Gemini variant comparison: quality versus latency

What this lab demonstrates

A paired comparison holds the request fixture constant while recording an outcome from both variants. Here A and B each execute once for every request, so twelve prompts require twenty-four model executions. The browser shows each pair of accepted-or-failed answers and two synthetic latencies. The selection rule can prioritize answer quality or response latency; when quality ties, the shorter latency wins. This is an offline comparison rule that uses both observed outcomes, not a production router that can know a future answer before selecting a live model.

How to use each control

Policy selects the traffic split or decision rule shown beside the simulation. Fault fixture chooses a normal cohort or a scripted failure case. Play requests advances one request every five seconds across a sixty-second sequence; Pause requests freezes the current state. Restart returns to the start, Timeline seeks directly to a second, and Speed changes playback to half, normal, or double rate without changing the result. Each numbered request card gives its route, accepted status, latency, and execution count. The summary counts only requests already completed at the selected time; full-cohort totals appear at sixty seconds.

Try a controlled comparison

Use Inspect answer quality with a normal B fixture and seek to the final totals. Switch to Inspect response latency: the selected winner can differ on a request where the faster response failed. Inject Variant B answer regression and inspect the affected request cards one by one. Compare the A and B accepted fractions with the selected output; they answer different questions. Pause halfway to see why a partial cohort is not the whole result.

What this model simplifies

Gemini is only a topic label in this synthetic lab; no Google model, SDK, token meter, or external judge is called. The predetermined answer labels are not claims about any real variant. Real paired evaluation needs representative prompts, blinded judging, cost and latency distributions, safety review, and uncertainty intervals.

Compare paired observations

This experiment evaluates the same twelve numbered prompts with fictional variants A and B. Every prompt costs two model executions regardless of which response the selection rule prefers. The latency numbers and acceptance labels are deterministic teaching data, not measurements of Gemini or ADK Java. The observed-record table retains both outcomes in its evidence column. Selecting a response does not erase the other execution or its failed answer. This separation prevents a common reporting mistake: interpreting the quality of the chosen response as the standalone quality of the candidate model.

Reproduce the exact quality experiment

Choose Inspect answer quality and Normal variant B, then seek to sixty seconds. A accepts nine of twelve prompts and B accepts ten of twelve. The selected outputs accept eleven of twelve because the rule can choose an accepted answer when the other variant fails. R01 fails for both variants and cannot be repaired by selection. This offline rule uses already known answer labels. A live router cannot know those labels before execution, and a real evaluation judge can be wrong. A deployment claim would require a separate routing policy and representative evaluation evidence.

Expose the latency tradeoff

Switch Policy to Inspect response latency and finish again. There are still twenty-four executions and the same standalone A and B acceptance counts, but only ten selected answers pass. Seek to thirty seconds to preview R07: A is accepted at 162 milliseconds while B fails at 156 milliseconds. The latency rule picks the failed B response. The quality rule picks A for this prompt. The final selected mean is 120 milliseconds for latency selection and 121 for quality selection, rounded to whole milliseconds. These means describe the selected response latency, not the wall-clock cost of running and judging both variants.

Stress the evaluator and its limits

Inject Variant B answer regression. B now accepts eight of twelve. Quality selection still accepts eleven, concealing the deterioration if you inspect only the chosen outputs. Latency selection accepts eight and retains its 120-millisecond mean because the fault changes answer labels, not timing. Inspect individual rows before trusting an aggregate. Twelve fixed prompts establish neither statistical significance nor provider ranking. A real study should retain prompts, responses, configurations and judge criteria, test representative tasks, and account for uncertainty and cost. This lesson provides no token charges, external inference or current model recommendations.

Inspect the same cohort at your own pace

Policy and Fault fixture restart the twelve fixed requests at zero. Play requests advances the sixty-second inspection; Pause requests freezes it, and Resume requests continues. Restart retains the chosen policy and fault but rewinds the observations. Timeline pauses and seeks directly; a request completes at each five-second boundary. Speed selects half, normal or double inspection pace and does not change assignments, answer labels or modeled milliseconds. At sixty seconds Replay requests starts again. Keyboard users can activate buttons with Space and move the timeline with arrow keys. The observed-record table shows completed requests only; the current request text previews the active request before its completion. Export fixture JSON deliberately includes the full computed fixture and separately labels the current inspection position, so it must not be mistaken for a partial observation export.

Observed request records

The export includes the full computed fixture and separately identifies the inspection position.