SIMULATOR · SAME CASES, BOTH VERSIONS
Agent simulator: replay the same case before comparing versions
Compare a baseline and a candidate agent on simulated conversations, with fresh or paired scenario seeds and a deterministic or sampling user simulator.
Your experiment
A user simulator generates support conversations from 40 scenario seeds of known difficulty. v1 resolves scenarios below difficulty 0.6. Start with fresh seeds for each version and 10 episodes, then pair the seeds, add episodes, change what v2 really does and let the simulator sample.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, Speed changes the pace and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Deterministic stand-in for a simulator: each seed has a fixed difficulty and a partial-shipment flag, and a version resolves a scenario when the difficulty is below its skill. With sampling on, borderline scenarios flip on a fixed pattern that differs between versions. The conclusion rule calls a difference of one episode or more a change. No model or simulator runs.
A better score from easier cases
A user simulator generates support conversations from 40 scenario seeds of known difficulty. v1 resolves scenarios below difficulty 0.6. Candidate v2 resolves harder ones, up to 0.7, but fails every partial-shipment case, so across all 40 seeds it is truly worse: 3 fewer resolved. With the opening fresh seeds, v1 runs seeds 1 to 10 and v2 runs seeds 21 to 30. v2 resolves 8 of 10 against v1’s 5 and looks better, but its scenarios had a mean difficulty of 0.28 against v1’s 0.56: MISLEADING COMPARISON.
Pair the seeds
Choose The same seeds for both versions. Both now face seeds 1 to 10 with mean difficulty 0.56, and the bias disappears: 5 against 5. That is still misleading, because these ten seeds contain one partial-shipment case that v2 loses and one hard case it gains. With Episodes per version at 20 the paired run shows 10 against 9 and the right conclusion; with only 5 it happens to show 3 against 2. Pairing removes the scenario mix from the comparison; it does not remove the need for enough cases.
When the bias points the right way
Set What really changed in v2 to Better on hard cases: v2 is truly better by 5 across all seeds, and fresh seeds show 5 against 10, which is correct only by luck; paired seeds show 5 against 7. Choose Nothing: fresh seeds still report v2 as better, 5 against 10, for a candidate that did not change at all, while paired seeds show 5 against 5. Fresh seeds measure the scenario sample as much as the agent.
A sampling simulator adds its own noise
Clear Simulator is deterministic per seed. Borderline scenarios now flip between runs, differently for each version. With Better on hard cases and 10 paired episodes, both versions resolve 6 and the real improvement disappears; at 15 or 20 episodes it shows again, 9 against 10 and 11 against 12. Fix the simulator seed or temperature when comparing versions, or run enough episodes to average the noise out.
Reading the run and its limits
Play, Next event, Restart, Speed and Inspection time step through the setup, the first episodes, the comparison and the conclusion across sixty seconds; the table lists every episode’s seed, difficulty and outcome for both versions. Difficulties and outcomes are a deterministic fixture standing in for a real simulator; the conclusion rule ignores statistical significance, which a real comparison should test.