EVALUATION HARNESS · REPRODUCIBILITY
Evaluation harness: repeat a deterministic run
Run the same ten recorded cases several times and find out whether sampling and a live tool clock make the gate decision change between runs.
Event trace
Computed data
Final metrics
Each case has a fixed quality margin; a run adds temperature × a fixed pseudo-random draw in [−1, 1] that changes per repeat unless the seed is fixed. This is a stand-in for sampling, not a language model, and real providers do not always guarantee identical output for a fixed seed. Case C7 depends on the wall clock: repeats start at 16:59 and run two minutes apart; the store closes at 17:00.
The same suite, three answers
Defaults: Sampling temperature 0.7, seed not fixed, 3 repeats, live tool clock, Gate decision rule First run only, Gate threshold 70%. The repeats score 60%, 70% and 60%. Run 1 fails C3, C5, C6 and C9; run 2 fails C3, C4 and C7 at 17:01; run 3 fails C3, C5, C7 and C9. The first run blocks, but run 2 alone would ship: FLAKY GATE, with C4, C5, C6, C7 and C9 changing outcome. The board shows one bar per repeat against the threshold.
Two sources of variance
Tick Fix the sampling seed for every repeat: 80%, 70%, 70%. Every run now ships, but C7 still flips once the clock passes 17:00 closing time: UNSTABLE CASES. Tick Freeze the tool clock at the recording time as well: 80% in every repeat, REPRODUCIBLE PASS. Freezing the clock alone gives 60%, 80%, 70%, still FLAKY GATE. A temperature of 0 removes sampling the same way and also scores 80%, 70%, 70% until the clock is frozen.
What the decision rule hides
Back at the defaults, Mean of runs gives 63.3% and Every run must pass uses the lowest run, 60%; both block, yet neither makes the run reproducible. A threshold of 50% ships every repeat, so the verdict becomes UNSTABLE CASES. The case table lists each repeat as pass or fail with the sampled term, or closed for the clock failure.
How many repeats
Set Repeats of the suite to 1: the verdict is SINGLE RUN, because one sample cannot show variance. At 5 repeats the defaults score 60%, 70%, 60%, 70%, 60%; with the seed fixed and the clock frozen, all five score 80%. More repeats cost more and expose variance; they do not remove it.
Replay and limits
Any change recomputes the run and rewinds it; the Complete-run preview states the outcome at once. Play and Pause walk through the events, Next event jumps one event, Restart rewinds, Speed runs from 0.25x to 2x and Inspection time seeks from 0 to 60 seconds, where the outcome and final metrics appear. Each case has a fixed margin and a run adds temperature times a fixed pseudo-random draw between minus 1 and 1. This stands in for sampling; real providers do not always guarantee identical output for a fixed seed.