CANARY · STEP ANALYSIS
Canary exposure: speed versus blast radius
A bad candidate goes through a canary ladder; trade the first step’s size, bake time and sample requirement against how many users see failures.
Your experiment
1,000 requests a minute; the candidate fails 8% of requests against 1% for the stable version. The ladder runs the first step, then 25%, 50% and 100%, and analyses each step at its end. The budget is 50 extra failed requests. Start with a 1% first step and work out why it does worse than 5%.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, Speed changes the pace and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Deterministic model: the analysis detects the regression exactly when a step has enough requests, extra failures are counted as 7% of canary requests, and traffic is constant. Real analysis is statistical, uses several metrics and can be run continuously; the trade-off between sample size, step size and exposure remains.
A small first step that learns nothing
1,000 requests a minute arrive and the candidate fails 8% of them against 1% for the stable version. The ladder is the first step, then 25%, 50% and 100%, each baked for 10 minutes and analysed at its end with at least 300 canary requests. A 1% first step serves only 100 requests: inconclusive, and the rollout advances anyway. The 25% step then serves 2,500 requests before the analysis rolls back at 20 minutes: 182 extra failures, CAUGHT LATE against a budget of 50.
Size the first step to the sample
Set First canary step to 5%: 500 requests are enough, the regression is caught at 10 minutes with 35 extra failures: CAUGHT EARLY. At 10% it costs 70 and at 25% 175; at 2% the first step is again inconclusive and the cost is 189. The smallest step is not the safest one; the safest is the smallest that produces a conclusive result.
Inconclusive should mean wait
Set When a step is inconclusive to Stay and extend the step with a 1% start. The step runs 30 minutes until it has 300 requests, then rolls back with 21 extra failures: safe, but slow. With Requests the analysis needs per step at 100, the 1% step is conclusive at 10 minutes with 7 extra failures. Asking for 600 with a 5% start and extension costs 70, because the step must run 20 minutes.
Bake time and a good candidate
Bake time per step scales everything that happens before the analysis: 20 minutes with a 1% start costs 364 extra failures, 5 minutes costs 91. Clear Candidate is bad: a healthy candidate reaches 100% in 30 minutes when inconclusive steps advance, in 50 minutes when they extend, and in 20 minutes from a 25% start. Speed and blast radius are traded against each other by the same three numbers.
Reading the run and its limits
Play, Next event, Restart, Speed and Inspection time step through every step decision and the summary across sixty seconds; the table lists each step’s window, canary requests, extra failures and analysis result. Detection here is exact once the sample is reached. Real canary analysis is statistical, compares several metrics against a concurrent baseline and may run continuously.