RELEASE ENGINEERING · TRAFFIC CONTROL ROOM
Canary failure: stop a bad candidate
Route individual requests, observe both versions and stop candidate traffic when the measured release gate fails.
Scroll the diagram horizontally to inspect every label. The panels below give the same state as text.
Your experiment
Run with candidate errors at 70%, then disable automatic rollback. Compare user-visible errors and the exact request where the gate rejects the release.
The preview is the computed end of this same replay. Capture a baseline, change one parameter and compare the cohort evidence. “Within limits” is not proof of safety or an automatic production promotion.
Recent request trace
Offline, seeded traffic exercise; no deployment, provider API or real flag service is used. Two requests per second are evaluated without a capacity queue. Control latency is 100–200ms; candidate latency spans 75–125% of its configured center. Outcomes use independent fixed pseudo-random streams. The gate requires enough samples in both cohorts, then rejects an excessive observed error-rate difference or candidate p95. It is not a statistical significance test. Metrics accumulate from start, without a sliding window, confidence interval or dependency failures. Shadow responses never serve users; safe handling of real side effects is not implemented. Google SRE: canarying releases.
How the opening canary fails
A canary sends a small share of real traffic to a candidate and compares it with the version serving the rest. One hundred twenty requests arrive at 2 per second, 25 percent go to the candidate, and from 10 seconds the candidate fails with probability 30 percent. The first candidate request is at 9.5 seconds. At 25 seconds the candidate has 4 errors in 8 requests, 50.0 percent against 2.4 percent for control, and Stop candidate traffic on rejection rolls back. The final line shows control at 0.9 percent because control keeps serving, ending with 1 error in 112. Users saw 5 errors, 4 from the candidate.
Share decides how soon evidence exists
Set Initial candidate share (%) to 5. The candidate gets only 5 requests, the first at 22.5 seconds, never reaches 8 samples, and the gate reads Insufficient samples at 60 seconds even with Candidate error probability (%) at 70 and 4 of the 5 failing. At 10 percent rollback comes at 49 seconds, at 50 at 19, and at 75 at 11.5. User errors do not fall steadily with smaller shares: 20 percent gives 7, 25 gives 5 and 40 gives 4, because each request uses fixed pseudo-random draws. Size the share so the minimum sample arrives within the analysis window.
Late failures are diluted
Set Candidate fault begins (s) to 30. Rollback waits until 42.5 seconds, after 17 candidate requests with 2 errors, 11.8 percent against 1.5. At 40 it comes at 52.5 seconds after 24 requests and 3 errors, 12.5 against 1.2. Metrics accumulate from time zero, so healthy early requests dilute a late fault; real canary analysis compares recent windows. Onset at 0 or 5 changes nothing, since no candidate request arrives before 9.5 seconds.
Minimum samples and automatic rollback
Minimum samples per cohort 1 rolls back at 10 seconds after 2 candidate requests and 1 error, 50.0 against 5.6 percent. At 4 rollback comes at 21 seconds, at 12 at 29, and at 20 at 49 with 9 user errors. A tiny minimum would also react to one unlucky failure of a healthy release; this fixture cannot show it, because Candidate error probability 2 draws no candidate error in 31 requests. Restore 8 and clear Stop candidate traffic on rejection: the gate says Reject candidate but keeps routing, ending with 11 errors in 31 candidate requests and 12 user-visible errors.
Thresholds and remaining controls
Allowed error difference (points) matters only near the limit. Set Candidate error probability to 10: rollback at 25 seconds on 1 error in 8, 12.5 against 2.4. Raise the allowed difference to 15 and the same candidate ends Within configured limits, 2 errors in 31. Control error probability (%) 20 still rolls back at 25 seconds, 50.0 against 23.8, but users see 30 errors. With candidate errors at 2, Candidate latency center (ms) 420 gives p95 508 ms, above the 500 ms Candidate p95 limit (ms), and rolls back at 25 seconds; 400 gives 489 ms and passes. Strategy also offers progressive, blue/green and shadow; share is disabled for the last two, and Blue/green cutover (s) applies only to blue/green.
Playback and limits
Play and Pause run the 60-second replay, Next request steps half a second, Restart rewinds, Speed changes pace and Time scrubs; editing an input recomputes and replays. This seeded fixture is not an ADK API or deployment system. There is no significance test, sliding window, session pinning, warm-up or drain, and rollback is instant.