ROLLBACK · EXPOSURE, EVIDENCE, STATE
Rollback: preserve evidence and stop exposure
A bad prompt release gives wrong refund answers; choose the alert window and rollback method, and decide what happens to its evidence and sessions.
Your experiment
v2 serves 120 requests a minute and answers 15% of refund questions wrongly. The incident budget is at most 150 wrong answers, preserved evidence and no broken sessions. Start with a five-minute alert and a full redeploy, then change one choice at a time.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, Speed changes the pace and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Deterministic incident model: a constant error rate, an alert that fires exactly at the end of its window, and fixed rollback durations per method. Revision-scoped logs are assumed to disappear with a deleted revision, as they do when retention is tied to the deployment. The flag rollback keeps the v2 binary, so its session format stays readable. All numbers are illustrative.
Three things a rollback must do
A new prompt release, v2, serves 120 requests a minute and gives a wrong refund policy in 15% of answers, 18 a minute. It also writes 40 sessions in a new state format. The incident budget allows at most 150 wrong answers, requires the v2 evidence to survive for the review, and allows no broken sessions. In the opening run the alert fires after its 5-minute window and the team rebuilds and redeploys v1, which takes 8 minutes: 13 minutes of exposure and 234 wrong answers. The v2 revision is deleted, and v1 cannot read the 40 sessions: all three requirements are missed.
Stop exposure with a switch, not a build
Set Rollback method to Switch traffic to warm v1: exposure stops at 6 minutes after 108 wrong answers. Turn off the v2 prompt flag is faster still, 5.5 minutes and 99 wrong answers. The Alert evaluation window is the larger term: with a switch, a 7-minute window still fits the budget at 144 wrong answers, but a 10-minute window reaches 198. With the flag, an 8-minute window already reaches 153. A 1-minute alert followed by a redeploy still exceeds the budget at 162.
Keep the evidence
Deleting the bad revision feels like cleaning up, but its request logs, traces and exact prompt version go with it when they are scoped to the deployment. Set The bad v2 revision to Scale it to zero and keep it: the revision serves nothing, costs almost nothing and remains queryable for the incident review. Evidence that disappears with the rollback turns a fixable defect into a guess.
State written by the bad version
Sessions written by v2 decides whether rolling back breaks users. With New format v1 cannot read, the 40 sessions written by v2 fail on their next turn after a traffic switch or redeploy. Optional additions v1 ignores keeps them all readable, so the switch with kept evidence becomes CONTAINED at 108 wrong answers. The flag rollback keeps the v2 binary, so it never strands those sessions; it also only works when the change was put behind a flag before release.
Reading the run and its limits
Play, Next event, Restart, Speed and Inspection time step through the release, the alert, the rollback, the evidence decision, the session outcome and the review across sixty seconds; the table summarises each choice and its effect. The error rate is constant, alerts fire exactly at the end of their window and rollback durations are fixed. Real incidents add detection noise, partial rollbacks and data repair, which this model leaves out.