CI/CD · RELEASE GATE
Agent CI/CD: keep a failing integration test from release
Push one commit through four integration tests with retries, a quarantine list and a required check, and see whether a real regression reaches release.
Your experiment
Commit c41 breaks the payments webhook, and its test int-3 was quarantined last month for flakiness. Start there, then try a regression in an active test, a flaky test with and without retries, and a release job that does not wait for integration.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Pipeline policy model: test results are fixed by the chosen ground truth (a real regression fails every attempt; the flaky test fails only its first attempt). No build, test runner or repository is executed. Retries apply per failing test. A release job that does not require the integration stage is modeled as proceeding regardless of its result.
Four integration tests and a release gate
Commit c41 has built and passed its unit tests. Four integration tests follow: int-1 checkout flow, int-2 search, int-3 payments webhook and int-4 email digest. A failing test is retried as many times as Retries for a failing test allows. Tests on the quarantine list are skipped when Skip tests on the quarantine list is checked, and the release waits for the integration stage only when it is a required check. Ground truth for this commit chooses what is really wrong.
The opening escape: a quarantined real failure
The payments webhook is broken, but int-3 was quarantined for flakiness. It is skipped, the other three tests pass, the stage is green and the release proceeds with the regression: an escaped regression. Clear the quarantine switch: int-3 runs, fails all three attempts (one try plus two retries) and the release is blocked, which is correct.
Required checks and real regressions
Choose Real regression in int-1. Checkout fails every attempt and the required integration stage blocks the release. Retries never turn a real regression green; they only add attempts to the trace. Now clear Integration stage is a required check for release: the stage is still red, but the release job no longer waits for it, so the regression ships. The required check is the gate, not the test result itself.
Flaky tests and retries
Choose No defect; int-2 flaky on first attempt. With Retries at 0 the first timeout fails the stage and a good build is blocked: a false block. With one retry int-2 passes on its second attempt and the release proceeds cleanly. Retries are the right tool for a known-flaky test and the wrong tool for hiding one: the quarantine example shows what happens when a test that once flaked is silenced instead.
Reading the run and its limits
Play, Next event, Restart, Speed and Inspection time step through the build, each test attempt, the gate and the verdict across sixty seconds; the table shows attempts and results per test. Results are dictated by the chosen ground truth, so the model shows the decision logic, not test design. Real pipelines also need quarantine expiry, owners for quarantined tests and evaluation suites for agent behaviour.