Agent alerting: open an incident only after enough failures

Compare cumulative errors with a bounded window

Policy opens the alert when the selected count reaches two or three failed requests. Fixture chooses twelve fixed requests: Routine traffic fails only E08; Four-error incident fails E03, E04, E08 and E09. Evaluation window chooses all completed events since restart or the last four completed events, including the newest. Every request completes at a five-second boundary. Before that boundary its card says Awaiting completion and contributes nothing to the summary. E04 is observed at twenty seconds, not fifteen. These seconds provide an inspectable teaching timeline; they are not measured request latency or a monitoring scrape schedule.

Find the exact opening boundary

Keep the default threshold of two, cumulative window and routine traffic. Seek to sixty seconds: twelve events, one failure and zero openings. Select Four-error incident and seek to nineteen seconds: only E01 through E03 have completed, so one observed failure is below threshold. At twenty seconds E04 contributes the second failure and the alert opens once. At sixty seconds the summary has four total failures and nine completed steps with the alert open. Nine open steps are nine observations of one continuing alert, not nine incidents or nine notifications. Raise Policy to three; the first opening moves to E08 at forty seconds and five completed steps are open.

Observe recovery and a second episode

Select threshold two and Last four completed events. At twenty seconds E03 and E04 are both in the window, opening the first episode. At thirty-five seconds the window is E04 through E07 and has only one failure, so the alert resolves. It opens again at forty-five seconds when E08 and E09 are both present, then resolves at sixty seconds when the window is E09 through E12. Final counts are two openings, two resolutions and six completed steps open. Choose threshold three with the four-event window: the two failure clusters never supply three failures together, so no episode opens. A larger threshold reduces sensitivity here; it does not prove that the underlying failures are harmless.

Control playback and record a comparison

Play events runs a sixty-second inspection at normal speed. Pause events settles elapsed time and freezes the completed prefix; Resume events continues from there. Replay events starts again after sixty seconds. Restart preserves the selected policy, fixture, window and speed but clears the timeline. Timeline seeks and pauses. Speed selects half, normal or double pace; a change accounts for time already elapsed at the old speed. A hidden browser tab pauses. Changing any scenario input resets time to zero. Expand experiment and Exit full screen change the view. Capture baseline saves the visible outcome and settings; Replace baseline updates that comparison and Clear baseline removes it. Notes persist in this browser; Export notes downloads them. Event cards and the current/summary text provide outcomes without relying on color.

State the model limits before using the result

This is a deterministic count over twelve requests, not an ADK service, Prometheus query, statistical anomaly detector or production incident system. The four-event window is a count window, not a duration window; irregular traffic would give it a variable time span. There is no pending duration, hysteresis, cooldown, missing-data rule, severity, ownership, paging, notification routing, deduplication or acknowledgment. An opening and resolution are transitions of this one rule only. The cumulative counter cannot recover until restart, illustrating why a forever-growing counter needs an explicitly defined observation interval. The finite window can close even when a failed request remains inside it because the threshold is no longer met. Real alert rules and their notification delivery must be designed and tested separately.