Agent metrics: count the requests behind the chart

Last eight completed requests

A counter and a distribution answer different questions

Each fictional request has an arrival time, completion time, tenant, latency and success status. The cumulative error counter increments only when a failed request completes. A latency observation is recorded for every completed request, including failures. The cumulative completion count is therefore the histogram population, while the error count is only a subset. Arrived equals completed plus in-flight at every snapshot. This conservation check is more useful than treating a changing dashboard badge as evidence of a correctly instrumented service.

Try the incident while comparing cumulative and windowed values

Restore defaults and inspect second thirty. There are 297 completed requests in total. The trailing ten-second window contains 98 completions and 28 failures; its nearest-rank p95 is 540 milliseconds. The denominator is not assumed to be one hundred simply because the arrival rate is ten per second: completion times move observations across window boundaries. Inspect second fifty-five. The current window has one hundred completions and zero failures, even though the cumulative failure counter still retains the incident. A falling window error count does not mean the cumulative counter went backwards.

Read the histogram without double counting

The upper bounds are one hundred, two hundred fifty, five hundred, one thousand and positive infinity milliseconds. A two-hundred-millisecond request contributes to every bucket whose bound is at least two hundred. Consequently the infinity bucket equals all completed observations in the selected window, and neighboring cumulative counts must not be summed as independent populations. Increase Slow extra latency and inspect which bounds those observations cross. Changing Target changes the policy verdict, not the request timings or histogram. Compare tenant filters by exact counts rather than averaging their p95 values to estimate the combined p95.

Create a repeatable request cohort

Requests per second sets zero through twenty evenly spaced arrivals during sixty modeled seconds. Baseline error percentage and Incident error percentage feed an integer remainder accumulator; each time it reaches one hundred, that request fails and the remainder is reduced by one hundred. Incident start and Incident duration choose which arrival seconds use incident behavior. Base latency is increased by a deterministic zero-to-forty-millisecond request-index variation. Slow extra latency is added to ten percent of baseline arrivals and sixty percent of incident arrivals using another remainder accumulator. These schedules can correlate with alternating alpha/beta tenant assignment; they are a repeatable teaching workload, not independent random production observations.

Choose the aggregation and good-event rule

Tenant filter selects all, alpha or beta without regenerating the underlying request records. Trailing window sets one through sixty seconds. At inspection time t, it includes matching completed requests strictly after max of zero and t minus window, through t inclusive. Availability counts successful completed requests as good. Successful and fast additionally requires duration at or below Latency limit; a failed fast request is still bad. Target sets eighty through 99.99 percent in hundredths of a percentage point. The comparison uses integer good-count arithmetic against the configured target, not the rounded percentage printed on a card.

Inspect progression and the evidence

Play advances through sixty seconds; Pause freezes the selected snapshot. Next second advances one integer second, Restart rewinds, Show final result selects sixty, Speed changes viewing pace and Inspect time scrubs. Each input change rebuilds the cohort or aggregation and pauses at zero; Restore defaults resets every setting. The line shows the selected trailing-window SLI against the target. Bucket bars show counts at or below each latency bound, so they are cumulative, not disjoint bars to add together. The last eight completed requests table exposes start, completion, latency, tenant and result. Exact evidence includes all selected sample IDs and every aggregate denominator.

Keep the missing-data and model limits explicit

At zero traffic or before a matching completion, there is no measured success fraction, percentile or objective verdict. Unknown is not healthy. Cumulative counters only cover observed completions since time zero and never reset during a run. No counter-reset detection, exporter loss, scrape cadence, clock skew, timeout censoring, histogram interpolation or Prometheus server is implemented. Percentile p95 is nearest-rank over the full raw window sample, not estimated from bucket bounds. The fixed arrival and response schedule has no queue, shared capacity, retries or feedback. Some final arrivals complete after sixty seconds and remain in flight at the horizon. This is not evidence that a real agent meets an SLO.