Agent dashboards: separate error rate from tail latency

Compare two panels over the same requests

Policy selects an Error-rate panel or Tail-latency panel. Both use the same twelve fictional completed requests and retain the full-cohort summary. Fixture chooses Routine traffic or Incident traffic. Routine traffic fails only E08; all twelve latencies remain below the teaching threshold of 250 milliseconds. Incident traffic fails E03, E04, E08 and E09 and adds 300 milliseconds to E04, E08 and E09. These are predetermined observations, not requests sent to an agent. The four-record window labels identify groups for inspection; the summary covers every completed event since restart, not just the newest group.

Separate a fast failure from a slow response

Select routine traffic and the error panel, then seek to sixty seconds. One of twelve requests failed, an error rate of 8.33 percent; no request was slow and nearest-rank p95 is 191 milliseconds. Switch to the latency panel and seek to sixty seconds again: no card is highlighted as slow, although E08 still failed. With incident traffic, the complete cohort has four errors, three slow requests and p95 of 491 milliseconds. E03 failed in 154 milliseconds: the error panel highlights it but the latency panel does not. Switching the panel changes the emphasis and labels, never the request latency or failure outcome.

Check a percentile boundary

With incident traffic seek to nineteen seconds: E01 through E03 are complete, so there is one error, no slow request and p95 of 154 milliseconds. Move to twenty seconds to include E04: two errors out of four, one slow request and p95 of 491 milliseconds. This nearest-rank calculation sorts the observed latencies and chooses rank ceiling of 0.95 times the sample size. With twelve or fewer records it selects the maximum, so this tiny fixture cannot demonstrate a stable production tail estimate. At zero completed requests both percentile and error rate are unavailable, not zero latency or perfect health.

Inspect the completed prefix and record evidence

Play events reveals one completed record every five seconds across a sixty-second inspection. Pause events freezes elapsed time; Resume events continues. Restart clears time while retaining your chosen scenario and speed. Replay events begins again after completion. Timeline seeks and pauses; at nineteen seconds only three records have completed, while twenty seconds includes E04. Speed selects half, normal or double pace and settles the elapsed interval at the previous rate before changing. A hidden tab pauses. Changing Policy or Fixture starts again at zero. Expand experiment enters full screen and Exit full screen restores the page. Capture baseline saves the visible result and settings; Replace baseline updates it and Clear baseline removes it. Notes persist in this browser and Export notes downloads the observations. Read the text as well as the card colors: waiting and active cards expose no outcome before their completion boundary.

Define the query before using a dashboard

This local fixture has no telemetry backend, query engine, refresh delay, tenant filter, missing samples, time-weighted rate or histogram aggregation. Error percentage divides failed completed requests by all completed requests, including errors in the latency calculation. A production dashboard may need separate success and error latency distributions and explicit handling of timeouts. The 250 millisecond boundary is illustrative, not an SLO promise. The timeline is for inspection and does not measure request execution time. Neither the selected panel nor the synthetic incident changes an actual service. Compare multiple signals and document the denominator before drawing operational conclusions.