On-Call Runbooks

What is a runbook

A runbook is a playbook for responding to a specific alert or incident type. It's not theory—it's the exact commands, checks, and decisions that an engineer should execute when an alert fires at 3am. The best runbooks are written by engineers who have already debugged the problem and lived through the false-alarm rate. They answer: What does this alert mean? Is it real? How do I confirm? What's the fix? Who do I escalate to?

Runbooks live in version control, reviewed like code, and updated after every incident. They're the artifact of blameless postmortems: if an engineer made a wrong call, it means the runbook wasn't clear enough—fix the runbook, not the engineer.

Essential structure

Alert summary: What is this alert? (e.g., 'API p99 latency > 500ms'). What does it usually mean? In one sentence, why should an on-call engineer care right now?

Symptoms and confirmation: What does the customer see? Query dashboard, logs, or traces to confirm the alert is not a phantom (stale data, sensor misconfigration). Example: 'Check curl $APP_ENDPOINT/health and verify response time.'

Common root causes: List the top 3—5 things that have triggered this alert historically. 'High CPU from batch job', 'Database connection pool exhausted', 'Disk space low on log partition'. Each with a quick way to check (e.g., free -h, df -h /var/log).

Diagnostic commands: Exact commands to run. Not 'check logs'—run grep ERROR /var/log/app.log | tail -20. Include query examples for your observability stack (Prometheus query, Grafana dashboard link, etc.).

Remediation steps: The fix, in order. If step 1 doesn't work, try step 2. Include rollback procedures. If you restarted the service, how do you confirm it's healthy?

Escalation path: If after 5 minutes it's not resolved, who else gets paged? Database team? Oncall SRE? Customer success for notification?

Real example — database connection pool exhaustion

Alert: 'DB Connection Pool at 95%'

Confirm it's real: SELECT COUNT(*) FROM pg_stat_activity; Query Grafana dashboard 'Database Connections' tab. Are we seeing a genuine spike or sensor lag?

Top causes: Slow query hogging connections. N+1 query in recent deploy. Replication lag causing queuing. Third-party service timeout cascading into our retry loop.

Quick diagnosis:

SELECT query, state, wait_event FROM pg_stat_activity WHERE state != 'idle'; to find long-running queries. If one dominates, kill it: SELECT pg_terminate_backend(pid); Check recent deployments in git log --oneline -20. If there's a new service called, check its timeout config.

Fix (in order): Restart the problematic service that's opening too many connections. If that doesn't drop the count, increase pool size (temporary config, roll back after). If it's a slow query, kill the query and run REINDEX on the slow table. Escalate to database team if none of these work.

Escalate after: 10 minutes of no improvement → page database on-call. 15 minutes → notify support (customer might already be complaining).

Diagnostic commands best practices

Use absolute paths and full flags. 'Check the logs' is useless at 3am. Write tail -100 /var/log/nginx/error.log | grep ERROR. Include the full path; different deployments differ.

Provide metrics queries, not just commands. Include Prometheus or Grafana queries as examples. If your monitoring system changed between deployments, update the runbook. A query that worked in prod-us-west doesn't work in prod-eu?—document it.

Time-bound your checks. 'Recent errors in the last 5 minutes'—write grep '$(date -d "5 minutes ago" +"%H:%M:%S")' /var/log/app.log or use your observability tool's time-window syntax.

Show output shape. 'Run this query' is vague; 'Run this query, expect a row with 3 columns (name, value, timestamp)' is clear. If the output is empty, it means X.

Living documentation — keeping runbooks current

A runbook written after an incident is worth 10x a theoretical one. After every incident, the postmortem should include: 'Did the runbook for this alert match reality? If not, fix it now.' Assign one person to update the runbook same day.

Version-control your runbooks in your docs repo or wiki, same as code. Add a 'Last updated' date at the top. If it says 'Last updated: 2024', on-call engineers won't trust it. Rotate responsibility: each on-call shift, one person audits one runbook for accuracy and age.

Remove runbooks for alerts that no longer fire (they accumulate fast). After a major system redesign, deprecated dashboards and queries clutter the runbook—archive them or delete them.

Testing and validation

A runbook that's never been run doesn't work. Schedule quarterly runbook drills: simulate the alert (inject high latency, kill a service, fill the disk) and have an on-call engineer follow the runbook verbatim. Time it. Did they actually resolve it in 10 minutes, or did they get stuck on step 3?

The best validation is handing the runbook to someone who's never seen it before (new hire, engineer from another team) and letting them execute it. If they get confused, the runbook failed—rewrite it.

After production incidents, run the exact runbook steps again in a safe environment (staging) to confirm they still work. Tools change; log locations move; Kubernetes APIs shift. An untested runbook is a false runbook.

Escalation and handoff patterns

Clear escalation saves lives (of your sleep). Runbook should state: 'If after X minutes the alert isn't resolved, page Y. If after Z minutes it's still not better, wake up Z+1.' Specific times, specific people. 'Escalate to whoever is available' is chaos at 3am.

Document the handoff: what information does the next person need? Copy the last 50 lines of relevant logs. Screenshot the dashboard. Share the hypothesis (we think it's slow queries, not disk space). Don't just wake someone up and say 'it's broken', because they'll spend 10 minutes re-diagnosing.

Escalation should also be a learning signal: if you're escalating more than 2× per quarter for the same alert, the root cause is architectural, not operational. File a ticket to redesign the system, don't just escalate forever.

Common pitfalls

Overly technical runbooks. A runbook for the database team can assume kernel debugging; a runbook for on-call eng from any background cannot. Write for the audience: on-call shift is often someone from another team or a junior.

Assuming the on-call person knows your codebase. Don't write 'restart the cache service'—write 'SSH to cache-prod-1, run sudo systemctl restart redis, verify with redis-cli ping'.

Runbooks with no diagnostic phase. Alert fires, human immediately runs 'rolling restart all services'. This is dangerous. Always: confirm the problem, form a hypothesis, execute a specific fix. A runbook that skips diagnosis is a recipe for cascading failures.

No rollback procedure. Every action needs an undo. If the runbook says 'scale the service to 20 replicas', also say 'after 10 minutes, scale back to 5 if error rate doesn't drop'.

Runbooks that assume things are always broken in the same way. Document decision trees: 'If CPU is high, follow path A. If latency is high but CPU is normal, follow path B.' Branching logic saves time.

Tools and automation

Runbook platform: Confluence, a Git wiki (like Markdown in docs/ folder), or specialized tools (Incidentio, VictorOps, PagerDuty) often let you attach runbooks to alerts. Clicking an alert opens the runbook without a separate search.

Runbook automation: The ultimate runbook is one that auto-remediates. If 90% of occurrences are fixed by 'restart the service', build a alert-driven restart automation. But—only automate the parts you're 100% confident about. Manual steps for anything risky (data deletion, DNS changes).

Runbook templates: Don't start from scratch. Define a company template: structure, required sections, example output. Make it a checklist—'Alert Summary? Check. Confirmation steps? Check. Escalation? Check.'

Building effective runbooks — template and checklist

Start with an incident. The best runbooks are written immediately after an engineer spent 45 minutes debugging something. Capture that experience now, while it's fresh.

Get feedback from the person who will use it. If it's for on-call rotation, have on-call engineers review it. If it's for a new team member, have a new team member test-run it.

Include context, not just commands. 'This alert fires when the rate limiter key expires in Redis but the app still thinks it's cached' is more useful than just 'clear Redis cache'.

Link to dashboards and logs. Don't describe how to find the Grafana dashboard—link it. Include a Datadog, Splunk, or Loki query URL so the on-call engineer lands directly on the data.

Keep it short. A good runbook is 1—3 pages. If it's 10 pages, break it into multiple runbooks. Cognitive load at 3am is high; be concise.

Continuous improvement cycle

Runbooks aren't 'done'. After each incident or drill, capture what changed: did the command syntax work? Did the dashboard exist? Was the escalation path correct? Update it immediately.

Set a calendar reminder: every 6 months, pick one runbook at random and have someone unfamiliar execute it (in a safe environment) and mark it as 'validated' or 'needs update'. Over time, this keeps the whole fleet fresh.

Measure the impact: track MTTR for alerts with runbooks vs. without. You should see a 2—3× improvement. If you don't, the runbook is either not being used or not actually helpful—investigate why.

The best signals that a runbook is working: (1) on-call engineers say 'I just followed the runbook, took 5 minutes'; (2) escalations drop; (3) post-mortems don't blame 'didn't know what to do'.