What is a runbook
A runbook is a playbook for responding to a specific alert or incident type. It's not theory—it's the exact commands, checks, and decisions that an engineer should execute when an alert fires at 3am. The best runbooks are written by engineers who have already debugged the problem and lived through the false-alarm rate. They answer: What does this alert mean? Is it real? How do I confirm? What's the fix? Who do I escalate to?
Runbooks live in version control, reviewed like code, and updated after every incident. They're the artifact of blameless postmortems: if an engineer made a wrong call, it means the runbook wasn't clear enough—fix the runbook, not the engineer.
Essential structure
Alert summary: What is this alert? (e.g., 'API p99 latency > 500ms'). What does it usually mean? In one sentence, why should an on-call engineer care right now?
Symptoms and confirmation: What does the customer see? Query dashboard, logs, or traces to confirm the alert is not a phantom (stale data, sensor misconfigration). Example: 'Check curl $APP_ENDPOINT/health and verify response time.'
Common root causes: List the top 3—5 things that have triggered this alert historically. 'High CPU from batch job', 'Database connection pool exhausted', 'Disk space low on log partition'. Each with a quick way to check (e.g., free -h, df -h /var/log).
Diagnostic commands: Exact commands to run. Not 'check logs'—run grep ERROR /var/log/app.log | tail -20. Include query examples for your observability stack (Prometheus query, Grafana dashboard link, etc.).
Remediation steps: The fix, in order. If step 1 doesn't work, try step 2. Include rollback procedures. If you restarted the service, how do you confirm it's healthy?
Escalation path: If after 5 minutes it's not resolved, who else gets paged? Database team? Oncall SRE? Customer success for notification?
Real example — database connection pool exhaustion
Alert: 'DB Connection Pool at 95%'
Confirm it's real: SELECT COUNT(*) FROM pg_stat_activity; Query Grafana dashboard 'Database Connections' tab. Are we seeing a genuine spike or sensor lag?
Top causes: Slow query hogging connections. N+1 query in recent deploy. Replication lag causing queuing. Third-party service timeout cascading into our retry loop.
Quick diagnosis:
SELECT query, state, wait_event FROM pg_stat_activity WHERE state != 'idle'; to find long-running queries. If one dominates, kill it: SELECT pg_terminate_backend(pid); Check recent deployments in git log --oneline -20. If there's a new service called, check its timeout config.
Fix (in order): Restart the problematic service that's opening too many connections. If that doesn't drop the count, increase pool size (temporary config, roll back after). If it's a slow query, kill the query and run REINDEX on the slow table. Escalate to database team if none of these work.
Escalate after: 10 minutes of no improvement → page database on-call. 15 minutes → notify support (customer might already be complaining).
Diagnostic commands best practices
Use absolute paths and full flags. 'Check the logs' is useless at 3am. Write tail -100 /var/log/nginx/error.log | grep ERROR. Include the full path; different deployments differ.
Provide metrics queries, not just commands. Include Prometheus or Grafana queries as examples. If your monitoring system changed between deployments, update the runbook. A query that worked in prod-us-west doesn't work in prod-eu?—document it.
Time-bound your checks. 'Recent errors in the last 5 minutes'—write grep '$(date -d "5 minutes ago" +"%H:%M:%S")' /var/log/app.log or use your observability tool's time-window syntax.
Show output shape. 'Run this query' is vague; 'Run this query, expect a row with 3 columns (name, value, timestamp)' is clear. If the output is empty, it means X.
Living documentation — keeping runbooks current
A runbook written after an incident is worth 10x a theoretical one. After every incident, the postmortem should include: 'Did the runbook for this alert match reality? If not, fix it now.' Assign one person to update the runbook same day.
Version-control your runbooks in your docs repo or wiki, same as code. Add a 'Last updated' date at the top. If it says 'Last updated: 2024', on-call engineers won't trust it. Rotate responsibility: each on-call shift, one person audits one runbook for accuracy and age.
Remove runbooks for alerts that no longer fire (they accumulate fast). After a major system redesign, deprecated dashboards and queries clutter the runbook—archive them or delete them.
Testing and validation
A runbook that's never been run doesn't work. Schedule quarterly runbook drills: simulate the alert (inject high latency, kill a service, fill the disk) and have an on-call engineer follow the runbook verbatim. Time it. Did they actually resolve it in 10 minutes, or did they get stuck on step 3?
The best validation is handing the runbook to someone who's never seen it before (new hire, engineer from another team) and letting them execute it. If they get confused, the runbook failed—rewrite it.
After production incidents, run the exact runbook steps again in a safe environment (staging) to confirm they still work. Tools change; log locations move; Kubernetes APIs shift. An untested runbook is a false runbook.
Escalation and handoff patterns
Clear escalation saves lives (of your sleep). Runbook should state: 'If after X minutes the alert isn't resolved, page Y. If after Z minutes it's still not better, wake up Z+1.' Specific times, specific people. 'Escalate to whoever is available' is chaos at 3am.
Document the handoff: what information does the next person need? Copy the last 50 lines of relevant logs. Screenshot the dashboard. Share the hypothesis (we think it's slow queries, not disk space). Don't just wake someone up and say 'it's broken', because they'll spend 10 minutes re-diagnosing.
Escalation should also be a learning signal: if you're escalating more than 2× per quarter for the same alert, the root cause is architectural, not operational. File a ticket to redesign the system, don't just escalate forever.
Common pitfalls
Overly technical runbooks. A runbook for the database team can assume kernel debugging; a runbook for on-call eng from any background cannot. Write for the audience: on-call shift is often someone from another team or a junior.
Assuming the on-call person knows your codebase. Don't write 'restart the cache service'—write 'SSH to cache-prod-1, run sudo systemctl restart redis, verify with redis-cli ping'.
Runbooks with no diagnostic phase. Alert fires, human immediately runs 'rolling restart all services'. This is dangerous. Always: confirm the problem, form a hypothesis, execute a specific fix. A runbook that skips diagnosis is a recipe for cascading failures.
No rollback procedure. Every action needs an undo. If the runbook says 'scale the service to 20 replicas', also say 'after 10 minutes, scale back to 5 if error rate doesn't drop'.
Runbooks that assume things are always broken in the same way. Document decision trees: 'If CPU is high, follow path A. If latency is high but CPU is normal, follow path B.' Branching logic saves time.