Incident Runbooks for OCI: Templates That Get Used

A runbook is not documentation. Documentation explains how a system works to someone with time and coffee. A runbook tells a stressed person with partial information what to do next, at the moment their judgment is most degraded and the cost of error is highest. Those are different documents with different physics, and most runbook libraries fail because they were written as the first kind and filed as the second. This article describes the runbook that survives contact with a real incident: how it is structured, how it gets found at 3 a.m., and the maintenance loop that keeps it from rotting back into documentation. It is part of our series on advanced OCI operations.

Why runbooks rot

Three forces kill runbook libraries, and naming them shapes the cure. Scope greed: the library tries to cover everything, so most entries are thin, stale, and untrusted, and the distrust spreads to the good ones. Distance from the alert: the runbook lives in a wiki, the alert lives in a pager, and at 3 a.m. nobody goes searching, so even accurate runbooks go unused. And maintenance by intention: everyone agrees runbooks should be updated, no mechanism makes it happen, and each console redesign or architecture change quietly invalidates another page. The working library inverts all three: few scenarios chosen by evidence, every alarm carrying its runbook with it, and updates triggered by events, drills, and incidents rather than by good intentions.

Choose scenarios by evidence, not imagination

The library should start small and earned: the ten or fifteen scenarios that the estate's own history says will recur. Pull a year of incidents and pages and rank by frequency and pain. For a typical OCI estate the list converges on a recognizable core: a database node eviction or failover, a full filesystem or archive destination, an expiring or expired certificate, a connectivity drop on the fast connect or VPN path, an autoscaling event that hit a service limit, a backup job failure streak, an identity misconfiguration locking out a team, and the cost anomaly that is technically not an outage but pages someone anyway. Each scenario earns a runbook when it has happened, or when its blast radius is too large to learn by experience, the regional failover being the canonical example. Everything else waits. A library of twelve trusted runbooks outperforms a library of eighty plausible ones, every time it matters.

The anatomy of a runbook that works under stress

The template that survives incidents is rigid and short, one screen of structure before any detail. Trigger: which alarm or symptom opens this runbook, in the alarm's own words. Impact statement: what the business feels, so severity calls take seconds. Verification: two or three checks that confirm this is the scenario the runbook thinks it is, because misdiagnosis at step one poisons everything after. Actions: numbered, imperative, copy ready commands with placeholders clearly marked, each step stating what success looks like before the next. Stop signs: the explicit conditions under which the responder stops and escalates rather than improvises, with the escalation path named, role by role, not person by person. Communication: who gets told what and when, with a template sentence the responder can paste rather than compose. Aftermath: what to record, what to restore to normal, and the link to the postmortem process. Every section earns its place by saving a decision under stress, and anything that does not save a decision belongs in documentation instead.

Putting the runbook inside the alert

The single highest leverage integration in incident response is also the simplest: the alarm carries the runbook link. OCI Monitoring alarms accept a body, and the body should contain the URL of the exact runbook for that alarm, so the page that wakes the responder contains its own instructions. The discipline this enforces upstream is healthy too: an alarm that cannot name its runbook is either an alarm that should not page, or a scenario missing from the library, and either finding improves the estate. The runbooks themselves belong in version control, plain markdown in a repository, reviewed through the same pull request flow as code, rendered wherever the team reads, because version control gives the library the two properties wikis lose: a change history someone can audit, and a review gate that catches the well meant edit that breaks the command syntax. Mature estates go one step further and automate the verification steps into scripts the runbook calls, shrinking the human's job to judgment, and the same philosophy that keeps infrastructure definitions honest, covered in drift detection on OCI, applies to the gap between what runbooks claim and what the estate actually runs.

Drills: the maintenance mechanism that works

The only reliable way to keep a runbook current is to execute it when nothing is wrong. A drill cadence of one runbook per month, run by someone who did not write it, in a test environment or a read only walkthrough where the actions are destructive, surfaces every form of rot at once: the renamed console screen, the changed command flag, the escalation contact who left, the step that assumed a system retired last spring. The drill's output is a pull request, and the runbook's header carries its last drill date, so staleness is visible at a glance. The other maintenance trigger is the incident itself: every postmortem asks whether the runbook was used, whether it was right, and what edit would have saved the most time, and the edit ships before the postmortem closes. Between drills and postmortems, no runbook goes more than a year untouched by a human hand, and the library's trust, the real asset, compounds instead of decaying. The backup and restore scenarios deserve special attention in the drill rotation, since they pair with the restore testing calendar described in an estate wide backup strategy on OCI, and the two exercises can share quarters.

A six step build sequence

Runbooks as an operating asset

A trusted runbook library changes the economics of the on call rota: it converts the knowledge of the most senior engineer into a form the newest responder can execute, which shortens incidents, flattens the 3 a.m. seniority lottery, and makes the rota survivable for a wider pool of people. It is also, candidly, the asset that makes external operations support work: when our OCI monitoring practice takes the first response seat for an estate, the runbook library is the contract between our responders and your systems, drilled and versioned exactly as above, backed 24/7/365 and informed by incident patterns across 500+ OCI engagements. Whoever holds the pager, the test of the library is the same: open a runbook at random and ask whether a competent stranger could execute it tonight. If the answer is yes twelve times, the estate has incident response. If the answer is mostly, it has a wiki.