Lesson 6 of 9 · DURING RECOVERY · 3 min read

Runbooks and automated recovery: what each is good for

Short answer

A runbook is a written procedure people follow during an incident; automated recovery is a tool that performs the steps itself. Automation shortens recovery and removes errors made under pressure, but it does not decide when to recover, where to, or whether the target is clean. Keep a short runbook around the tool.

What belongs in a runbook?

A useful cloud DR runbook is short. It names who declares a disaster and who can approve a failover. It lists the order in which applications come back, the target region or account, how to confirm the target is clean, and how to tell users. It links to the tool's own procedure rather than repeating it.

The CISA #StopRansomware Guide recommends restoring systems in order of the critical services they support, which is a decision for the runbook, not for a tool.

What can automation take over?

The slowest manual steps are working out dependencies, creating resources in the right order and reconnecting them. That is where tools differ most. Arpio describes orchestrated failover in every tier, automated failback in Premium and orchestrated ransomware recovery in Enterprise. Firefly describes scheduled and on-demand capture with dependencies recorded automatically, and a restore that generates Terraform for review and apply. Commvault Cloud Rewind describes automated capture and recovery of whole application stacks.

Where does automation stop?

  • Deciding to recover. A tool can make failover one step; someone still decides to take it.
  • Choosing a clean target after an attack. A tool can rebuild into an isolated account; people decide which point in time is clean.
  • Data that the tool does not hold. An infrastructure rebuild tool without data protection needs the data restore to be coordinated alongside it.

Write these three limits into the runbook as named decisions, each with an owner and a fallback, so nobody has to work out who decides while the incident is running.

How do you keep both current?

The AWS guidance on testing recovery applies to people as much as tools. Run the runbook during every test, and update it when the test shows a step is wrong or missing.

What should a recovery test check in the runbook?

Use each test to check the parts of the runbook a tool cannot: that the people named are still the right people, that approvals can be given out of hours, that contact details and escalation paths work, and that the order of applications still matches what the business needs. Record how long each decision took as well as how long the tool took. In many incidents the waiting time between steps is longer than the steps themselves.

What should you take from this lesson?

Let the tool do the rebuild and keep the runbook to decisions, order, targets and communication. A one-page runbook that has been rehearsed is worth more than a long one that has not.

Sources