Run a disaster recovery exercise

Everything in this course comes together here. A restore drill tests one dataset. A disaster recovery exercise tests the whole story. It starts as if the original machine is gone. Not “unavailable for a bit”. Gone, along with every convenient shortcut it held: shell history, SSH keys, that one script only it had.

The discipline that makes it real: use a clean disposable system and the same boundaries a real incident would impose. A fresh VM or a spare machine, your written runbook, your protected credentials. Nothing from memory. Every time you catch yourself typing something the runbook doesn’t say, you found a gap. The exercise is working.

Follow the runbook in order

These are the steps, and the order matters:

1. establish clean recovery host
2. install verified tools
3. obtain protected credentials
4. select known-good recovery point
5. restore data and configuration
6. verify application behavior
7. record time and gaps

Each step fails in its own way. Step 2 catches undocumented tool versions. Step 3 is where circular credential dependencies surface. Step 4 forces you to choose a snapshot and justify it, instead of grabbing latest by reflex. Step 6 is the honest finish line. Files on disk mean nothing until the application starts and behaves correctly against them.

Keep a timer running and a notes file open. Write a timestamp per step, every surprise, every improvisation:

14:02  host ready
14:19  restic installed, version pinned in runbook? NO - gap
14:31  credentials located (map was accurate)
15:58  restore complete, app starts, orders count matches

Score it against your targets

Compare the measured recovery point and time with your RPO and RTO. If the exercise took four hours against a one-hour RTO, you now know the target is fiction. You learned it cheaply and calmly. Either improve the process or renegotiate the target.

Then turn every manual guess into an updated instruction or a piece of automation. Schedule the next exercise too. Runbooks rot as systems change.

Stay isolated

Keep the exercise away from production names, credentials, and writable storage until the restored system is verified. A recovered application with stale config can email real customers or write to the real database.

The exercise should prove recovery works. Quietly. If you did it right, nobody outside the room noticed anything happened.