Complete a recovery drill

A full drill reveals dependencies that file-level tests miss: DNS, certificates, packages, service units, firewall policy, and external credentials.

You can restore every file perfectly and still have no working service, because the service was never just files. It was also a DNS record, a TLS certificate, a package list, a systemd unit, a firewall rule, and an API key that lived only in someone’s password manager. The only way to find those gaps is to rebuild for real, before the day you have no choice.

Run the drill

Start from a clean host. A fresh VM, not the production server — the drill must prove you can come back from nothing. Restore or rebuild each layer in runbook order, and keep a timeline as you go:

14:02  fresh Ubuntu 24.04 VM provisioned
14:05  restic repo credentials retrieved from password manager
14:11  packages installed from runbook list
14:19  /etc/nginx and TLS certs restored
14:24  BLOCKED: nginx unit needs /var/www/shop -- not in runbook
14:31  postgres roles + dump restored, counts match
14:58  service up, responds locally

Every BLOCKED line is the drill paying for itself. Record every undocumented dependency the moment it bites you.

Verify the application externally — from another machine, the way a user would, not with a local check that skips DNS, TLS, and the firewall:

curl -sS -o /dev/null -w "%{http_code} %{time_total}s\n" --resolve shop.flaviocopes.com:443:203.0.113.40 https://shop.flaviocopes.com/healthz
# 200 0.412s

The --resolve flag points the request at the drill host without touching real DNS, so you can test the full HTTPS path while production stays live.

Turn evidence into improvement

Produce a short report with actual recovery time, latest usable recovery point, missing inputs, and the next improvement owner. Compare the measured numbers against the RPO and RTO you committed to earlier in this course — a 4-hour promise and a 6-hour drill is a finding, not a failure.

Update the runbook while evidence is fresh, the same day. And when a full rebuild is genuinely too expensive this quarter, a tabletop drill — walking the runbook step by step on paper, checking each input exists — still catches missing credentials and stale instructions. Cheaper than a real drill, infinitely better than nothing.

Run or tabletop a complete recovery. Produce a short report with actual recovery time, latest usable recovery point, missing inputs, and the next improvement owner.