Operate and recover
Write and rehearse the runbook
Document inventory, access, startup, shutdown, backup, restore, update, and hardware-replacement procedures.
10 minute lesson
A home server should remain recoverable when you are tired, away, or replacing failed hardware. Six months from now you won’t remember which UUID goes in fstab or where the Caddy root certificate lives. The runbook holds operational memory outside the machine — written for a future you who remembers nothing.
What goes in it
Create a checklist with no embedded secrets:
inventory
network and DNS
accounts and key locations
service start and stop
backup and restore
update and rollback
power-loss recovery
complete rebuild
Each heading becomes a short section of commands and facts, in the order you’d need them. Inventory is the hardware and disk map from the storage module. Network and DNS records the reserved address, the subnet, and which device answers lab.test. Accounts and key locations says where keys and recovery material live — the password manager entry, the drawer — never the secrets themselves, because the runbook will be copied to places with weaker protection than your secrets deserve.
The entries should be commands, not prose:
## service start and stop
cd /opt/services/whoami
docker compose up -d # start
docker compose down # stop
verify: curl -I https://homeserver.lab.test -> HTTP/2 200
Every step states how to verify it worked. A runbook that says “start the service” without the check line still requires the knowledge it was supposed to replace.
Rehearse it
An unrehearsed runbook is fiction — pleasant to have written, wrong in the details. Follow it to rebuild one disposable service from configuration and backup. Pick the whoami service, remove it completely, and rebuild using only what the document says:
docker compose down --volumes
# now rebuild from the runbook, restoring config from backup
The rule during the drill: if you have to remember something, the runbook failed. Fix every missing assumption you discover — the unstated directory, the firewall rule you forgot existed, the “oh, right, the CA certificate” moment. Those gaps are the entire value of rehearsing.
Keep it reachable
Store a copy away from the server and keep secret recovery material in an appropriate protected location. A runbook stored only on the machine it describes is unreachable during the exact failures it exists for. A printout in a drawer, a copy in your password manager’s notes, a file in your synced documents — any of these works. Re-run one drill whenever the setup changes; the runbook is only current until the next lesson you apply.
Lesson completed