Updates, backups, and restore

Updates and backups solve different problems. Updates lower the chance that a known vulnerability hurts you. Backups limit the damage when something does: a bad deploy, a typo with rm, a dead disk. Neither covers for the other.

Patch on purpose

Ubuntu Server installs security updates on its own through unattended-upgrades. Good, but someone still has to check what happened and restart services that need it. Once a week, run:

sudo apt update
apt list --upgradable
sudo apt upgrade
test -f /run/reboot-required && cat /run/reboot-required.pkgs
sudo systemctl --failed
sudo tail -n 50 /var/log/unattended-upgrades/unattended-upgrades.log

The test line prints the packages that need a reboot, or nothing. systemctl --failed should print 0 loaded units listed. The log shows what the updater did while you weren’t looking.

Do this when you have time to check the app afterward. When it’s done, hit the public health endpoint and click through the main user flow. If a reboot is required, check your backups first, reboot, and repeat the checks.

Back up state, not a virtual machine

The instinct is “I have DigitalOcean backups turned on, I’m fine”. Backups and snapshots are a useful layer, not the only copy your data should have. They live in the same account, one leaked password away from everything else.

What you’d need to bring the app back on a blank server:

• application uploads and other persistent files
• the database, dumped with pg_dump or sqlite3 .backup, not a file copy of a live database
• /etc/notes-app.env, the systemd unit and the Nginx server block
• the deployed commit hash and the build steps
• DNS, firewall, monitoring and third-party service settings

Everything on that list needs a second encrypted copy somewhere else, with a retention rule. The Backup and Restore course shows how with restic.

A backup you haven’t restored is a guess

A green backup job tells you a file was written, not that it’s usable. Check that backups exist, are recent and aren’t zero bytes. Then create a throwaway Droplet, restore the data and the database into it, start the services and click around.

The failures you’ll find are always the same. The database dump is inconsistent. The only backup sat on the server that died. The restore steps need a secret nobody kept. Better to learn that now than at 3 a.m.

What to restore when

If an update breaks the app, start small: roll back to the last working release and configuration. Restore the whole server only when the system itself can’t be repaired. And a code rollback doesn’t undo a schema migration. The database needs its own tested recovery plan.

Pick one file and one database record. Back them up, restore them somewhere isolated, and write down every command and how long it took. That’s version one of your recovery runbook.