Updates, backups, and restore

Updates reduce known risk. Backups reduce the damage when an update, deployment, operator, disk, or provider event destroys useful state. Neither one replaces the other.

Patch deliberately

Ubuntu Server normally applies security updates through unattended-upgrades, but you still need to inspect its result and plan service restarts:

sudo apt update
apt list --upgradable
sudo apt upgrade
test -f /run/reboot-required && cat /run/reboot-required.pkgs
sudo systemctl --failed
sudo tail -n 50 /var/log/unattended-upgrades/unattended-upgrades.log

Run updates during a window when you can verify the application. Keep a working SSH session open. Afterward, test the public health endpoint and the main user workflow. If a reboot is required, announce it, verify backups first, reboot, and repeat the checks.

Back up state, not just the virtual machine

Inventory everything needed to recover:

• application uploads and persistent files
• database data using a database-aware backup
• /etc/notes-app.env, systemd units, and Nginx configuration
• the deployed commit and build instructions
• DNS, firewall, monitoring, and external-service settings

DigitalOcean backups and snapshots are useful recovery layers. They are not the only copy for critical data. Keep another protected copy in a separate failure domain with deliberate retention and encryption.

Check that backup files exist, are recent, and have non-zero size, but do not stop there. Create a disposable Droplet or isolated restore location. Restore the application data and database, supply test credentials, start the services, and run integrity and application checks.

The strongest evidence is a successful restore, not a green backup job. Common failures include omitting database consistency, keeping the only backup beside the source server, or discovering that the restore instructions require a secret nobody retained.

If an update breaks the application, first restore the last working release and configuration. Use a full-server restore only when the system state itself cannot be repaired safely. Database rollback needs a tested database recovery plan; code rollback does not undo schema changes automatically.

Your action is to choose one disposable file and one database record, back them up, restore them somewhere isolated, and record the commands and elapsed time. That becomes the first version of the recovery runbook.