Automate and alert on backups

A backup that depends on you remembering will eventually stop. You’ll be busy, then on vacation, then it’s been four months. Automation fixes that. It also creates a new failure mode: jobs that break silently and keep “running” as no-ops for months. So automation must make silence and repeated failure visible.

Wrap the backup with an honest exit

Write a wrapper that logs the result and exits with a meaningful status:

if restic backup /srv/data; then
  printf '%s backup=success\n' "$(date -u +%FT%TZ)"
else
  status=$?
  printf '%s backup=failure status=%s\n' "$(date -u +%FT%TZ)" "$status" >&2
  exit "$status"
fi

The script logs a timestamped line and exits non-zero on failure. That last part is the important one. Schedulers and alerting can only react to what the script reports.

Schedule it with cron and capture the output:

# /etc/cron.d/backup
0 3 * * * root /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1

Cron is where silent failures breed. It runs with a minimal environment: a stripped PATH, no RESTIC_REPOSITORY, no RESTIC_PASSWORD_FILE. A script that works perfectly in your shell fails at 3 AM because the variables only existed in your session.

Export everything the job needs inside the script itself. Then test from a bare environment with env -i /usr/local/bin/backup.sh.

Test the failure path

Now break it on purpose. Make the repository unreachable, or revoke read access to /srv/data. Confirm the scheduler records a non-zero status and the alert reaches the responsible person.

An alert channel nobody checks is decoration. If you’ve never seen your backup alert fire, you don’t have alerting. You have optimism.

Alert on silence, not just errors

There’s a failure the error path can’t catch: the job never ran at all. Disabled cron, powered-off machine, deleted crontab. No run means no error, and no error means no alert.

You need to detect “job did not run” too. The dead-man’s-switch pattern works well. The wrapper pings a monitoring URL after each success, and the monitor alerts when the pings stop arriving:

restic backup /srv/data && curl -fsS https://hc-ping.com/6c2ea1d4-backup

Alternatively, run a daily check that the newest snapshot is younger than 24 hours. restic snapshots --json plus a timestamp comparison catches the same silence from the repository side.