Automate retention and alerts

A timer that exits successfully does not prove useful data reached durable storage.

I’ve seen backup jobs “succeed” for months while the destination volume was unmounted, so every run wrote into an empty directory on the root disk. Exit code 0 the whole time. Automation without evidence is just scheduled false confidence.

A backup job worth trusting

A production backup script needs locking, a timeout, and evidence. Locking so overlapping runs can’t corrupt each other, a timeout so a hung run can’t block tomorrow’s, and logged evidence so you can audit what actually happened:

#!/usr/bin/env bash
set -euo pipefail
exec 9>/run/lock/backup-data.lock
flock -n 9 || { echo "previous run still active"; exit 1; }

start=$(date +%s)
snapshot=$(restic backup /srv/data --json | jq -r 'select(.message_type=="summary") | .snapshot_id')
restic check --read-data-subset=5%
echo "$(date -Is) snapshot=$snapshot bytes=$(du -sb /srv/data | cut -f1) duration=$(( $(date +%s) - start ))s" \
  >> /var/log/backup-data.log

Log the source, destination, bytes, duration, and snapshot identifier. When something looks wrong later, this log answers “when did backups quietly shrink to 2 MB?”. Run it from a systemd service with TimeoutStartSec=2h and a daily timer, rather than a bare cron line — you get journal logging and timeout handling for free.

Retention is a policy, not a disk-full reaction

Apply a documented retention policy. Deleting old backups because the disk filled means deleting them at the moment you least control. Decide the policy up front and let the tool enforce it:

restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

Recent history dense, older history sparse. Write down why those numbers, so nobody “cleans up” the monthlies during an incident.

Alert on silence, not just failure

Failure alerts miss the worst case: the job that never ran. Server off, timer disabled, credential expired — nothing fails, so nothing alerts.

The fix is a dead man’s switch: the job pings a monitor on success, and the monitor alerts when the ping stops arriving. Alert on failed runs, missing recent backups, unexpected size changes, and repository integrity failures. The size check is the sneaky one — a backup that suddenly drops from 18 GB to 40 KB “succeeded”, and it’s telling you the source path is wrong.

Create a daily backup service and timer design. Include locking, timeout, retention, success evidence, and an alert that detects a silent missed run.