Notice unexpected change

You cannot recognize unusual server behavior without a picture of normal behavior. Keep the baseline small enough to review.

Capture the baseline

Watch new accounts, authorized keys, sudo rules, enabled units, listeners, packages, cron jobs, application files, CPU, disk, and outbound traffic. Save the stable parts as plain text you can diff later:

{
  getent passwd | awk -F: '$3 >= 1000'
  sudo sha256sum /home/*/.ssh/authorized_keys
  sudo ls /etc/sudoers.d/
  systemctl list-unit-files --state=enabled
  sudo ss -lntu
  ls /etc/cron.d/ /etc/cron.daily/
} > baseline-2026-08-03.txt

Store the file off the server, next to your threat-model notes. A baseline stored on the host can be edited by the same attacker it is supposed to expose.

Compare on a schedule

Weeks later, run the same capture into a new file and diff the two:

diff baseline-2026-08-03.txt baseline-2026-09-01.txt
# > 99-deploy-tmp

One new file in /etc/sudoers.d/ you cannot explain — that single line is exactly the needle this habit exists to find.

A new cron job may be a deployment change or attacker persistence. A reviewed baseline provides the evidence needed to distinguish expected work from drift. Configuration management and deployment manifests make unexpected differences easier to see: when the server is built from a script, the script itself is the baseline.

Keep the noise out

Exclude expected volatile data such as temporary files and counters. A smaller, deliberate baseline makes a changed key or unit visible instead of burying it in harmless differences.

The failure mode is a diff of 400 lines of routine package bumps every week. You stop reading it, and that is how the one real change slips through. Drop version numbers from noisy sections, or split packages into their own report reviewed less often.

Save a small baseline of users, keys, sudo rules, listeners, units, packages, and scheduled jobs. Add one harmless unauthorized change and prove the comparison names it without flooding the report with normal log growth.