Design actionable alerts

An alert nobody investigates is noise. Start with events that could cause real harm.

Examples include a new sudo user, an unexpected listening port, repeated SSH failures followed by success, a disabled firewall, full disk, or changed deployment files. Each of those is rare, meaningful, and has an obvious first response. That combination is what makes an event worth waking someone for.

Define the whole alert, not just the trigger

An alert definition needs more than a threshold. Write down who receives it and what they do first:

signal:    new file in /etc/sudoers.d/
threshold: any occurrence
owner:     flavio
evidence:  baseline diff + auth.log for the same hour
first act: match against the deployment log; if absent, treat as incident

Set thresholds from normal traffic and test notification delivery. Alerting on every failed SSH login creates noise. Alerting when failures are followed by success highlights a plausible account attack but can still catch a forgetful administrator.

Deliver through an independent path

Send alerts through a path independent from the host when possible. A full disk or compromised server may break the same local mail command that was supposed to report it. An external uptime monitor, a provider metric alarm, or a push service running elsewhere keeps talking when the host cannot speak for itself.

Fire it on purpose

An alert you have never triggered is a hope, not a control. Create the sudoers file in the lab, watch the notification arrive, then check the quiet case too:

sudo journalctl -u ssh --since -1h | grep -cE 'Failed password'
# 3   <- below threshold, no alert expected

The long-term failure mode is fatigue. When a channel fills with ignorable messages, the one that matters gets dismissed with the rest. My rule: an alert that fired three times without causing any action gets tuned or deleted.

Define signal, threshold, owner, evidence link, and first action for three alerts. Trigger one real alert and one near-threshold case, then prove notification delivery and suppression both behave as documented.