Observe and back up
Read authentication and service logs
Use the system journal and authentication records to investigate access, failures, restarts, privilege use, and application behavior.
Logs show what the server observed. Learn the normal records before you need them during an incident.
On Ubuntu, the systemd journal collects everything, and the classic text files still exist under /var/log. Both views are useful.
Query the journal
Use journalctl by unit, time, priority, and boot:
sudo journalctl -u ssh --since today
sudo journalctl -p err -b
sudo journalctl -u nginx --since "14:00" --until "15:00"The first shows today’s SSH activity. The second shows every error-priority message since the last boot — a fast way to spot a crashing service. The third narrows one unit to a suspicious hour.
Know the lines that matter
Authentication events also land in /var/log/auth.log:
sudo grep sshd /var/log/auth.log | tail -3
# Failed password for invalid user admin from 198.51.100.7 port 51022 ssh2
# Failed password for invalid user admin from 198.51.100.7 port 51040 ssh2
# Accepted publickey for dana from 203.0.113.44 port 55910 ssh2: ED25519 SHA256:kq2R...Review SSH and sudo events, service restarts, kernel messages, and application failures. A successful SSH login after many failures matters more than either event alone. Sudo entries in the same file record the user, the working directory, and the exact command that ran.
Get copies off the host
Send important logs off the server so a compromise cannot erase the only copy. Local logs help investigation, but an attacker with root may edit or erase them. A remote syslog target or a log-shipping agent turns the journal into evidence the host cannot rewrite after the fact.
Synchronize clocks and preserve stable host identifiers. Without consistent time and source information, responders cannot reliably connect provider, firewall, SSH, and application events. Check that timedatectl reports System clock synchronized: yes on every machine involved.
Generate a success, failure, sudo command, and service restart with known timestamps. Find each event locally and remotely, then stop log forwarding and prove the missing stream creates an alert.
Lesson completed