Preserve state before changing it

A restart can restore service and erase the state that explained the failure. The stuck process, its open files, the socket backlog, the memory numbers: all gone the moment you type systemctl restart.

So before you change anything, spend two minutes capturing what’s there. You’re allowed to restart afterwards. You’re just not allowed to restart first.

The capture

Save everything into a timestamped directory so it doesn’t get mixed up with the next incident:

d=/var/tmp/incident-$(date -u +%Y%m%dT%H%M%S)
mkdir "$d" && cd "$d"

date -u > time.txt
uptime > uptime.txt
systemctl --failed > failed-units.txt
ps auxww --sort=-%cpu > processes.txt
free -h > memory.txt
df -h > disk.txt
ss -tnp state established > sockets.txt
journalctl -u app.service --since "-30 min" --no-pager > app-log.txt
dmesg --ctime | tail -100 > kernel.txt

Each file answers one later question. processes.txt tells you what was running and in which state. sockets.txt tells you who was connected. failed-units.txt tells you whether the problem was wider than one service.

Record time, uptime, recent changes, failed units, processes, load, memory, storage, sockets, and focused logs. That list covers almost every “I wish I knew what it looked like” moment I’ve had.

Verify the capture before acting

Check the directory actually has content:

wc -l *.txt
# 42 processes.txt
# 118 sockets.txt
# ...

An empty file here means a command failed silently, and you want to know now, not during the review.

Avoid giant uncontrolled dumps containing secrets or unrelated user data. A full journalctl export or a core dump of a process that handles passwords is a liability. Capture focused slices, and note which files might contain sensitive values before you share them in a ticket.

The trap: capturing evidence about the wrong thing. If the symptom is network timeouts and you only save CPU and memory output, the capture was theater. Match the capture to the symptom you defined in the previous lesson.

Create a ten-command evidence checklist for your own server. Mark which outputs are volatile (lost on restart) and which might contain sensitive information. When an incident hits, you paste it, not compose it.