Respond and rebuild
Recognize a possible compromise
Treat unexpected accounts, keys, listeners, processes, outbound traffic, files, and privilege events as evidence to investigate, not proof to ignore or panic over.
One strange process does not prove compromise. It does justify preserving evidence and checking what changed.
The signals worth stopping for: an account you did not create, an authorized key you cannot place, a listener on an unexpected port, a process with a slightly misspelled system name, outbound traffic to an address nothing should call, or a sudo event at an hour nobody was working.
Record before you react
Record time, symptoms, recent deployments, access events, network connections, process trees, and relevant provider activity:
date -u
ps auxf > /tmp/evidence-ps.txt
sudo ss -tnp > /tmp/evidence-net.txt
last -a | head -20
sudo stat /home/dana/.ssh/authorized_keys
That stat matters more than it looks. The modification time on a key file tells you when access changed, which anchors the whole timeline you are about to build. Copy these captures to another machine right away.
Avoid installing many new tools on the suspect host. Every package install rewrites parts of the filesystem you are trying to read, and a compromised host may hand you tampered tools anyway.
Trust independent sources first
Collect evidence from independent systems first when practical. Provider audit logs and remote authentication records remain useful even if timestamps or files on the host were changed. Compare against known-good configuration and independent logs — the baseline file and the forwarded logs you set up earlier are the reference this moment was built for.
Do not destroy context
An unknown SSH key may come from automation, a former administrator, or an attacker. Deleting it immediately reduces access but can destroy context before you identify its origin and use. Capture first, then act.
The classic mistake is rebooting “to see if it clears up”. A reboot wipes memory, running processes, and active connections — often the only volatile evidence — while leaving any persistence exactly where it was.
Build a timeline from key metadata, authentication logs, provider events, processes, and network connections. Test the checklist with a harmless planted key and prove it preserves evidence before containment begins.
Lesson completed