Automation, rotation, and recovery

Respond to a lost key or host change

Distinguish a compromised user identity from a changed server identity and respond to the right trust boundary.

Two very different things go wrong with SSH trust, and they need opposite responses. Your private key may be in someone else’s hands. Or the server you are talking to may not be the server you think. Module one taught you these are separate questions. This is where that pays off.

Confuse them and you fix the wrong side. You rotate keys while an attacker still owns the server, or rebuild a server while an attacker still holds your laptop’s key.

Case one: a stolen laptop

The private key on that laptop is now compromised. The passphrase buys time, not safety. The response is on the user side: revoke that public key everywhere it is authorized.

Start from your inventory and remove the line on every server, using a different trusted key or the console:

ssh -i ~/.ssh/backup_ed25519 [email protected] \
  "sed -i '/flavio notes-server operator$/d' ~/.ssh/authorized_keys"

Then look for use you did not make. The server log records the fingerprint on every login:

sudo journalctl -u ssh --since '7 days ago' | grep 'Accepted publickey'
Accepted publickey for deploy from 198.51.100.7 port 51234 ssh2: ED25519 SHA256:9kLm3nQ7...
Accepted publickey for deploy from 45.83.12.9 port 40102 ssh2: ED25519 SHA256:9kLm3nQ7...

Same fingerprint, two addresses. One of them is not you. Now you know the key was used, when, and from where. Save that log before you change anything else. It is evidence, and cleaning up destroys it.

Revoke the key on GitHub and anywhere else it was authorized too. Then generate a new key on a machine you trust and install it as in module two.

Case two: the host key changed

You connect and get the big warning block:

WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!
Offending ED25519 key in /Users/flavio/.ssh/known_hosts:14

Your key is fine. Nothing about your laptop is in question. The doubt is on the server side, and the response is to stop and investigate, not to type ssh-keygen -R and move on.

Did anyone rebuild this server? Did the provider migrate it? If nobody knows, go in through the console, the channel that does not pass through the network you distrust, and print the real fingerprint:

ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

If it matches what your client saw, the server really did change its key. Update the one known_hosts entry. If it does not match, something between you and the server is answering in its place. Do not log in. Do not send your key to it. Check DNS, check with the provider, and treat any credentials typed since the change as exposed.

Practice both on paper

Run both scenarios as a tabletop with a colleague: the laptop is gone; a warning appears on Monday morning. For each, list the accounts and servers affected, the logs to preserve, the key to revoke or the entry to update, and who to tell. Write the two lists side by side.

They should barely overlap. If they look the same, you have not separated the two trust boundaries yet. Fix that before the real call comes.

Lesson completed