Finish the operator runbook

Everything in this course lives in your head and your shell history. That is fine until someone else has to get in, or you do, from a different machine at 2am.

A runbook fixes that. It is what another authorized person follows to restore safe access without you, and without ever receiving a secret from you.

What goes in

One section per thing you did in this course, each with the command, its expected output, and the way back:

• Connect: the ~/.ssh/config block and its ssh -G output
• Host identity: the host key fingerprint and where to re-check it
• Accounts: who has one, their fingerprint, their sudoers.d file
• Transfer: staging name, then checksum
• Tunnels: the exact -L line for the database
• Hardening: the drop-in, the sshd -T lines, the firewall rules
• Automation: the CI key fingerprint, its authorized_keys line, where the private key lives
• Rotation: add, prove, remove, prove
• Diagnosis: the six failures from the logs lesson
• Break-glass: how to reach the console and what to do there

What stays out

No private keys, passphrases, or CI tokens. The runbook names fingerprints. A fingerprint identifies a key without being one:

Operator key (flavio): SHA256:9kLm3nQ7rT2vW5xY8zA1bC4dE6fG0hJ3kM7nP2qR5sU
CI deploy key:         SHA256:zX9cV2bN5mM8qW1eR4tY7uI0oP3aS6dF9gH2jK5lZ8x
Host key (ed25519):    SHA256:wQ2vT9xk8mR0lHq3sJ7bN1cP4fD6eA8gK5iM2oU0yXs

If the document leaks, an attacker learns which doors exist and gets no key to any of them. That is what makes it safe to commit to the team repo, where people will actually read it.

The happy path is not enough

The runbook people usually write is one line: ssh notes-server. The first real problem makes it useless. Every section needs three things beyond the command: the output that proves it worked, what to do when it does not, and how to undo it.

For hardening, the rollback line sits next to the change. For rotation, the Permission denied check for the old key is the written success condition. For the CI key, the revoking sed line is ready to paste.

Test it the only way that counts

Destroy the disposable server and create a new one at the same address. Hand the runbook to someone else, or to yourself on a clean machine, and ask them to restore safe access.

They should verify the new host key from the console, create the accounts, install the listed public keys, apply the hardening drop-in with the safe sequence, and confirm a second login works. Every question they ask you is a hole in the document. Fix the hole, not the person.

When the rebuilt server accepts your key, refuses passwords and root, and the CI job deploys again, you are done. You own a procedure now, and the server is just where it runs.