Automation, rotation, and recovery
Build narrow automation
Give deployment jobs a dedicated identity, known host, restricted command, and minimal server permissions.
When a person runs ssh, there is a human checking the fingerprint, reading the error, and deciding what to do. A deploy job in CI has none of that. It runs at 3am with whatever key and whatever settings you gave it, and it will happily do the wrong thing forever.
So automation gets a narrower identity than any person. It can do one thing, on one host, and every run leaves a trace.
A key that belongs to the job
Never reuse your own key in CI. Generate one for the job, with no passphrase because nobody is there to type it:
ssh-keygen -t ed25519 -f ./ci_deploy_ed25519 -N '' -C 'ci deploy notes-server'
The private file goes into the CI system’s secret store. The .pub line goes on the server, and only there. If this key leaks, you revoke one line in one file and no person loses access.
A key that can do one thing
Install it on the server with the restrictions from module two:
command="/usr/local/bin/deploy-notes",restrict,from="198.51.100.20" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGx8... ci deploy notes-server
command= ignores whatever the client asks for and runs the release script. restrict removes forwarding and terminals. from= accepts the key only from the CI runner’s address. Three options, and the key went from “a shell as deploy” to “run this script from that machine”.
The script itself should log every run:
#!/bin/sh
set -eu
logger -t deploy-notes "run by ${SSH_ORIGINAL_COMMAND:-<none>} from ${SSH_CONNECTION%% *}"
tar -xzf /var/www/notes/release.tar.gz -C /var/www/notes/current
systemctl --user restart notes-api
SSH_ORIGINAL_COMMAND holds what the client actually asked for, so you can see attempts to run something else.
Pin the host in the job
CI tutorials love StrictHostKeyChecking=no. That line throws away the whole first module. Instead, commit the server’s public host key to the repo and point SSH at it:
ssh -o UserKnownHostsFile=./known_hosts \
-o StrictHostKeyChecking=yes \
-o BatchMode=yes \
-i ./ci_deploy_ed25519 [email protected]
BatchMode=yes makes SSH fail instead of hanging on any prompt. If the host key ever changes, the job fails loudly, which is what you want a machine to do.
Prove the boundary
Now try to abuse the key from your laptop. The allowed action:
ssh -i ./ci_deploy_ed25519 [email protected]
The release script runs and the connection closes. On the server, journalctl -t deploy-notes shows the run. Now ask for something else:
ssh -i ./ci_deploy_ed25519 [email protected] 'cat /etc/shadow'
The release script runs again. The requested command was ignored, and the log line shows run by cat /etc/shadow. A tunnel:
ssh -N -L 5433:127.0.0.1:5432 -i ./ci_deploy_ed25519 [email protected]
The forced command still runs, and any connection through the tunnel gets administratively prohibited. Three attempts, one behavior. That is what narrow means.
Minimal permissions on the server
The deploy account should own /var/www/notes and nothing else. If the script needs one privileged command, give it that one line in sudoers.d, exactly as we did for people. A deploy key that can sudo anything is a root key with extra steps.
Lesson completed