Plan rotation and recovery

Certificates expire, and private keys leak. Both are certainties, not risks. One has a date printed inside it. The other has a date nobody knows yet. Operations need renewal before expiry and an emergency replacement path, and neither can depend on the one person who remembers where the files live.

Start with knowing what you have. Let’s build a small inventory from the certificate files in the lab folder:

for file in *.crt; do
  echo "$file"
  openssl x509 -in "$file" -noout -subject -issuer -enddate -fingerprint -sha256
done

For each certificate you get the identity, the signer, the deadline, and a SHA-256 fingerprint that pins exactly which certificate is deployed where:

app.crt
subject=CN=app.lab.test
issuer=CN=Practical TLS Lab Intermediate CA
notAfter=Sep 22 16:46:04 2026 GMT
sha256 Fingerprint=45:CD:0E:0D:66:A8:09:87:86:8E:20:0D:64:E3:E9:3F:C9:04:75:D8:47:3E:B9:61:62:19:22:F0:C4:16:AF:5E

The certificate can’t tell you the rest. Who owns it, which service uses it, where it’s deployed, how it gets renewed, who to call when it breaks. Add those columns yourself, in a tracked file or a wiki page. Memory doesn’t count. When notAfter is 30 days out, that row is what turns an alert into an action.

Renewal should be boring. Public-facing certificates belong on ACME automation: a Let’s Encrypt client renews and deploys with no human involved, and your monitoring only confirms it keeps happening. For an internal CA like our lab, the renewal column names the exact commands and who runs them. Alert on remaining lifetime, not on expiry day. An alert with two weeks left is a task. An alert at midnight is an outage.

Then there’s the emergency path: the private key leaks. Scheduled renewal doesn’t cover this. You need a replacement now. The drill is always the same: issue a new key and certificate, deploy them, then revoke the old certificate so the CA stops vouching for it. The order matters. Revoke first and you take the service down yourself.

Rehearse it in the lab. Issue a second leaf from your lab CA, swap the files, restart the server, and confirm the new certificate is live:

openssl s_client -connect 127.0.0.1:8443 -servername app.lab.test </dev/null 2>/dev/null | openssl x509 -noout -fingerprint -sha256

Compare the fingerprint with the one in your inventory. If they differ, the rotation happened. If the drill takes an afternoon of guessing in the lab, it will fail at 2am in production.

One policy note: never carry an old key into a new certificate unless your rotation policy allows key reuse on purpose. A rotation that keeps the compromised key has rotated nothing.