Mutual TLS and operations
Plan rotation and recovery
Track owners, expiry, renewal, deployment, rollback, revocation, and private-key compromise.
10 minute lesson
Certificates expire and private keys can be exposed. Both are certainties, not risks — one has a printed date, the other has an unknown one. Operations need renewal before expiry and an emergency replacement path, and neither can depend on the one person who remembers where the files live.
Start with knowing what you have. Create a small inventory from certificate files:
for file in *.crt; do
echo "$file"
openssl x509 -in "$file" -noout -subject -issuer -enddate -fingerprint -sha256
done
For each certificate you get the identity, the signer, the deadline, and a SHA-256 fingerprint that pins exactly which certificate is deployed where:
app.crt
subject=CN = app.lab.test
issuer=CN = Practical TLS Lab CA
notAfter=Aug 17 10:45:00 2026 GMT
SHA2-256 Fingerprint=7A:1B:0C:...
The certificate cannot tell you the rest. Add owner, service, deployment location, renewal method, and recovery contact outside the certificate itself — a tracked file or a wiki page beats memory. When notAfter is 30 days out, this row is what turns an alert into an action.
Renewal should be boring. Public-facing certificates belong on ACME automation — a Let’s Encrypt client renews and deploys with no human in the loop, and your monitoring only confirms it keeps happening. For an internal CA like our lab, the renewal method column names the commands and who runs them. Alert on remaining lifetime, not on expiry day; an alert that fires with two weeks left is a task, one that fires at midnight is an outage.
Then there is the emergency path: the private key leaks. Renewal on a schedule does not cover this — you need replacement now. The drill is always the same: issue a new key and certificate, deploy, then revoke the old certificate so the CA stops vouching for it. The order matters; revoking first takes the service down.
Rehearse replacing the lab certificate without disabling HTTPS: issue a second leaf from your lab CA, swap the files, restart, and confirm with openssl s_client that the new fingerprint is being served. If that drill takes an afternoon of guessing in the lab, it will fail at 2am in production.
One policy note for the inventory: never copy old keys into a new certificate unless the rotation policy deliberately permits key reuse. A rotation that keeps the compromised key has rotated nothing.
Lesson completed