Debug production safely

Finish with a debugging record

Document symptom, scope, evidence, cause, repair, verification, cleanup, and prevention after the system recovers.

10 minute lesson

~~~

The bug is fixed and everything works. The temptation is to close the ticket and move on — which throws away the most valuable output of the whole investigation. A debugging record turns one repair into reusable operational knowledge: the next person hitting a similar symptom searches, finds your record, and skips the first three hours.

Write it while the evidence is fresh. A week later the details blur, and the record becomes fiction.

The template

Use a compact template:

symptom and user impact
start and end time
first failed boundary
evidence
root cause
mitigation and permanent repair
verification
follow-up owner and deadline

Keep it factual: every claim should point at evidence — a log line, a graph, a commit. And distinguish contributing conditions from root cause. “The pool was small” contributed; “the new image worker held a connection open for the length of each job” is the cause. Mixing the two produces fixes that only raise limits, then wait for the same failure at the new limit.

Separating mitigation from permanent repair matters for the same reason. “Rolled back at 10:07” restored service. “Connection now released before processing starts, commit def456” fixed the bug. If the record cannot name the permanent repair, the incident is not actually finished.

Test the record

Have someone unfamiliar with the incident reproduce the reasoning from the evidence. If they read it and ask “but how did you know it was the pool?”, the evidence section has a hole. Remove speculative claims that were disproved along the way — half-remembered wrong hypotheses presented as fact are how folklore starts. “The database is flaky”, forever, in every future incident.

The follow-up owner and deadline line keeps the record honest. Prevention items without an owner — “add pool metrics”, “alert on queue depth” — are wishes, not follow-ups.

Do not use the review to blame individuals. A record that names the engineer who deployed teaches everyone to hide details next time, and the details are the whole point. Improve systems, checks, ownership, and recovery paths — the bad deploy was possible because nothing prevented it, and that is the fixable part.

Lesson completed

Take this course offline

Get every free book and course as PDF and EPUB files.

Get the download library →