Build a change timeline

Systems that worked yesterday and fail today were changed — by a deploy, a config edit, a certificate expiry, a dependency, or the traffic itself. Time correlation narrows the search without proving causation. Build the timeline from recorded events rather than memory, because memory reorders things to fit whatever story you already believe.

Assemble the timeline

Pull timestamps from systems that record them: deploy logs, monitoring dashboards, cron schedules, certificate validity dates. Create a UTC incident timeline:

10:02 deploy abc123 completed
10:04 error rate rose from 0.1% to 18%
10:05 database connections reached limit
10:07 rollback started
10:09 error rate returned to baseline

Add evidence links and the observer for each entry. “Error rate rose (dashboard link, seen by Ana)” is verifiable; “the site got slow around ten” is not.

Two minutes between deploy and error spike is suggestive. The rollback restoring baseline strengthens it: the change tracks the symptom in both directions. Still correlation — but correlation strong enough to direct where you look next, inside the diff of abc123.

Test whether the suspected change explains all affected and unaffected paths. If the deploy only touched image processing but checkout also degraded, either the connection limit links them, or your suspect is wrong. A cause that explains half the symptoms is usually half the story.

Bisect when the timeline is long

When the symptom appeared “sometime in the last two weeks” and fifty commits are candidates, stop reading diffs and bisect. git bisect runs a binary search over the commit history:

git bisect start
git bisect bad                 # the current commit fails
git bisect good v2.4.0         # this release was fine
# git checks out the midpoint; run your reproduction, then answer:
git bisect good   # or: git bisect bad

Each answer halves the range, so fifty commits take about six tests. At the end git names the exact commit that introduced the failure, collapsing your timeline to a single entry. The reproduction card from earlier in this course is what makes each test cheap and trustworthy.

Keep clocks synchronized and use one timezone in incident notes. A timeline mixing local time, UTC, and an unsynced server clock will convince you of an order of events that never happened.