Build a timeline

Most incidents become easier when events are placed in order. “The service crashed” is a mystery. “The deploy finished at 14:02, memory climbed from 14:05, the OOM killer fired at 14:31” is almost a conclusion.

Confirm the clock first

A timeline built on a drifting clock is worse than no timeline. Confirm the server clock and timezone before you trust a single timestamp:

timedatectl

               Local time: Mon 2026-08-03 14:40:12 UTC
           Universal time: Mon 2026-08-03 14:40:12 UTC
                 Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
              NTP service: active

The two lines that matter: System clock synchronized: yes and the time zone. If synchronization is no, note it in the timeline itself, because every local timestamp is now suspect.

Pull events with absolute timestamps

Use absolute timestamps everywhere. “5 minutes ago” is useless in a document you’ll read tomorrow.

journalctl -u app.service --since "2026-08-03 13:50" --until "2026-08-03 14:35" -o short-iso --no-pager

2026-08-03T14:02:11+0000 web1 systemd[1]: Started app.service.
2026-08-03T14:31:04+0000 web1 kernel: Out of memory: Killed process 3721 (node)

The -o short-iso flag gives you sortable timestamps you can paste next to events from other systems. If the machine rebooted during the incident, add -b -1 to read the previous boot’s journal — the messages right before a crash are usually the interesting ones.

Deployment records come from your CI system, provider events from the cloud status page or maintenance emails, and user reports from wherever they landed. Each source gets its own line in the timeline, with its origin noted, so you can weigh it later.

Now align journal entries, deployment records, provider events, and client reports in one list:

13:58 UTC  deploy pipeline started (CI log)          — evidence
14:02 UTC  app.service restarted (journal)           — evidence
14:05 UTC  RSS starts climbing (metrics)             — evidence
14:31 UTC  OOM kill of node PID 3721 (kernel log)    — evidence
14:33 UTC  first user report (Slack)                 — evidence
           new release leaks memory                  — inference, needs verification

Mark which line is evidence and which is an inference. Correlation suggests a hypothesis but does not prove causation: the deploy is the obvious suspect, but a traffic spike at 14:00 would fit the same timeline.

The trap is mixing time zones. CI logs in local time, journals in UTC, a teammate quoting their own wall clock. One event lands an hour off, and the timeline “proves” the crash happened before its cause. Convert everything to UTC as you write each line down.