Use metrics, logs, and traces together

Production observability gives you three signal types, and each answers a different question. Metrics show scope and timing: how many, how slow, since when. Traces follow selected requests across boundaries: where one request spent its time. Logs explain discrete events inside those components: what exactly happened at that moment.

Debugging goes wrong when you start in the wrong one. Grepping all logs for “error” during an incident returns thousands of lines, most of them normal background failure. The efficient path runs in one direction: metrics to scope it, traces to localize it, logs to explain it.

Follow one path

Follow one path:

metric: checkout latency rose at 10:04
trace: waiting 8 seconds on inventory call
log: inventory pool timeout for request r-42

Each step shrinks the search. The metric says checkout, since 10:04 — not the whole system, not all day. The trace for one slow checkout request shows a waterfall of spans, and one span dominates: eight seconds waiting on the inventory service. Now you know the component and the dependency. Only then open logs, scoped to the inventory service around 10:04, filtered by the request ID the trace handed you:

grep 'r-42' inventory.log
# {"event":"pool_timeout","requestId":"r-42","waitedMs":8000}

One log line, found in seconds, because two other signals aimed you at it.

Compare healthy and failing

Compare a healthy and failing request side by side: pull one trace from before 10:04 and one from after. The difference between the two waterfalls is the incident — 40 ms on inventory before, 8 seconds after. Everything the two traces share is background you can ignore.

Confirm the chosen signal represents user impact rather than background noise. A latency percentile on an endpoint nobody calls, or an error count dominated by one retrying bot, will have you debugging a non-incident while the real one continues.

Protect telemetry like the production data it describes. Traces and logs carry URLs, identifiers, and sometimes payloads — keep secrets and customer data out of them, and restrict access and retention.