Add useful observability

Observability should help answer a concrete operational question, not collect every possible value. The test for any new log line, metric, or alert is: which question, during which future incident, does this answer? If you can’t name the question, you’re collecting noise you’ll pay to store.

The best source of questions is your last incident. Every time you thought “I wish I knew what the memory looked like before the restart” or “which request started this?”, that’s a missing signal.

Make logs answerable

Add request identifiers and structured errors where they shorten diagnosis. The difference in practice:

error: query failed

versus:

level=error msg="query failed" request_id=req_8f3a21 route=/api/orders user_id=1842 db_wait_ms=2980

The first line tells you something broke. The second tells you which request, on which route, and that the database took 2.9 seconds — the diagnosis is half done. With systemd, structured fields also make the journal searchable:

journalctl -u app.service -o json --since "-1 hour" | grep req_8f3a21

Measure the service, not just the host

CPU and memory graphs describe the machine. Users experience request rate, error rate, and latency — measure those per service, plus dependency timing (how long your calls to the database or upstream API take). When latency spikes, one graph comparison tells you whether the slowness is yours or inherited.

A health check is the cheapest signal of all, as long as it checks something real:

curl -fsS https://app.example.com/health
# {"status":"ok","db":"ok","queue_depth":12}

A health endpoint that returns ok without touching its dependencies verifies only that the process can print ok.

Alerts that get acted on

Alerts need an owner, urgency, and action. “Disk 80% on web1” paging nobody in particular at 3am, with no hint of what to do, trains people to ignore it. A useful alert names who gets it, why it can’t wait, and the first command to run. If an alert fires repeatedly and the response is always “ignore it”, delete it or fix the threshold — every tolerated false alarm buries the real one.

Two things to keep out: secrets and unbounded labels. Passwords and tokens in logs become an incident of their own, and a metric labeled by user ID or request ID explodes into millions of series that cost real money.

The trap is adding observability during the incident and trusting it immediately. New instrumentation has its own bugs. Take one difficult diagnostic question from this course and add the smallest signal that would answer it — before the next incident, while you can verify the signal is telling the truth.