Method and evidence

Define the symptom and scope

Turn a vague report into an observable failure with affected users, operations, hosts, and time boundaries.

8 minute lesson

~~~

The symptom is where every investigation starts, and most people skip it. Someone says “the server is slow” and you start typing commands. Ten minutes later you’re deep in top output with no idea what you’re looking for.

Start by writing what fails, for whom, from where, and since when. One sentence, testable, with numbers in it.

Separate unavailable, slow, incorrect, and intermittent behavior. Each one points at different tools. “Slow” needs timing. “Unavailable” needs a connection test. “Incorrect” needs a request and its wrong response. “Intermittent” needs a counter, because you’ll have to catch it in the act.

Turn the report into a measurement

“The server is slow” becomes measurable with one command:

curl -sS -o /dev/null -w 'status=%{http_code} total=%{time_total}s\n' https://app.example.com/api/orders
status=200 total=4.812s

Now you have a fact. This endpoint answers with 200 but takes 4.8 seconds. Run it a few times, and run it against a healthy path too:

curl -sS -o /dev/null -w 'status=%{http_code} total=%{time_total}s\n' https://app.example.com/health
# status=200 total=0.031s

Comparing one affected path with one healthy path narrows the scope immediately. Here the host, TLS, and web server are fine. Something specific to /api/orders is slow.

Write the statement

Record exact errors and request identifiers while they’re fresh. Then write the symptom in this shape:

Operation: GET /api/orders from the office network
Expected: 200 in under 300ms
Measured: 200 in 4.8s, every request since 09:42 UTC
Scope: all users; /health unaffected
First seen: 09:42 UTC (first Slack report 09:55)

Everything after this point tests that statement. If a command you’re about to run can’t confirm or narrow it, don’t run it yet.

One trap to avoid: taking the report’s timestamp as the start time. Users report incidents late. Check your logs or metrics for when the numbers actually changed, because “what changed at 09:42” is the question that usually solves the case, and asking it about 09:55 sends you to the wrong deploy.

Lesson completed

Take this course offline

Get every free book and course as PDF and EPUB files.

Get the download library →