Test one hypothesis at a time

A hypothesis should predict an observable result. If it cannot be tested, it does not yet guide the investigation. “Maybe it’s the network” predicts nothing. “The app can’t reach the database, so a TCP connect to port 5432 from this host will fail” predicts something you can check in five seconds.

Make the prediction, then run the smallest test

Say the symptom is API requests timing out. Hypothesis one: the database is unreachable from the app host. The prediction: a plain TCP connection will hang or be refused.

timeout 3 bash -c 'cat < /dev/null > /dev/tcp/10.0.2.15/5432' && echo open || echo closed-or-filtered

open

The connection opens. Hypothesis rejected, in one read-only command, without touching the app. That’s the shape you want: cheap, reversible, and capable of disproving the idea.

Hypothesis two: the database accepts connections but answers slowly. Prediction: a trivial query will take much longer than usual.

time psql -h 10.0.2.15 -U app -d shop -c 'SELECT 1;'
# real    0m2.984s

Three seconds for SELECT 1 is a real finding. Now you know where to dig, and you got there by rejecting one explanation at a time.

Rank before you test

You’ll usually have more hypotheses than time. Rank explanations by evidence, impact, and test cost. Test the ones that are cheap to check and would explain the most, even if they feel less likely. A ten-second df -h is worth running before a two-hour packet capture, whatever your gut says.

Prefer reversible diagnostics first. Reading logs, timing queries, and probing ports change nothing. Restarting services and editing configs change the system, and belong later.

When a test does require a change, change one variable, record the result, and return to the previous state when the test fails. If you bump a timeout, disable a cron job, and restart the service together, and things improve, you’ve learned almost nothing and you can’t cleanly undo it.

That’s the trap: the shotgun fix. Three changes at once that “solve” the incident tonight and leave you unable to say which one mattered, which one was harmless, and which one is a new bug waiting for traffic. Take one symptom from your own system, write three hypotheses, and for each define one read-only test and the result that would make you reject it.