Verify recovery from the outside

A green process status proves only that a process is running. systemctl status says active (running) while the app returns 500 on every request — a process can be alive and broken at the same time. Declaring victory from the inside is how incidents get “resolved” twice in one evening.

Repeat the operation that failed

Repeat the exact failed operation from the relevant client path. Not a simpler one, not the health endpoint — the request users were making when things broke, from where they make it:

curl -sS -w '\nstatus=%{http_code} total=%{time_total}s\n' https://app.example.com/api/orders

[{"id":18234,"status":"shipped"},...]
status=200 total=0.214s

Both numbers matter. A 200 in 4 seconds means the outage became a slowdown, and users still feel it. If the incident affected external users, run the check from outside your network — an internal test can’t see a broken load balancer, CDN, or DNS record.

Check what a request can’t show

One good response is a start, not proof. Check logs, latency, errors, queues, and data:

journalctl -u app.service --since "-5 min" --no-pager | grep -ci error
# 0

Then look at whatever accumulated during the outage. A message queue that grew to 40,000 jobs will keep the system degraded for an hour after the “fix”. Writes that failed silently leave data gaps that no status page reveals — spot-check records created during the incident window.

Watch past the trigger

Watch long enough to cover the trigger that caused the incident. If the crash came from the nightly backup, a service that survives ten quiet minutes at 3pm proves nothing; recovery is confirmed after tonight’s backup runs clean. If load triggered it, watch through the next peak:

watch -n 30 'curl -sS -o /dev/null -w "%{http_code} %{time_total}s\n" https://app.example.com/api/orders'

The trap: verifying with a different, easier operation. The health check hits /health, which does almost nothing, so it recovered instantly — while /api/orders, which touches the database that’s still rebuilding, stays broken. You close the incident, users reopen it. Always verify with the operation from the original symptom definition.

Write a recovery checklist for one service you run. Include an external request, one data check, one dependency check, and a monitoring window tied to the trigger.