Choose mitigation before root cause

The fastest safe mitigation and the complete root cause are often different tasks. When users are down, your first job is stopping the bleeding, not explaining it. Understanding why the new release leaks memory can happen tomorrow; rolling back to the release that didn’t takes five minutes tonight.

The standard mitigations

You have four moves, roughly in order of preference. Rollback: return to the last known-good version or config. Fail over: move traffic to a healthy replica or region. Shed load: rate-limit or turn away some traffic so the rest succeeds. Disable one feature: flag off the code path that’s hurting, keep everything else running.

Pick based on evidence, not reflex. If the timeline says the incident started with the 14:02 deploy, rollback is justified. If nothing was deployed and one host is sick, failover fits better. Rollback, fail over, shed load, or disable one feature when evidence supports it — a mitigation chosen at random is just another uncontrolled change.

Preserve, then mitigate

Record the state first. The mitigation will destroy the evidence, so spend the two minutes from the earlier lesson before you pull the trigger:

d=/var/tmp/incident-$(date -u +%Y%m%dT%H%M%S); mkdir "$d"
ps auxww > "$d/ps.txt"; free -h > "$d/mem.txt"; ss -tnp > "$d/ss.txt"
journalctl -u app.service --since "-20 min" --no-pager > "$d/journal.txt"

Then mitigate, and define the exit check before you act:

systemctl revert app.service && systemctl restart app.service
watch -n 10 'curl -sS -o /dev/null -w "%{http_code} %{time_total}s\n" https://app.example.com/api/orders'

The user-visible check proves impact ended. Internal green lights don’t count; the request that was failing has to succeed. Decide in advance what result means “the mitigation worked” and what result triggers the next option, so you’re not inventing criteria under pressure.

The trap: the risky workaround. Under pressure, “just open the port to everyone”, “disable auth temporarily”, or “chmod 777 the directory” all feel like mitigations. They’re not — they trade an availability incident for a security or data-integrity incident, which is a much worse trade. Avoid a risky workaround that creates a larger security or data-integrity problem; a legitimate mitigation reduces total risk, and if your idea only moves the risk somewhere darker, keep looking.

For a simulated bad deployment, write down your mitigation, the rollback trigger, the evidence to preserve, and the user-visible check that proves impact ended.