Mitigate and roll back safely

During an incident, recovery may matter before complete root cause. Users are failing now; the full diagnosis can take hours. Mitigation is the deliberate middle move: reduce harm first with the smallest reversible action, while keeping the evidence you will need to find the actual cause.

The menu is usually short: a feature flag, traffic shift, capacity limit, or rollback can contain impact. Prefer the option smallest in scope and easiest to undo. Disabling one feature beats rolling back a deploy that carried ten unrelated changes. A rollback beats restarting everything and hoping.

Write the gate before you act

Even under pressure — especially under pressure — write down what you are about to do and how you will know it worked. Write the mitigation gate:

action: disable new image processing
expected impact: uploads queue instead of failing
verification: error rate and queue depth
rollback of mitigation: re-enable after worker fix
evidence to preserve: failed job IDs and deploy logs

This takes ninety seconds and prevents the two classic incident mistakes: an action nobody can undo, and an action nobody verified. The expected impact line is a prediction. If you disable image processing and errors don’t drop, your theory about the cause was wrong — and you learned that without wasting an hour.

Apply it first in the safest available scope — one region, a small traffic percentage, a canary — monitor the predicted signals, and stop if new harm appears. A mitigation that makes things worse and keeps running is a second incident.

Preserve the crime scene

The evidence line is the one people skip. Restarting a process clears its memory state. Rolling back removes the failing code path. Clearing a queue deletes the poisoned messages that would have explained everything. Before each action, capture what it will destroy: failed job IDs, a copy of current logs, the deploy identifier.

Do not delete logs, restart every system, or clear all queues before capturing evidence and understanding recovery consequences. Mass restarts create their own problems too — every service reconnecting at once can take down the database that was healthy.

Mitigation is temporary by definition. The gate’s rollback line — “re-enable after worker fix” — needs an owner, or the workaround quietly becomes permanent and the feature stays off for a year.