Restore and learn

Recovery is not “the server starts again.” We need confidence that the attacker’s access is gone, the original weakness is fixed, and the data we restored is complete. Skip any of those and you restart the incident with extra steps.

Restoring after a security incident

Restoring after a compromise carries a trap that ordinary disaster recovery does not: the backup may contain the compromise. Restore from known-good inputs — a snapshot from before the intrusion, checked against the timeline you preserved during containment.

Then, in order:

• Rotate every credential the attacker could have touched. A restored system that still accepts stolen passwords is not recovered.
• Patch the root cause before going live. Restoring the same vulnerable code invites the same visitor back.
• Monitor for recurrence with specific checks — the indicators from this incident, not generic dashboards.

Trust no backup you have not restored

The nastiest failure hides here. A backup job reports success every night, but the archive misses uploaded files referenced by the database. The restore runs cleanly, the service starts, and every attachment in the product is broken — discovered only now, when it is too late to fix the backup job.

The defense is dull and non-negotiable: test backups before an incident.

pg_restore --dbname=restore_drill backup_2026-08-01.dump

# verify relationships, not just "it started"
psql restore_drill -c "SELECT count(*) FROM attachments a
  LEFT JOIN files f ON f.id = a.file_id WHERE f.id IS NULL;"
# 0 means no attachment points at a missing file

Time the whole drill. Recovery time includes validation, credential rotation, and chasing missing dependencies. Measuring only download time gives the team false confidence about a number someone may one day promise a customer.

The learning half

Afterward, write a blameless timeline: what happened, when it was detected, what each response step cost. Blameless is not politeness — people who fear blame omit the details you need most.

End with concrete improvements, each with an owner and a date:

action: alert on >20 failed logins/hour per account
owner: marta    due: 2026-09-12
action: move build logs to private storage
owner: jonas    due: 2026-08-21

“Improve monitoring” fixes nothing; a named, dated action might. Then close the loop: restore one dated backup into an isolated environment, verify record counts, relationships, files, and one real user flow, and test one intentionally corrupted or incomplete backup so you know what failure looks like before it matters.