Prepare an incident plan

An incident is the worst time to decide who can shut down production. Decisions made at 02:00, under pressure, with half the team asleep, are the worst decisions a team makes all year. Prepare the first moves while the system is healthy.

An incident plan for a small team is not a fifty-page policy. It is one page that answers the questions the first responder will actually have.

Names, not roles

Write down the incident lead, the technical contact per system, who holds decision authority (who may take production down, who may notify customers), the customer communication owner, and how to reach external providers. Names, with a fallback for each — the primary contact is on a plane exactly when you need them.

incident lead:    marta          fallback: jonas
prod shutdown:    marta or CTO
customer comms:   elena
cloud provider:   support plan 4412, portal + phone

Keep the supporting information current and reachable without production access: the architecture diagram, the asset table from module one, the secret inventory with owners, and the backup locations with restore steps. A plan that lives only behind the VPN that is down is not a plan.

The 02:00 test

Run the scenario in your head. A production database credential appears in a public build log at 02:00. The on-call developer can see it but does not know who may rotate it or whether rotation restarts the app. So they wait for the morning standup, and the credential stays live for seven hours.

That gap is what the plan removes. The relevant page looks like this:

secret: DATABASE_URL (production)
owner: marta (backup: jonas)
rotate: update in secret manager, redeploy — ~2 min downtime
authority: on-call may rotate WITHOUT approval at any hour
evidence: export the build log BEFORE rotating

Two lines carry the weight. “Evidence first” preserves the timeline before containment erases it. And the authority line pre-approves the safe action, so nobody waits for permission at night. A long policy is useless if the first responder cannot find the credential owner; the first-hour plan needs names, authority, commands, and a fallback contact.

Rehearse one scenario

A plan that has never been exercised is fiction. Run a timed tabletop for the leaked credential: walk through detection, decision, rotation, and communication, and record every decision and delay. Confirm the contacts answer, the rotation works in a test environment, and the team knows how to preserve the relevant logs. Every delay found in rehearsal is a delay removed from the real incident.