Rebuild from known-good inputs

After privileged compromise, proving every hidden change is gone is difficult. Rebuilding is usually the clearer recovery path.

Removing the visible backdoor does not prove a root-compromised host is clean. A rebuild costs time, but creates a baseline you can reproduce and review.

What “known-good” means

Create a new host from a supported image, apply reviewed automation, deploy patched artifacts, rotate all reachable credentials, and restore verified data. Every input must come from somewhere the attacker could not touch:

image:    provider Ubuntu 24.04 LTS image, not a snapshot of the old host
config:   setup script from Git, reviewed after the incident
app:      artifact rebuilt in CI from patched source
secrets:  all new values — the old ones are burned
data:     backup checked for integrity and inspected for planted content

Do not copy executables or system configuration from the suspect filesystem. Restore only required data after validation, and keep the old host isolated for the agreed evidence-retention period.

Verify before exposing

Compare configuration and expose traffic gradually while monitoring. Diff the new host’s package list against what the setup script is supposed to install:

dpkg -l | awk '/^ii/ {print $2}' > new-host-packages.txt
diff expected-packages.txt new-host-packages.txt

An empty diff means the machine matches its manifest. Then prove the old credentials really are dead:

ssh -i ~/.ssh/old_lab_key [email protected]
# [email protected]: Permission denied (publickey).

Point DNS at the new host, watch it under real traffic for a day, and keep the old machine isolated rather than deleted.

The tempting shortcut is restoring “just one file” from the old disk — a binary or a config that is annoying to regenerate. That single file can be the persistence you rebuilt to escape. If it cannot be rebuilt from source, treat it as data to inspect, not software to run.

Build a fresh host from the supported image, reviewed configuration, patched artifact, rotated secrets, and verified data. Compare it with the intended manifest, then prove an old credential and an unlisted package are absent.