Design restart policy

Restart=on-failure can recover a service from crashes and non-zero exits. It cannot repair bad configuration. A service that dies because its config file is invalid will die again on every restart, forever, and the restart policy will hide that from a casual glance.

The knobs

[Unit]
StartLimitIntervalSec=60
StartLimitBurst=5

[Service]
Restart=on-failure
RestartSec=2

Restart=on-failure restarts on crashes, unclean signals, and non-zero exit codes, but not on a clean stop. Restart=always also restarts clean exits, which is right for daemons that should never exit on their own.

RestartSec= adds a delay between attempts. The default is 100 milliseconds, which turns a broken service into a tight loop. Two seconds is a saner floor.

StartLimitIntervalSec= and StartLimitBurst= (in the [Unit] section) rate-limit starts: with the values above, five failures within sixty seconds put the unit in a permanent failed state instead of looping.

How Restart masks a crash loop

Here is the trap. You check the service, it says active (running), and you move on. But it has been crashing every few seconds and systemd keeps resurrecting it. Uptime in systemctl status resets on each restart — a service that is always “up since 4 seconds ago” is a loop, not a healthy daemon.

The restart counter makes it unambiguous:

systemctl show demo-api.service -p NRestarts

NRestarts=47

A non-zero, climbing NRestarts means the policy is papering over a real failure. The journal shows the same story as repeated Scheduled restart job, restart counter is at N lines. An operator should still see a service that repeatedly fails — alert on the counter or on the failed state, not just on “is it running”.

Try it

Make a disposable test service exit with status 1 (ExecStart=/bin/false works). Start it and watch the restart count and the rate limit trigger:

demo-fail.service: Start request repeated too quickly.
demo-fail.service: Failed with result 'start-limit-hit'.

Fix the cause, then clear the state with systemctl reset-failed demo-fail.service before starting it again.