Prevent overlapping runs

A backup that takes 40 minutes, scheduled every 30, will eventually run alongside itself. Backups, deployments, and cleanup jobs can corrupt state or overload systems when two copies overlap: two tar processes writing the same archive, two deployments moving the same symlink. The fix is a lock. The first run takes it, and later runs see it is held and stop.

Use flock on Linux

exec 9>/tmp/practical-job.lock
if ! flock -n 9; then
  printf '%s\n' 'job already running' >&2
  exit 75
fi

exec 9> opens file descriptor 9 writing to the lock file and keeps it open for the rest of the script. flock -n 9 asks the kernel for an exclusive lock on that descriptor. -n means do not wait. Fail immediately if another process holds it.

The kernel releases the lock when the process exits, however it exits. Crash, Ctrl-C, kill: the lock disappears with the process. That is what makes flock better than the homemade “write a pidfile and check it” approach. A stale pidfile from a crashed run blocks every future job until a human deletes it.

Exit status 75 is the conventional EX_TEMPFAIL, temporary failure, try again later. It tells a scheduler this was not an error. Another copy was doing the work.

Verify the lock holds

Hold the first process open and start a second:

./nightly-backup &      # first run, holds the lock
./nightly-backup
# job already running
printf '%s\n' "$?"
# 75

The second should fail quickly with a distinct status. Then wait for the first run to finish and start again. It must acquire the lock and proceed normally. Both checks matter. A lock that never blocks is useless, and a lock that never releases is worse.

Know the lock’s limits

A lock needs stable scope and cleanup behavior. The scope of this one is a single machine. The path identifies the job, so two different jobs need two different lock files. On a shared machine consider a per-user directory instead of /tmp so another user cannot squat on the name.

For distributed jobs (the same task scheduled on several servers), a local file lock is not enough. Those need coordination all the machines share, such as a database lock.