Processes and jobs

Linux commands: nohup

Learn how the Linux nohup command keeps a process running after you log out or close the terminal, which is handy for long-lived jobs on a remote server.

When you close a terminal, the shell sends a hangup signal (HUP) to every job it started. Most programs exit when they receive it. nohup starts a command with that signal ignored, so the command survives the terminal closing.

The typical use is a long job on a remote server. You SSH in, start something that takes an hour, and you want to log out without killing it:

nohup ./generate-report >report.log 2>&1 &

Each part has a job:

  • nohup protects the command from the hangup signal.
  • >report.log sends standard output to a file.
  • 2>&1 sends errors to the same file.
  • & gives you the prompt back while the process runs in the background.

The shell prints [1] 41522. That second number is the process ID, and you want to keep it.

If you skip the redirection, nohup picks a file for you and tells you about it:

nohup ./generate-report &
nohup: ignoring input and appending output to 'nohup.out'

That works, but I prefer choosing the file name myself. nohup.out in a random folder is easy to forget about.

Grab the PID right away with $!, which holds the ID of the last background job, then check the process is alive and watch its log:

job_pid=$!
ps -p "$job_pid"
tail -f report.log

ps -p prints one line with the PID and the command name if the process exists, and only the header if it’s gone.

Later, stop the process with kill "$job_pid". If you lost the PID, find it again with ps ax | grep generate-report.

What nohup does not do

Running a command with & alone only puts it in the background. It still receives the hangup signal when the session closes. nohup and & solve two separate problems, and for a job that must outlive your login you need both.

nohup is not a service manager. It does not restart a crashed process, start it after a reboot, rotate logs, or tell you the status. For a production service use systemd, a container orchestrator, or another supervisor. Use tmux or screen when you want to reconnect to an interactive terminal later, rather than detach a non-interactive job.

Two things go wrong in practice. First, the log grows forever. nohup.out and other logs nobody rotates can fill the disk on a small server, so check its size once in a while. Second, the program asks a question. A command waiting for input from a terminal that no longer exists just sits there, or exits. Make sure the job runs without prompts before you detach it.

Try it with nohup sh -c 'sleep 10; date' >result.log 2>&1 &, close the terminal, open a new one, and cat result.log. The timestamp is there, written after you left.

Lesson completed