Find file descriptor problems

Every process uses file descriptors for files, sockets, pipes, and other kernel objects. When a process runs out, you see this in the logs:

Error: EMFILE: too many open files, open '/var/lib/app/cache/items.json'

The symptom looks like a file problem. It’s really an accounting problem: the process hit its descriptor limit.

Count and compare

Get the service’s main PID and count its open descriptors:

systemctl show myapp -p MainPID
# MainPID=2143

ls /proc/2143/fd | wc -l
# 3987

Then check the limit that count is racing toward:

grep 'Max open files' /proc/2143/limits
# Max open files    4096    4096    files

3987 of 4096. This process is about to fail, and now you know why.

A single count doesn’t tell you whether it’s a leak or just a busy service. Compare descriptor counts over time: measure twice, minutes apart, under known activity. A steady 3900 on a server that handles thousands of connections may be legitimate sizing. A count that climbs and never comes down is a leak.

Group by type to find the leak

Inspect /proc/PID/fd or use lsof and group what you find:

sudo lsof -p 2143 | awk '{print $5}' | sort | uniq -c | sort -rn | head

   3418 sock
    521 REG
     14 CHR
      8 FIFO

3418 sockets tells a very different story than 3418 regular files. Sockets piling up usually means connections opened and never closed — often to a database or an upstream API. Regular files piling up means file handles are opened without being closed, and lsof will show you which paths.

Fix the right thing

If it’s a genuine capacity issue, raise the limit where systemd will actually apply it:

# systemctl edit myapp
[Service]
LimitNOFILE=16384

Then systemctl daemon-reload && systemctl restart myapp and re-check /proc/PID/limits.

The trap: raising a limit can delay a leak without fixing it. If descriptors grow without bound, 16384 just moves the crash from Tuesday to Friday, with four times more open sockets to clean up. Identify the descriptor type and owner first, fix the code path or connection pool that never closes, and treat the limit bump as breathing room, not the repair.