Read filesystem and kernel evidence

Write failures can come from full blocks, full inodes, permissions, quotas, read-only filesystems, or device errors. They often share one misleading error message, so the job is walking the possibilities in order instead of guessing.

The symptom: an application logs No space left on device or Read-only file system and stops writing.

Blocks, then inodes

df -h /var/lib/app
# Filesystem      Size  Used Avail Use% Mounted on
# /dev/sda1        50G   31G   17G  65% /var

65% used, plenty of space — and yet the write fails. Check inodes, because No space left on device also fires when the filesystem runs out of them:

df -i /var/lib/app
# Filesystem      Inodes  IUsed  IFree IUse% Mounted on
# /dev/sda1      3276800 3276800     0  100% /var

There it is. Millions of tiny files — often a cache, session store, or mail queue — consumed every inode while using a third of the blocks. The fix is deleting or consolidating the small files, not adding disk.

Confirm you’re looking at the right filesystem, too. findmnt /var/lib/app shows which mount actually receives the write; a path that looks like it lives on the big data volume may sit on the small root filesystem because a mount failed at boot.

Space that won’t come back

If df says full but du can’t find the data, a process is holding deleted files open. The kernel frees those blocks only when the last descriptor closes:

sudo lsof +L1 /var
# COMMAND  PID  USER  FD  SIZE/OFF NLINK NAME
# app     2143  app   4w  18734211072    0 /var/log/app/debug.log (deleted)

18GB in a deleted log. Restart the writer (or make it reopen its logs) and the space returns.

That’s also the trap in this lesson: “clearing” a live log with rm. The file vanishes from the directory, the process keeps writing to it, and the disk stays full while you can no longer see why. Truncate instead: truncate -s 0 /var/log/app/debug.log frees the space and keeps the descriptor valid.

When the kernel took the filesystem away

Read-only file system on a mount that should be writable usually means the kernel saw device errors and remounted it read-only to protect the data. Check the kernel log with journalctl -k or dmesg for lines like I/O error, dev sda or EXT4-fs error. If the kernel remounted a filesystem read-only, preserve data and investigate the storage path before forcing it writable — remounting rw over a failing disk converts a warning into corruption.