Monitor capacity and disk health

Storage fails in two ways, and both announce themselves in advance if you look. A filesystem can run out of bytes or inodes. A disk can also report media errors before complete failure. Services on a full filesystem don’t crash politely — they corrupt their own state mid-write, which turns a capacity problem into a restore problem.

Capture a baseline

df -h
df -i
sudo dmesg --level=err,warn | tail -50
sudo smartctl -H /dev/sdX

df -h shows used space per filesystem. df -i shows inodes — the metadata slots filesystems use, one per file. A service writing millions of tiny files can hit 100% inode usage while df -h still shows free space, a genuinely confusing failure until you know to check.

The dmesg filter surfaces kernel-level I/O errors: lines mentioning I/O error or a specific device resetting are early hardware warnings. smartctl (from the smartmontools package, sudo apt install smartmontools) queries the disk’s own health assessment. Replace sdX with your real device only after resolving the exact disk against your inventory — you want PASSED:

SMART overall-health self-assessment test result: PASSED

Anything else, and it’s time to plan a replacement while the disk still reads.

Watch growth, not snapshots

A single reading tells you little; the trend is what predicts trouble. Review growth by directory when a filesystem fills faster than expected:

sudo du -xh --max-depth=1 /srv/data | sort -h

The biggest line at the bottom is usually a log directory or a cache nobody rotates. Set alerts below full capacity — 80% is a good threshold, because it leaves you time to act on a weekend rather than during the failure. Re-run the baseline weekly at first; a later lesson folds this into one observation checklist.

The mistake

Treating a healthy SMART status as a safety guarantee. SMART data is one signal, not a backup. A drive can fail without warning — controller deaths in particular skip the polite degradation phase entirely. Monitoring buys you early warnings most of the time; the backup module is what covers the rest.