Diagnose storage latency

A filesystem can have free space while storage latency makes every request slow. df -h says 60% free, the app times out anyway, and everyone is confused. Capacity and speed are different problems.

The symptom pattern: requests slow down across the board, load average climbs, but CPU sits mostly idle with high I/O wait. That combination says “storage,” and the next job is finding which device and which process.

Measure the device

iostat -xz 5 3

Device   r/s    w/s   rkB/s   wkB/s  r_await  w_await  aqu-sz  %util
sda     2.1   312.4    84.2  48120.5     3.1    187.4    18.2   99.6

Skip the first sample — it’s the average since boot, not the current state. Then read three fields. r_await and w_await are the average milliseconds each read or write waits, queue time included: single-digit values are fine for SSDs, and 187ms per write is a disaster. aqu-sz is the average queue depth — 18 requests waiting means work arrives much faster than the device completes it. %util near 100 says the device was busy the whole interval.

Map the busy device to what’s mounted on it:

findmnt -o TARGET,SOURCE,FSTYPE /dev/sda1
# /var  /dev/sda1  ext4

Find the process creating the work

I/O wait needs context: it can coexist with idle CPUs and does not identify the responsible process alone. pidstat -d does:

pidstat -d 5 2

      UID       PID   kB_rd/s   kB_wr/s  Command
      998      4110      0.00  46210.30  pg_dump
     1001      2143     12.40     88.10  node

There’s the answer: a backup job writing 46 MB/s to the same disk the application lives on. The fix isn’t faster hardware, it’s moving the backup window, throttling it with ionice, or pointing it at a different volume.

Confirm from the application side too — database slow-query logs or request timing should improve the moment the writer stops. Capture device samples during a known read or write, and match the busy device to its mount and the process creating work.

The trap is trusting %util on SSDs and NVMe. Those devices serve many requests in parallel, so “100% busy” can still mean plenty of spare capacity. On modern flash, judge saturation by await climbing and aqu-sz growing, not by the utilization column.