Files and data

Read a file line by line

Preserve leading whitespace and backslashes while reading every line, including a final line without newline.

10 minute lesson

~~~

The reliable Bash pattern for reading a file line by line disables backslash escapes and preserves surrounding whitespace. It’s more ceremony than you’d expect, and every piece earns its place.

The pattern

Read servers.txt:

while IFS= read -r server || [[ -n $server ]]; do
  printf 'server=%s\n' "$server"
done < servers.txt

Three details do the work:

  • IFS= clears the field separator for this one read, so leading and trailing spaces survive instead of being trimmed.
  • -r stops read from treating backslashes as escape characters. Without it, a \ in the file arrives mangled or vanishes.
  • || [[ -n $server ]] handles a final line without a trailing newline. read returns non-zero at end of file, but it still filled the variable — this check processes that last piece instead of silently dropping it.

The redirection < servers.txt feeds the file to the loop as a whole.

Verify with a hostile file

Build a test file that covers the edge cases:

printf '  indented\nback\\slash\nlast line' > servers.txt
./read-servers
# server=  indented
# server=back\slash
# server=last line

Test blank lines, spaces, a backslash, and a final unterminated line. Then decide which lines the application should ignore — many scripts skip blanks and comments with a guard at the top of the loop:

[[ -z $server || $server == \#* ]] && continue

The failure mode: piping into the loop

This variant looks equivalent and isn’t:

count=0
cat servers.txt | while IFS= read -r server; do
  count=$((count + 1))
done
printf '%s\n' "$count"
# 0

Each part of a pipeline runs in its own subshell, so count was incremented in a child process and thrown away when the pipeline finished. The done < servers.txt redirection keeps the loop in your shell, and its variables survive. Reach for the redirection first; it’s also one process cheaper.

One last thing: a line is still untrusted data. Validate hostnames, paths, and options before using them in commands — a config file is an input, not a promise.

Lesson completed

Take this course offline

Get every free book and course as PDF and EPUB files.

Get the download library →