A quick guide to the `uniq` command, used to work with duplicate records/lines in text
uniq is a command useful to sort lines of text.
You can get those lines from a file, or using pipes from the output of another command:
uniq dogs.txt ls | uniq
You need to consider this key thing:
uniq will only detect adjacent duplicate lines.
This implies that you will most likely use it along with
sort dogs.txt | uniq
sort command has its own way to remove duplicates with the
-u (unique) option. But
uniq has more power.
By default it removes duplicate lines:
You can tell it to only display duplicate lines, for example, with the
sort dogs.txt | uniq -d
You can use the
-u option to only display non-duplicate lines:
You can count the occurrences of each line with the
Use the special combination:
sort dogs.txt | uniq -c | sort -nr
to then sort those lines by most frequent:
uniq command works on Linux, macOS, WSL, and anywhere you have a UNIX environment