uniq
The uniq command will remove duplicated lines from a sorted file:
[student@localhost ~]$ cat names.txt adm bin bin operator root root root shutdown [student@localhost ~]$ uniq names.txt adm bin operator root shutdown
For the uniq command to work correctly, the file must first be sorted. Because the sort command has a “unique only” option (see the “sort” section in this chapter for details), it is more common to use the sort command, rather than the uniq command, to remove duplicates.
However, the uniq command has an interesting feature in that it will report how many duplicate lines were present; this feature is not available with the sort command:
[student@localhost ~]$ uniq -c names.txt 1 adm 2 bin 1 operator 3 root 1 shutdown