How to generate a list of unique lines in a text file using linux shell script?

Question

Suppose I have a file that contains a bunch of lines, some repeat:

line1 line1 line1 line2 line3 line3 line3 

Which linux commands should be used to create a list of unique lines:

 line1 line2 line3 

Does this change if the file is unsorted, i.e. duplicate lines may not be in blocks?

Answer 1

If you do not mind the sort being displayed, use

 sort -u 

This sorts and removes duplicates.

Answer 2

cat , to output the contents sent to sort to sort them, redirected to uniq to print unique values:

cat test1.txt | sort | uniq

you do not need to do the sort part if the contents of the file are already sorted.