A new guide has been published: Linux Shell: Remove Duplicate Lines from File
You can use the Linux shell to remove duplicate lines from a file, as demonstrated in a brief tutorial by TecAdmin.
Read more @ Linux Compatible