Linux Commands Examples

A great documentation place for Linux commands

uniq

report or omit repeated lines


see also : comm - join - sort

Synopsis

uniq [OPTION]... [INPUT [OUTPUT]]


add an example, a script, a trick and tips

: email address (won't be displayed)
: name

Step 2

Thanks for this example ! - It will be moderated and published shortly.

Feel free to post other examples
Oops ! There is a tiny cockup. A damn 404 cockup. Please contact the loosy team who maintains and develops this wonderful site by clicking in the mighty feedback button on the side of the page. Say what happened. Thanks!

examples

1
The example, 'sort ~/.bash_history | uniq | grep $1' will fail if $1 contains whitespace. Variable references should be quoted:



sort ~/.bash_history | uniq | grep "$1"
example added by Chris F.A. Johnson
0
source
            
sort *_movies.txt|uniq > all.txt
0
source
            
cat tags.csv | sort tags.csv | uniq > uniq_tags.csv
0
source

Remove duplicates in each line of a file

Since ruby comes with any Linux distribution I know of:

ruby -e 'STDIN.readlines.each { |l| l.split(" ").uniq.each { |e| print "#{e} " }; print "\n" }' < test

Here, test is the file that contains the elements.

To explain what this command does—although Ruby can almost be read from left to right:

  • Read the input (which comes from < test through your shell)
  • Go through each line of the input
  • Split the line based on one space separating the items, into an array (split(" "))
  • Get the unique elements from this array (in-order)
  • For each unique element, print it, including a space (print "#{e} ")
  • Print a newline once we're done with the unique elements
0
source

linux sort -n uniq -c

One way using awk:

awk '!array[$0]++' file.txt

Results:

valA
valB
valC
valZ


Food for thought:

sort -u file.txt

valA
valB
valC
valZ

< file.txt sort | uniq

valA
valB
valC
valZ

< file.txt sort | uniq -u                 # only print unique lines

valA
valC
valZ

< file.txt sort | uniq -d                 # only print duplicate lines

valB

0
source

option -f for linux command

From man uniq:

A field is a run of blanks (usually spaces and/or TABs), then non-blank characters. Fields are skipped before chars.

With -f2 you're skipping all your fields, so only the first line gets output, all the others are equal after the second field (none have more than two).

description

Filter adjacent matching lines from INPUT (or standard input), writing to OUTPUT (or standard output).

With no options, matching lines are merged to the first occurrence.

Mandatory arguments to long options are mandatory for short options too.
-c
, --count

prefix lines by the number of occurrences

-d, --repeated

only print duplicate lines

-D, --all-repeated[=delimit-method]

print all duplicate lines delimit-method={none(default),prepend,separate} Delimiting is done with blank lines

-f, --skip-fields=N

avoid comparing the first N fields

-i, --ignore-case

ignore differences in case when comparing

-s, --skip-chars=N

avoid comparing the first N characters

-u, --unique

only print unique lines

-z, --zero-terminated

end lines with 0 byte, not newline

-w, --check-chars=N

compare no more than N characters in lines

--help

display this help and exit

--version

output version information and exit

A field is a run of blanks (usually spaces and/or TABs), then non-blank characters. Fields are skipped before chars.

Note: ’uniq’ does not detect repeated lines unless they are adjacent. You may want to sort the input first, or use ’sort -u’ without ’uniq’. Also, comparisons honor the rules specified by ’LC_COLLATE’.

copyright

Copyright © 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

reporting bugs

Report uniq bugs to bug-coreutils[:at:]gnu[:dot:]org
GNU coreutils home page: <http://www.gnu.org/software/coreutils/>
General help using GNU software: <http://www.gnu.org/gethelp/>
Report uniq translation bugs to <http://translationproject.org/team/>


see also

comm , join , sort

The full documentation for uniq is maintained as a Texinfo manual. If the info and uniq programs are properly installed at your site, the command

info coreutils 'uniq invocation'

should give you access to the complete manual.


author

Written by Richard M. Stallman and David MacKenzie.

How can this site be more helpful to YOU ?


give  feedback