Monday, February 7, 2011

Simple awk Command to de-deplicate file

$ cat file.txt
DD:12:A
AA:11:N
EE:13:B
AA:11:F
BB:09:K
DD:13:X

#Based on first field. Duplicates are DD,AA
$ awk '!x[$1]++' FS=":" file.txt
DD:12:A
AA:11:N
EE:13:B
BB:09:K

Again,

$ cat file.txt
DD:12:A
AA:11:N
EE:13:B
AA:11:F
BB:09:K
DD:13:X


#This time, based on first and 2nd field.Only duplicate combination is (AA:11)
$ awk '!x[$1,$2]++' FS=":" file.txt
DD:12:A
AA:11:N
EE:13:B
BB:09:K
DD:13:X

No comments:

Post a Comment