awk inside perl

what all characters to escape while executing awk script in perl:

my $cmd = "awk -F\"^A\" '\$1==\"r\" && \$5 == $time {sum+=\$16} END{printf(\"\%f\",sum)}' ma.959.200.$time.*.done" ;

my $bdl = exec $cmd;

print "$bdl";

awk multiple spaces as literal seperators

I want to treat single space also as a separator instead of default awk setting using -F , which will treat consecutive spaces as a single space.This way, we can do it:

echo hello1 192.168.1.9|awk 'BEGIN {FS="[. ]"};{print $2 "." $3 "." $4}'

192.168.1

Above, I set Field Separator to either a dot or a space. But one warning: The default field separation of white space allows any number of consecutive spaces/tabs to count as a single separator. But when setting FS like this, multiple consecutive spaces will be taken as multiple separators, so this can mess you up with additional null fields.

And that is why I did not use the quotes on my echo command. The presence of quotes would preserve the extra spaces, but without the quotes, echo will output each operand with only one space between.

removing duplicates words from a line

I have a file with each line having repetitive words. I want to remove duplicate words out of each line and print all the lines in the same order.

Input file:

bash$ cat line.txt

i am fine sriram
krishna how r u , how r u sriram , How r u jadu
thank you , Thankyou sir, thank you

Required output:

i am fine sriram
krishna how r u , sriram How jadu
thank you, Thankyou sir,

Here is the awk one-liner :

 awk '{ while(++i<=NF) printf (!a[$i]++) ? $i FS : ""; i=split("",a); print "" }' line.txt

using multiple delimiter in awk

I have text like this:
cat my_details :

"125"|"raghu"|"grep"
"125"|"sriram"|"ms"
"450"|"khanna|"nokia"
"451"|""hari"|"maxplore"

expected output : get only lines whose first field is equal to 125 :

awk -F "[\",|]" '$2==125' my_details

append a character at the end of line

one way i appended a character at the end of each line using awk .

awk 'BEGIN{OFS=""} { print $NF,"\\r"}' filename

it appends the character "\r" at the end of every line

sort and uniq combined using awk

Say, I have the following IP addresses file ip.txt :

cat > ip.txt
1.2.3.4
4.5.6.7
1.2.3.4
8.9.0.1
4.5.6.7

How to see the unique no. of ips in the above file ?

way 1: sort ip.txt | uniq

way 2 using awk : cat ip.txt | awk '!x[$0]++'

sum of all values of a column

Summation of all values from 2nd to 6th field in a file with fields seperated by a space delimiter :

for i in `seq 2 6`; do awk ' $1 !~ /Name/ {sum+=$"'"$i"'"} END {print "field" "'"$i"'",")",sum}' data.out ;done

Here is out data.out :

Name salary incomtax pf hra allowance

sachin 4000 400 200 400 100
dhoni 5000 500 200 400 100

Replacing a column value of a file

Replace number 7 with 8 of the last field of all the data.*.out files (every line of the file begins with a field "sriram") all the fields are seperated by | delimiter:

awk 'BEGIN {FS=OFS="|"} $1=="sriram" && $NF==7 {$NF=8} {print}' data.1.out

Note :

For the multiple data.[1-4].out files above , inorder to replace at a time, we can write as shown below:

for file in `ls data.*`; do echo $file; awk 'BEGIN {FS=OFS="|"} $1=="sriram" && $NF==7 {$NF=8}' $file > $file.tmp; mv $file.tmp $file; done

number of first fields beginning with a #

find number of fields or columns in a line beginning with # of a file :

awk '$1 ~ /^#/ { print NF,$NF}' pop.pl

Exp : Here, we are trying to match first field which begins with a #.

sum of 2nd column of 4 files

suppose i have 4 files like ip1,ip2,ip3,ip4. I want to sum all the second column values of all the four files with each file containing dot seperated content like 172.22.21.21 etc..How to do using awk in one line?

awk -F"." '{sum+=$2} END {print sum}' ip*

get the number of columns in the heading of a file

Number of fields or columns in the first line of a file

take a file menu.sh which is of content like this:

Name empid Salary role DOB

robin 2981 10000 Engineer 27-8-93

awk 'NR==1 {print NF}' menu.sh

print even or odd numbered lines

Print the even-numbered lines in the data file:

awk 'NR % 2 == 0' data

If you use the expression `NR % 2 == 1' instead, the program would print the odd-numbered lines.

wc -l in awk

Count the lines in a file:

awk 'END { print NR }' file.dat

list of usernames in a unix system

Print a sorted list of the login names of all users:

awk -F: '{ print $1 }' /etc/passwd | sort

calculate size of a file

Print the total number of bytes used by a file called files:

ls -l files | awk '{ sum += $5 } END { print "total bytes: " , sum }'

Similar way for Kilo-bytes :

Print the total number of kilobytes used by a file called files:

ls -l files | awk '{ x += $5 } END { print "total K-bytes: " (x + 1023)/1024 }'

random number generation

Print seven random numbers from 0 to 100, inclusive:

awk 'BEGIN { for (i = 1; i <= 7; i++) print int(101 * rand( )) }'

Remove blank lines in a file

Print every line that has at least one field:

This is an easy way to delete blank lines from a file (or rather, to create a new file similar to the old file but from which the blank lines have been removed).

awk 'NF > 0' data

Longest line in a file

Print the length of the longest line in data file :

expand data | awk '{ if (x < x =" length(">

Line longer than x characters

Print every line that is longer than 80 characters:

        awk 'length($0) > 80' data

The sole rule here has a relational expression as its pattern and it has no action—so the default action, printing the record, is used.

Some handy awk one liners

Print the length of the longest input line:

awk '{ if (length($0) > max) max = length($0) } END { print max }' data