Fetch Uniq elements from the array efficiently using grep or hash

Through hash:

my %faculty_hash = ();
foreach my $facs (@faculty) {
  $faculty_hash{$facs} = 1;
}
my @faculty_unique = keys(%faculty_hash);


Please note: Some of the answers containing a hash will change the ordering of the array. Hashes dont have any kind of order, so getting the keys or values will make a list with an undefined ordering.
To preserver the order of the elements use grep:


my %seen;
my @unique = grep { ! $seen{$_}++ } @faculty;

Other ways:
use List::MoreUtils qw/ uniq /;
my @unique = uniq @faculty;
foreach ( @unique ) {
    print $_, "\n";
}

Reference: 

http://stackoverflow.com/questions/439647/how-do-i-print-unique-elements-in-perl-array 

Perl One-lIners

The top 10 one-liner tricks

Trick #1: -l

Smart newline processing. Normally, perl hands you entire lines, including a trailing newline. With -l, it will strip the trailing newline off of any lines read, and automatically add a newline to anything you print (including via -p).

Suppose I wanted to strip trailing whitespace from a file. I might naïvely try something like

perl -pe 's/\s*$//'

The problem, however, is that the line ends with "\n", which is whitespace, and so that snippet will also remove all newlines from my file! -l solves the problem, by pulling off the newline before handing my script the line, and then tacking a new one on afterwards:

perl -lpe 's/\s*$//'

Trick #2: -0

Occasionally, it's useful to run a script over an entire file, or over larger chunks at once. -0 makes -n and -p feed you chunks split on NULL bytes instead of newlines. This is often useful for, e.g. processing the output of find -print0. Furthermore, perl -0777 makes perl not do any splitting, and pass entire files to your script in $_.

find . -name '*~' -print0 | perl -0ne unlink

Could be used to delete all ~-files in a directory tree, without having to remember how xargs works.

Trick #3: -i

-i tells perl to operate on files in-place. If you use -n or -p with -i, and you pass perl filenames on the command-line, perl will run your script on those files, and then replace their contents with the output. -i optionally accepts an backup suffix as argument; Perl will write backup copies of edited files to names with that suffix added.

perl -i.bak -ne 'print unless /^#/' script.sh

Would strip all whole-line commands from script.sh, but leave a copy of the original in script.sh.bak.

Trick #4: The .. operator

Perl's .. operator is a stateful operator -- it remembers state between evaluations. As long as its left operand is false, it returns false; Once the left hand returns true, it starts evaluating the right-hand operand until that becomes true, at which point, on the next iteration it resets to false and starts testing the other operand again.

What does that mean in practice? It's a range operator: It can be easily used to act on a range of lines in a file. For instance, I can extract all GPG public keys from a file using:

perl -ne 'print if /-----BEGIN PGP PUBLIC KEY BLOCK-----/../-----END PGP PUBLIC KEY BLOCK-----/' FILE

Trick #5: -a

-a turns on autosplit mode – perl will automatically split input lines on whitespace into the @F array. If you ever run into any advice that accidentally escaped from 1980 telling you to use awk because it automatically splits lines into fields, this is how you use perl to do the same thing without learning another, even worse, language.

As an example, you could print a list of files along with their link counts using

ls -l | perl -lane 'print "$F[7] $F[1]"'

Trick #6: -F

-F is used in conjunction with -a, to choose the delimiter on which to split lines. To print every user in /etc/passwd (which is colon-separated with the user in the first column), we could do:

perl -F: -lane 'print $F[0]' /etc/passwd

Trick #7: \K

\K is undoubtedly my favorite little-known-feature of Perl regular expressions. If \K appears in a regex, it causes the regex matcher to drop everything before that point from the internal record of "Which string did this regex match?". This is most useful in conjunction with s///, where it gives you a simple way to match a long expression, but only replace a suffix of it.

Suppose I want to replace the From: field in an email. We could write something like

perl -lape 's/(^From:).*/$1 Nelson Elhage <nelhage\@ksplice.com>/'

But having to parenthesize the right bit and include the $1 is annoying and error-prone. We can simplify the regex by using \K to tell perl we won't want to replace the start of the match:

perl -lape 's/^From:\K.*/ Nelson Elhage <nelhage\@ksplice.com>/'

Trick #8: $ENV{}

When you're writing a one-liner using -e in the shell, you generally want to quote it with ', so that dollar signs inside the one-liner aren't expanded by the shell. But that makes it annoying to use a ' inside your one-liner, since you can't escape a single quote inside of single quotes, in the shell.

Let's suppose we wanted to print the username of anyone in /etc/passwd whose name included an apostrophe. One option would be to use a standard shell-quoting trick to include the ':

perl -F: -lane 'print $F[0] if $F[4] =~ /'"'"'/' /etc/passwd

But counting apostrophes and backslashes gets old fast. A better option, in my opinion, is to use the environment to pass the regex into perl, which lets you dodge a layer of parsing entirely:

env re="'" perl -F: -lane 'print $F[0] if $F[4] =~ /$ENV{re}/' /etc/passwd

We use the env command to place the regex in a variable called re, which we can then refer to from the perl script through the %ENV hash. This way is slightly longer, but I find the savings in counting backslashes or quotes to be worth it, especially if you need to end up embedding strings with more than a single metacharacter.

Trick #9: BEGIN and END

BEGIN { ... } and END { ... } let you put code that gets run entirely before or after the loop over the lines.

For example, I could sum the values in the second column of a CSV file using:

perl -F, -lane '$t += $F[1]; END { print $t }'

Trick #10: -MRegexp::Common

Using -M on the command line tells perl to load the given module before running your code. There are thousands of modules available on CPAN, numerous of them potentially useful in one-liners, but one of my favorite for one-liner use is Regexp::Common, which, as its name suggests, contains regular expressions to match numerous commonly-used pieces of data.

The full set of regexes available in Regexp::Common is available in its documentation, but here's an example of where I might use it:

Neither the ifconfig nor the ip tool that is supposed to replace it provide, as far as I know, an easy way of extracting information for use by scripts. The ifdata program provides such an interface, but isn't installed everywhere. Using perl and Regexp::Common, however, we can do a pretty decent job of extracing an IP from ip's output:

ip address list eth0 | \

  perl -MRegexp::Common -lne 'print $1 if /($RE{net}{IPv4})/'

Linux: Convert File Types ( MIME)

Working on difference between .txt files :

vimdiff     httpserver_directory.txt Entire_productsxml.txt       

Commands used for Comparision:

1.vimdiff     httpserver_directory.txt  Entire_productsxml.txt   

 not  working as  MIME types are different. One is xml  and the other is text file.

2.I needed  iconv  command here  to conver the MIME types   from  text/xml   to ascii/xml
a. Do these files have  same character sets?

No.
Hint from google: http://www.google.co.in/url?sa=t&rct=j&q=iconv%3A%20conversion%20from%20%60text%2Fxml%27%20and%20to%20%60text%2Fplain%27%20are%20not%20supported%20%3F&source=web&cd=6&cad=rja&ved=0CEEQFjAF&url=http%3A%2F%2Funix.stackexchange.com%2Fquestions%2F11602%2Fhow-can-i-test-the-encoding-of-a-text-file-is-it-valid-and-what-is-it&ei=xvRjUK3_LoqHrAeD6IHYAQ&usg=AFQjCNELrZ1L4ptxCXQckmAXG67VBHdboQ
 file1: text/plain; charset=utf-8 file2: text/plain; charset=iso-8859-1

b.First see the character sets  of each file :
Command :

file -i *

 [root@blr-elorhel4x86 ndtrack]# file -i *

Entire_productsxml.txt:   text/plain; charset=us-ascii

Entire_productsxml.txt:   application/x-empty

httpserver_directory.txt:

httpsever_ouionly.txt:    text/plain; charset=us-ascii

ndtrack.ini:              text/plain; charset=utf-8

ndtrack.sh:               application/x-shellscript

run.sh:                   application/x-shellscript

c.To convert from one  format  to other format:
Command: I will use it later, because I need to  some   text parsing.

iconv -f FROM-ENCODING -t TO-ENCODING file.txt

d.How  am I doing text parsing?

I am using perl  command line to wipe off beginning spaces in the line to act on to file:

Re-learn : perl -nle '$_ =~ s/^(\w+)$/$&/g ; print  $_' Entire_productsxml.txt   

Beginning spaces in the line are not replaced. Simple, but I need to remember it.

remove r-lines with 5th field set to empty string

Here is the perl one-liner to do that :

I want to leave h-line and t-line headers as they are remove r-line with 5th field set to empty string.This is how i did.

perl -nlaF"^A" -e 'if( $F[0] =~ /^h|t/){ print }; next if $F[4] == "" ; print '

la.1288854355.multiple.set1.72.247.41.47.9.1288854355.1.9.done > /tmp/good.done

find count of files in each directory

using find it is easy to find all files in directories recursively and print the total count of files.But, i want files to be listed in each directory.This can be done in perl one liner.

perl -e '
sub z {
my(@l)=glob(($p=$_[0])."/*");
my(@x,$c);
foreach $f(@l){($c++) if -f $f;
push(@x,$f) if -d $f }
print"$p: $c\n" if $c >0;
foreach $f (@x){z($f)}}z(".");'

text extraction and reformat indexing

Input file:

cat filelist_input | head -10
1 ./ab-138.done.done
2 ./ab-69.done.done
3 ./ab-137.done.done
4 ./ab-109.done.done

Expected output :

1 ab-138.done.done
2 ab-69.done.done
3 ab-137.done.done
4 ab-109.done.done

Solution:

cat filelist_input | head -10 | perl -nle '$_=~s#^(\d+)\s./(\w+)#$1 $2#g and print $_;'

Note: perl is very fast incase of operation on lakhs of lines.

extract part of url

I want to extract strings after http:// and before querystring ? .This is how i did for the following url:

bash$echo "http://nwidget.networkedblogs.com/getnetworkwidgetmain?bid=224747&fancount=28"
\ | perl -ne 'print "input url : $_\n"; $_=~s#.*?.*//(.*?/.*?)\?.*#$1# and print "extract sting : $_\n";'

o/p:
input url : http://nwidget.networkedblogs.com/getnetworkwidgetmain?bid=224747&fancount=28
extracted string : nwidget.networkedblogs.com/getnetworkwidgetmain

find and perl

I want to find a set of files table_pid_id1.xml , table_pid_id2.xml and table_pid_id3.xml with ids witten inside the xmls in the first line as " tableid='id1' rowid='pid' " etc. Also, i want to rename these files to table_newpid_id1.xml etc and replace pid inside each file with "newpid" .This is the way i can do using combination of find and perl.

Assume dir structure as dir1/dir2/table_pid_id1.xml etc..

Command to find these files and replace the strings inside xml file using perl :

a)first find the files and rename them from table_pid_id1.xml to table_newpid_id1.xml etc..

for fn in `find dir1/ -maxdepth 2 -name "*pid*" ` ; do echo $fn ; mv $fn $(echo $fn | sed 's/pid/newpid/') ;done

b)then, find the new files and replace the pid inside the files with newpid .

find dir1/ -maxdepth 2 -name "*pid_*.xml" | xargs -i perl  -i.bak -wple '$_ =~ s/'pid'/'newpid'/g;' {} ;

find number of lines starting with letter z

I want to find the lines starting with "z" of all the files inside ./input/inputfiles and count them.

find ./input/inputfiles/ -type f | xargs -i perl -wnle '/^r/ and print $_; ' {}  | wc -l

extracting uniq epoch timestamps

I have a large set of files (2 lac files) of which every line has fields separated by a Ctrl-B(^B) character. Every line starts with letter "z" , I want to extract 5th field which is timestamp and see the uniq day time stamps of each file. This is how i can do:

1)looping accross all the files in the directory and printing filename and uniq epoch timestamps into a file.

for file in `ls -tr` ; do echo $file >> /tmp/epochs;  perl -nlaF"^B" -e 'if ($F[0] eq "z") { my @t= eval "$F[4]-($F[4] % 86400)" ; print @t; } ' $file  |
sort -u  >> /tmp/epochs ; done

or

for file in `ls -tr` ; do echo $file >> /tmp/epoch.two;  perl -nlaF"^B" -e 'if ($F[0] eq "z") { my $t= eval "$F[4]-($F[4] % 86400)" ; print $t; } ' $file
| sort -u  >> /tmp/epoch.two ; done

2)taking only the epochs from that file and converting them to readable format.

perl -wnl -e ' if ( length() == 10 ) {  print $_;} ' /tmp/epochs |  while read line ; do date --date "1970-01-01 UTC $line seconds" ; done

grep using perl

I want to grep lines containing words like Broadcasts or broadcasts from a file like:

cat tv.txt :
this is british broadcasting channel in competition with other abc Broadcasts \n
and foo broadcasts telecasting \n
cricket \n

One-liner grep using perl :

perl -wnl -e 'm|Broadcasts|i and print $_;'  tv.txt

replace a string in a file with another

replace string in a group of files and movr the original files to file.bkp to be safe.

perl -i.bak -wpl -e 's/include/exclude/g;' datasource-4*.xml

Note:

Option "p" is for automating printing of rest of the file along with the strings which are newly placed.

print only records matching a condition

I want to print the records of ":" seperated fields of every line of a file whose last field i.e my website whose length is not greater than 4k.

Also, Print the records in the same format as input with the same ":" seperator.

cat mydata.log:

name:city:mailid:phone:mywebsite
name1:city1:mailid1:phone1:mywebsite1
.
.
....

One-line:

perl -nlaF":" -e 'if ( length($F[4]) < 3950 ) { $"=":"; print "@F"} ; '  mydata.log

Note:

• @F array is the perl default array which takes the fields seperated by the delimiter specified after the "F" option.
• option "n" is for implicit looping.
• option "l" is for automating line ending.
• option "a" is for automatic split of line as specified by F" " delimiter.
• Since mywebsite is the last field we can write $F[-1] in place of $F[4] in the if loop.
• $" is the seperator by which elements of an array interpolated into a string are joined together.
  

Use perl map command to print hash

use map command to print hash key and value in one line :

print "R-line metrics:", map { "$_=$myhash{$_} " } sort keys %myhash;

extact attributes out of an xml section

I have the following xml section, I want to extract only attributes i.e values enclosed in " " inside key element. This is how i do,

bash$echo " <key name="extract1">
                    </key>
                <key name="extract2">
                </key>
                <key name="extract3">
                </key>
                <key name="extract4">
                </key>" | perl -nle '$_ =~ m#.*<key name="(\w+)">.*#s and print $1;'
extract1
extract2
extract3
extract4