Executable File Searching Techniques

Following are the steps to perform executable File Search if there is no Installer Evidence or known File Evidence found:


1)    Search for a String Recursively and Redirect it to File:
•    Recursively searching for a string in all the files and redirect it to a file.

Syntax:      
              grep -ir “<pattern>” <PATH>     >      <FileName>

Options:
•    i: Ignore case
•    r: recursive search
                                                                                                              
Example:

          grep -ir "bpa\|suite\|analysis"  *  >   filelist.tmp


2)    File and strings search:
•    Search for a string in contents of the file.

         Syntax:
                      grep -i  “<pattern>”  <filename>  | grep –v “<pattern>”

         Options:
•    v: ignore these strings

         Example: 

          grep -i "bpa\|suite\|analysis" filelist.tmp | grep -v ".jar\|.html\|.doc\|.xml\|man1\|txt\|.loc"

3)    Search for Executables from desired path:
•    Lists all the executables in the particular path.

         Syntax:
find <path> -type f -perm 0755  -a ! -name “<pattern>”

         Options:
•    type     : Type of file to be searched, a file or a directory.
•    perm    : Permissions of the file
•    a           : AND Condition
•    !            : Not Operator
•    name    : Name of the File.
  
         Example:

         find . -type f -perm 0755 -a ! -name "java"  -a ! -name "*.msb" -a ! -name "*.qm" -a ! -name "*.sh" -a ! -name "*.jar"

Check if a filesystem is mounted or not

 # Use "mount" here, do not use "df" as if there is a FS problem "df" might hang
       filesystem=/tmp

        if [[ `mount | grep $filesystem ` = "" ]]
        then
 

                echo "The filesystem is not mounted !"
                exit  127
        fi

Check if a file system is local or remote

I want to check whether there is any nfs file system mounted on my solaris or linux machine:

Here is the code snippet to do that :
 # Take the FS entry (static) snapshot

                        fs_string=`df -k $fs | grep -v "Filesystem" | head -1`

                        # Next is true if $fs_string contains ":" as the separator for the "node" field

                        if [[ `echo $fs_string | cut -f1 -d":"` = `echo $fs_string` ]]
                        then
                                # No ":" in the df-returned string: this is a local FS
                                fs_node=""
                        else
                                # The df-returned string contains a ":" (i.e.) a node name: this is a remote FS
                                fs_node=`df -k $fs | grep -v "Filesystem" | head -1 | cut -f1 -d":"`
                        fi

How to extract cpio.gz file

I want to extract  oracle cpio.gz file to get oracle installer files.
Here it is:

Directions to extract the files
1. Run "gunzip <filename>" on all the files. Eg. ship_9204_linux_disk1.cpio.gz
2. Extract the cpio archives with the command "cpio -idmv < <filename>" 

Eg. cpio -idmv < ship_9204_linux_disk1.cpio

    means

# gunzip <your_file_name>.cpio.gz
# cpio -idmv < your_archive.cpio

means make it first cpio then -idmv .

Command to list the processes with the memory size they comsume

ps -eo vsz,pid,args | sort -n  will work.

fork and timeout control in shell script

I want to run a system command (network commands like ping , nslookup and traceroute etc...) and see if they are successful or kill them if they take long time to execute:

 Here is the command function and sample commands i have written to do that:

# Execute the passed Unix command in background
_Command()
{
    command=$@

    rm -f $TMP_FILE 2>/dev/null
    rm -f $TMP_ERROR_FILE 2>/dev/null
    rm -f $TMP_RC_FILE 2>/dev/null

    _Report "## Executing command [$command] in backgound to monitor execution time..."

    # Executing $command, redirecting stderr to $TMP_ERROR_FILE, saving stdout to
    # $TMP_FILE and the return code to $TMP_RC_FILE
   
    (eval $command >$TMP_FILE 2>$TMP_ERROR_FILE && echo $? >$TMP_RC_FILE || echo $? >$TMP_RC_FILE) &

    i="0"

    while true
    do
        jobstring=`jobs |grep 'eval $command'`

        if [[ `echo $jobstring` != "" ]]
        then
            # Or command did not complete yet, or it is stopped or in a bad status.
            # Let's check if we exceeded the maximum execution time allowed for the command.

            if [[ $i -gt $max_exec_time ]]
            then
                _Report "## Timeout [$max_exec_time seconds] exceeded, killing background command..."

                bg_process_string=`ps -ef |grep $! |grep "$command"`

                bg_pid=`echo $bg_process_string |cut -f2 -d" "`

                _Report "## Background command process is [PID=$bg_pid]..."
                _Report "## Killing background process [PID=$bg_pid]..."

                kill -9 $bg_pid

                _Report "## Done."

                _Report "BEGIN Command Output : $command "
                _Report "[$command] Command timed out."
                _Report "END [Timeout]"

                rc=$Cmd_failed

                break
            else
                _Report "## Waiting for background command to complete [$i second(s) elapsed]..."

                sleep 1

                (( i = i + 1 ))
            fi
        else
            _Report "## Background command completed, checking command return code..."

            cmd_rc=`cat $TMP_RC_FILE`

            if [[ $cmd_rc -ne 0 ]]
            then
                if [[ `cat $TMP_ERROR_FILE` == "" ]]
                then
                    _Report "## Background command terminated in error with [RC=$cmd_rc] and no error message..."

                    # Collecting all return codes of commands into a variable
                    Returns=$Returns,$cmd_rc

                    _Report "BEGIN Command Output : $command "
                    _Report "[$command] Command not successful."
                    _Report "END [RC=$cmd_rc]"

                    rc=$Cmd_failed

                    break
                else
                    _Report "## Background command terminated in error with [RC=$cmd_rc] and the following error message:"

                    # Log the command stderr to logfile
                    _Log
                    cat $TMP_ERROR_FILE >> $LOGDIR/$logfile
                    _Log

                    # Collecting all return codes of commands into a variable
                    Returns=$Returns,$cmd_rc

                    _Report "BEGIN Command Output : $command "
                    _Report "[$command] Command not successful."
                    _Report "END [RC=$cmd_rc]"

                    rc=$Cmd_failed

                    break
                fi
            else
                _Report "## Background command terminated successfully [RC=$cmd_rc]..."

                _Report "BEGIN Command Output : $command "
                _Log

                # Log the command stdout to logfile
                cat $TMP_FILE >> $LOGDIR/$logfile

                _Log
                _Report "END [RC=$cmd_rc]"

                # Do not set/override $rc to 0 here as it is unknown if previous commands failed

                break
            fi
        fi
    done

    rm -f $TMP_FILE
    rm -f $TMP_ERROR_FILE
    rm -f $TMP_RC_FILE
}

# Run first ckeck command
_Command "ping -c 3 $resourcename"

_Report

# Run second command
_Command "traceroute $resourcename"

_Report

# Run third command
_Command "nslookup $resourcename"

Note : _Report function and _Log function would be like this:

# Write plain text to logfile
_Log()
{
    echo "$@" >> $LOGDIR/$logfile
}

# Write info to logfile
_Report()
{
    #echo $@
    echo "`date +'%m/%d/%Y %H:%M:%S'` [$$] : $@" >> $LOGDIR/$logfile
}

Various syntaxes of if statement in shell

The basic rules of conditions

When you start writing and using your own conditions, there are some rules you should know to prevent getting errors that are hard to trace. Here follow three important ones:

1. Always keep spaces between the brackets and the actual check/comparison. The following won't work:

   if [$foo -ge 3]; then

   Bash will complain about a "missing `]'".
2. Always terminate the line before putting a new keyword like "then". The words if, then, else, elif and fi are shell keywords, meaning that they cannot share the same line. Put a ";" between the previous statement and the keyword or place the keyword on the start of a new line. Bash will throw errors like "syntax error near unexpected token `fi'" if you don't.
3. It is a good habit to quote string variables if you use them in conditions, because otherwise they are likely to give trouble if they contain
   spaces and/or newlines. By quoting I mean:

   if [ "$stringvar" == "tux" ]; then

   There are a few cases in which you should not
   quote, but they are rare. You will see one of them further on in the tutorial.

Also, there are two things that may be useful to know:

1. You can invert a condition by putting an "!" in front of it. Example:

   if [ ! -f regularfile ]; then

   Be sure to place the "!" inside the brackets!
2. You can combine conditions by using certain operators. For the single-bracket syntax that we've been using so far, you can use "-a" for and and "-o" for or. Example:

   if [ $foo -ge 3 -a $foo -lt 10 ]; then

   The above condition will return true if $foo contains an integer greater than or equal to 3 and Less Than 10. You can read more about these combining expressions at the respective condition syntaxes.

And, one more basic thing: don't forget that conditions can also be used in other statements, like while and until. It is outside the scope of this tutorial to explain those, but you can read about them at the Bash Guide for Beginners.

Anyway, I've only shown you conditions between single brackets so far. There are more syntaxes, however, as you will read in the next section.

 

To know more : 

http://www.linuxtutorialblog.com/post/tutorial-conditions-in-bash-scripting-if-statements

Invoke an awk command from a function within shell script

If we pass awk '{print $3}' to another function in shell script. $3 will be considered as third argument for that function.To make it as literal for shell and a field for awk , then we should write it as: awk '{print " ' $3' "}'.

Example :

_Command()
{
    echo  "BEGIN Command : $@ "
    #Executing command and redirecting stderr to devnull
    output=$(eval $@)
    #Exit status of the command
    stat=$?
    echo $output

    if [[ $stat -ne 0 ]] ; then
        #collecting all return codes of commands into a variable
        Returns=$Returns,$stat
        _Report "$@ not successful"
        _Report "END"
    else
        #Returns=$Returns,$stat
        _Report "$output"
        _Report "END"
    fi
}

#List of commands to be run for unix cpu usage problem
    _Command "ps -ef | grep \"<defunct>\" | grep -v grep | awk  '{print "'$3'"}'"

CPU usage by each process

check what is the CPU usage for each process :

ps -eo pid,vsz,pcpu,pid,args | sort -n -k3 


output: 


 1051  1956  0.0  1051 udevd
  118     0  0.0   118 [kseriod]
    1  1980  0.0     1 init [5]
13445  5724  0.0 13445 -bash
13549     0  0.0 13549 [pdflush]
14198  3008  0.0 14198 ps -eo pid,vsz,pcpu,pid,args
14199 29368  0.0 14199 sort -n -k3
15782 10328  0.0 15782 vim all_autocheck_fss.pl
  183     0  0.0   183 [ata/0]
  184     0  0.0   184 [ata/1]
  186     0  0.0   186 [scsi_eh_0]
  187     0  0.0   187 [scsi_eh_1]
  198     0  0.0   198 [kjournald]
21602 11040  0.0 21602 cupsd
 2216  2464  0.0  2216 syslogd -m 0
 2220  2952  0.0  2220 klogd -x
 2231  1852  0.0  2231 irqbalance
 2242  3440  0.0  2242 portmap
22545  2012  0.0 22545 in.telnetd: blr-p7106178
22549  3372  0.0 22549 login -- root
 2262  1888  0.0  2262 rpc.statd
 2295  4800  0.0  2295 rpc.idmapd
23231  2740  0.0 23231 in.telnetd: micro10.nesscorp
23301  3484  0.0 23301 login -- root
 2393  3480  0.0  2393 /usr/sbin/acpid
 2432  6848  0.0  2432 dovecot-auth
 2433  4216  0.0  2433 pop3-login
 2434  4344  0.0  2434 pop3-login
 2435  4784  0.0  2435 pop3-login
 2483 36848  0.0  2483 /usr/sbin/named -u named -t /var/named/chroot
 2493  4964  0.0  2493 /usr/sbin/sshd
 2511  3028  0.0  2511 xinetd -stayalive -pidfile /var/run/xinetd.pid
 2520  4472  0.0  2520 su - varun
 2521  4172  0.0  2521 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
 2524  4980  0.0  2524 -bash
 2540  7220  0.0  2540 sendmail: accepting connections
 2548  7660  0.0  2548 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
 2559  2368  0.0  2559 gpm -m /dev/input/mice -t imps2
 2602 19536  0.0  2602 /usr/sbin/cannaserver -syslog -u canna
26164  1860  0.0 26164 in.telnetd: maximoimage
26168  4720  0.0 26168 login -- sriram
26300  6180  0.0 26300 -bash
28135  4928  0.0 28135 -bash
28617  9556  0.0 28617 vim test1.sh
   29     0  0.0    29 [kblockd/0]
29174  4836  0.0 29174 -bash
29802  3516  0.0 29802 in.telnetd: blr-p7106178.nesscorp
29806  4572  0.0 29806 login -- root
   30     0  0.0    30 [kblockd/1]
    3     0  0.0     3 [ksoftirqd/0]
30660  6228  0.0 30660 su - root
   31     0  0.0    31 [khubd]
 3289  3904  0.0  3289 crond
  365  5692  0.0   365 -bash
   40     0  0.0    40 [pdflush]
    4     0  0.0     4 [migration/1]
 4008  4736  0.0  4008 xfs -droppriv -daemon
   42     0  0.0    42 [kswapd0]
   43     0  0.0    43 [aio/0]
   44     0  0.0    44 [aio/1]
 4633 10188  0.0  4633 smbd -D
 4948  9064  0.0  4948 nmbd -D
    5     0  0.0     5 [ksoftirqd/1]
 5030 10188  0.0  5030 smbd -D
 5245  2196  0.0  5245 /usr/sbin/atd
    6     0  0.0     6 [events/0]
 6081  3548  0.0  6081 dbus-daemon-1 --system
 6287  4236  0.0  6287 cups-config-daemon
 6378  8472  0.0  6378 hald
 6473  3380  0.0  6473 /sbin/mingetty tty1
 6474  3380  0.0  6474 /sbin/mingetty tty2
 6475  3292  0.0  6475 /sbin/mingetty tty3
 6555  2284  0.0  6555 /sbin/mingetty tty4
 6563  1652  0.0  6563 /sbin/mingetty tty5
 6564  1548  0.0  6564 /sbin/mingetty tty6
 6565 12568  0.0  6565 /usr/bin/gdm-binary -nodaemon
 6997  3212  0.0  6997 in.telnetd: maximoimage
    7     0  0.0     7 [events/1]
 7001  2984  0.0  7001 login -- sriram
 7647 13096  0.0  7647 /usr/bin/gdm-binary -nodaemon
 7663 12404  0.0  7663 /usr/X11R6/bin/X :0 -audit 0 -auth /var/gdm/:0.Xauth -nolisten tcp vt7
    8     0  0.0     8 [khelper]
    9     0  0.0     9 [kacpid]
 9520 22256  0.0  9520 /usr/bin/gdmgreeter
  964  6260  0.0   964 -bash
  PID   VSZ %CPU   PID COMMAND
    2     0  0.1     2 [migration/0]
 2403  4572  0.1  2403 /usr/sbin/dovecot
 2589  3188  5.5  2589 /usr/sbin/htt -retryonerror 0

tar:can't create /./. @LongLink : Permission denied

I see a strange when i extract my installer.tar bundle using tar -xvf  install.tar.

I find an alternative to normal tar which gnu tar i.e gtar which will support symbolic links.

This worked for me.

gtar -xvf  install.tar

Null glob

nullglob : (special character * is called a glob which expands to filenames or dirs etc..)

Normally, when no glob specified matches an existing filename, no pathname expansion is performed, and the globs are not removed:

$ echo "Textfiles here:" *.txt
Textfiles here: *.txt

In this example, no files matched the pattern, so the glob was left intact (a literal asterisk, followed by dot-txt).

This can be very annoying, for example when you drive a for-loop using the pathname expansion:

for filename in *.txt; do
echo "=== BEGIN: $filename ==="
cat "$filename"
echo "=== END: $filename ==="
done

When no file name matches the glob, the loop will not only output stupid text (”BEGIN: *.txt”), but also will make the cat-command fail with an error, since no file named *.txt exists.

Now, when the shell option nullglob is set, Bash will remove the entire glob from the command line. In case of the for-loop here, not even one iteration will be done. It just won't run.

So in our first example:

$ shopt -s nullglob
$ echo "Textfiles here:" *.txt
Textfiles here:

and the glob is gone.

find vs forloop

I have a set of files in dir1/ over 1600. I clearly see the difference in speeds of using find over forloop.

time find dir1/ -type f -exec mv {} dir2/ \;

real 0m1.194s
user 0m0.427s
sys 0m0.764s

time for i in `ls dir2/` ; do mv dir2/$i dir1/ ; done

real 0m1.687s
user 0m0.489s
sys 0m1.171s

There is lot of gain in terms of realtime as well as system time.

sort and uniq

tip to do sort and uniq on who command and see the no.of times a user has occurred:

who | awk '[print $1}' | sort | uniq -c

3 sriram
2 jadu
1 krishna

Reverse numeric sort order by skipping the heading.

sort -r -n +1

Create a character special device

I got a typical problem with my desktop while redirecting a script's errors to /dev/null .

sh:/dev/null : permission denied

when i did an ls on that , i have observed that /dev/null some how got touched as a regular file instead of a character special file.I did a google and found a command to create a character special after which my script started working.

Here is the tip:

root@localhost# mknod /dev/null c 1 3
root@localhost# chmod 666 /dev/null

That last command isn't a joke, that's really the permissions you
want, read and write for everybody .

Notes :

crw-rw-rw-    1 root     root       1,   3 May 11 21:08 /dev/null
^                                   ^    ^

this means /dev/null is a character special device (the leading 'c')
with major mode 1 and minor mode 3, what that means is except that it reads/writes 1 character at a time not blocks of data like disk drives.

To find the hung a process uptime

To know the time of process that has been running for some time since machine uptime on linux-2.4:

pd=<put your pid>;expr $(awk '{print $1}' FS=\. /proc/uptime) - $(awk '{printf ("%10d\n",$22/100)}' /proc/$pd/stat)

disk sizes consumed by a dir non-recursively

To find disk space used by files in a dir non-recursively.

du -h --max-depth=1

Bash Command line editing

Just want to add this link to my blog to study the bash command line editing options as and when i want to...

http://web.mit.edu/gnu/doc/html/features_7.html

/etc/fstab entry description

The following is an example of an fstab file on a typical Linux system:

# device name   mount point     fs-type      options                 dump-freq pass-num
LABEL=/         /               ext3         defaults                1 1
/dev/hda6       swap            swap         defaults                0 0
none            /dev/pts        devpts       gid=5,mode=620          0 0
none            /proc           proc         defaults                0 0
none            /dev/shm        tmpfs        defaults                0 0

# Removable media
/dev/cdrom      /mount/cdrom    udf,iso9660  noauto,owner,kudzu,ro   0 0
/dev/fd0        /mount/floppy   auto         noauto,owner,kudzu      0 0

# NTFS Windows XP partition
/dev/hda1       /mnt/WinXP      ntfs-3g      quiet,defaults,locale=en_US.utf8,umask=0    0 0

# Partition shared by Windows and Linux
/dev/hda7       /mnt/shared     vfat         umask=000              0 0

# mounting tmpfs
tmpfs           /mnt/tmpfschk   tmpfs        size=100m              0 0

# mounting cifs
//pingu/ashare  /store/pingu    cifs         credentials=/root/smbpass.txt 0 0

#mounting NFS
pingu:/store    /store          nfs          rw                     0 0

The columns are as follows:

1. The device name or other means of locating the partition or data source.
2. The mount point, where the data is to be attached to the filesystem.
3. The filesystem type, or the algorithm used to interpret the filesystem.
4. Options, including if the filesystem should be mounted at boot. (kudzu is an option specific to Red Hat and Fedora Core.)
5. dump-freq adjusts the archiving schedule for the partition (used by dump).
6. pass-num indicates the order in which the fsck utility will scan the partitions for errors when the computer powers on. 0 = none, 1 = first, 2 = next (and all others in order)

For information further go to : http://en.wikipedia.org/wiki/Fstab

Select loop in Shell

print 'Select your terminal type:'

PS3='terminal? '
select term in \
'Givalt VT100' \
'Tsoris VT220' \
'Shande VT320' \
'Vey VT520'
do
case $REPLY in
   1 ) TERM=gl35a ;;
   2 ) TERM=t2000 ;;
   3 ) TERM=s531 ;;
   4 ) TERM=vt99 ;;
   * ) print 'invalid.' ;;
esac
if [[ -n $term ]]; then
   print TERM is $TERM
   break
fi
done

Sample script I am using to login to ips:

#!/bin/bash

select ip in '172.43.180.36' '172.43.180.35' 120 121 142 225 230 224 170 112 141 220 208 203 206 75 76 77 232 ;
do
[[ $ip -ne "172.43.180.36" || $ip -ne "172.43.180.35" ]] && ssh root@172.43.0.$ip  || ssh root@$ip ;
trap "exit" 0 3 ;
done

O/p:
./select.sh
1) 172.43.180.36    6) 225            11) 141            16) 75
2) 172.43.180.35    7) 230            12) 220            17) 76
3) 120              8) 224            13) 208            18) 77
4) 121              9) 170            14) 203            19) 232
5) 142             10) 112            15) 206
#?1

Note:Selecting 1 logs into 1) ip address using ssh.

trap command to catch untimely script exits

trap syntax :
trap [arg] [signal]...

The trap waits for a signal sent to the shell, traps it, and then executes arg. After setting traps, typing trap with no args lists all commands associated with signals.

For example:

$ temp="/tmp/xyz$$"
$ trap "rm -f $temp; exit" 0 2 3 15
$ trap
0:rm -f /tmp/xyz18996; exit
2:rm -f /tmp/xyz18996; exit
3:rm -f /tmp/xyz18996; exit
15:rm -f /tmp/xyz18996; exit

In the first line, a temporary file temp is defined, whose name includes xyz and the process id number. The second line sets a trap to remove the file (without complaining if it doesn't exist yet or if the remove fails). It then exits the shell if the shell exits (0) or receives one of a certain set of signals (2, 3, 15), which could be given by names (INT, QUIT, TERM). After setting the trap, trap with no options, lists all traps. The exit in the trap is necessary because otherwise the trap would be like an interrupt routine, returning to execution of the script on receipt of a signal.

If arg is omitted or is -, all trap signals are reset to their original values. If signal is ERR then arg will be executed whenever a command has a nonzero exit code. The ERR trap is not inherited by functions.

If the signal is 0 or EXIT and the trap statement is executed inside the body of a function, then the command arg is executed after the function completes. If signal is EXIT for a trap set outside any function then the command arg is executed on exit from the shell.