Warning: Cannot modify header information - headers already sent by (output started at /data/web/virtuals/85063/virtual/www/domains/waldauf.org/lib/plugins/color/syntax.php:1) in /data/web/virtuals/85063/virtual/www/domains/waldauf.org/inc/actions.php on line 180
linux:gentoo:shell:shell_commands - WiKi

Useful shell commands

Arithmetic expressions

Shell can not count with tenth numbers.

All arithmetical operations are performed on integers. For example:

i=2

i=$(( (i + 4) / 3 ))

let i+=1

let i=$i+5

i=$(( i++ ))

let i++

let $i++

(( $i++ ))

or

i=`expr $i + 5`


Using bc command:

echo "333*4/3" | bc
 
echo "scale=3;333.333*4/3" | bc

Using bc command to achieve more accurately result and cut tenth tail of result via command sed:

echo "scale=3;333.333*4/3" | bc | sed 's/^\(.*\)\.[0-9]*/\1/'

Bash filetesting

-b filename Block special file
-c filename Special character file
-d directoryname Check for directory existence
-e filename Check for file existence
-f filename Check for regular file existence not a directory
-G filename Check if file exists and is owned by effective group ID
-g filename true if file exists and is set-group-id
-k filename Sticky bit
-L filename Symbolic link
-O filename True if file exists and is owned by the effective user id
-r filename Check if file is a readable
-S filename Check if file is socket
-s filename Check if file is nonzero size
-u filename Check if file set-ser-id bit is set
-w filename Check if file is writable
-x filename Check if file is executable

Remove ^M characters

^M characters is MS sign for end of line. But some LINUX/UNIX tools cannot work with it (e.g. KSH). There are many ways how to remove it:

  • Using col utility:
    cat <bad_filename> | col -b > <new_filename>
  • Vi(m) editor:
    :%s/^M//g
    
  • Using dos2unix comand:
    dos2unix <bad_filename> <new_filename>
    

Count number of files and directories

Return the number of files in the directory + 1 (the directory itself):

du -a | cut -d/ -f2 | sort | uniq -c | sort -nr


Finding number of files in a folder:

find . -type f | wc -l


Finding number of subfolders in a folder:

  • This will give number of subfolders+1 because current folder also gets included. Note that the folders inside subfolders are also included in count.
    find . -type d | wc -l
  • If you want to see only number of folders in current folder. Here output is numberof subfolders+1. NOTE - folders inside subfolders not considered.
    find . -maxdepth 1 -type d | wc -l


Grep doesn't work for regular files that have certain properties (suid,…) so here's a solution:

$ for t in files links directories; do echo `find . -type ${t:0:1} | wc -l` $t; done 2> /dev/null
 
8 files
 
1 links
 
11 directories

If you don't want to recurse directories (only count files in the current dir):

$ for t in files links directories; do echo `find . -maxdepth 1 -type ${t:0:1} | wc -l` $t; done 2> /dev/null

Debugging

There is several ways to debug your script. One of them is using bashdb utility (app-shells/bashdb) which is bash debugger. The debbuger contains all strandart features which you need: line breakpoints, stepping (in and out of function), attaching to running script and so on. You can read documentation of bashdb for more information.

You also can use log4sh utility (dev-libs/log4sh) which is a flexible framework for shell scripts.

Traditional ways of debbuging are:

  • Turn on debugging from command line.
    $ bash -x your_script.sh


  • Turn on debugging in your script.
    #!/bin/bash/
     
    set -x


  • You also can turn on debugging for some tail of your script.
    #!/bin/bash
     
    set -x
     
    ~
     
    ~
     
    set +x
     
    ~
     
    ~


  • If you need to control script name, line numbers and function you can do this by setting the following environment variables.
    export PS4='+${BASH_SOURCE}:${LINENO}:${FUNCNAME[0]} - [${SHLVL},${BASH_SUBSHELL},$?]: '
    export PS4='+(${BASH_SOURCE}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}() - [${SHLVL},${BASH_SUBSHELL},$?] }'


  • Logging of your script you can achieve using this function.
    _log() {
     
      if [ "$_DEBUG" == "true" ]; then
     
        echo 1>&2 "$@"
     
      fi
     
    }

    Logging is provided only in case if the variables _DEBUG is set to true. This allows you to toggle the printing of log messages depending on your needs. If you need to switch on log messages for whole script you don't need modify your script, you can set it on the command line:

    $ _DEBUG=true ./your_script.sh

Step by step debug

Add a trap like the following into your bash script, it has the effect you want, eg:

#!/usr/bin/env bash
set -x
trap read debug
 
< YOUR CODE HERE >


If your script is reading content from files, the above listed will not work. A workaround could look like the following example.:

#!/usr/bin/env bash
echo "Press CTRL+C to proceed."
trap "pkill -f 'sleep 1h'" INT
trap "set +x ; sleep 1h ; set -x" DEBUG
 
< YOUR CODE HERE >


If you simply want to wait a few seconds before proceeding to the next command in your script the following example could work for you. 'm adding set +x and set -x within the trap command to make the output more readable:

#!/usr/bin/env bash
trap "set +x; sleep 5; set -x" DEBUG
 
< YOUR CODE HERE >

Dynamic variables

I found this three function schemes:

  1. #!/bin/bash
     
    file=file
     
     
     
    for i in 1 2 3; do
     
      eval a=$file$i
     
      cat $a
     
      done
  1. #!/bin/bash
     
     
     
    a=foo
     
    b=bar
     
    foobar=barfoo
     
    barfoo=nonsense
     
    nonsense="sed 's/barfoo/barstool/g'"
     
     
     
    echo $a$b
     
    eval echo \$$a$b
     
    eval eval echo \\$\$$a$b
     
    eval echo "\$$a$b | $nonsense"
  1. #! /bin/bash 
     
    #set -xv 
     
     
     
    idents=(a b c); 
     
    count=${#idents[*]}; 
     
    for ((i=0; i < $count; i++)); do 
     
            ident=${idents[${i}]}; 
     
            eval "${ident}=${i}"; 
     
    done 
     
    echo "${idents[0]} = $a" 
     
    echo "${idents[1]} = $b" 
     
    echo "${idents[2]} = $c"

Enable/Disable multiple “echo statements”

The absolute easiest would be to insert the following line after the hashbang line:

echo() { :; }

When you want to re-enable, either delete the line or comment it out:

#echo() { :; }

If you're not using echo but printf, same strategy, i.e.:

printf() { :; }

If you absolutely need to actually echo/printf something, prepend the builtin statement, e.g.: builtin echo “This 'echo' will not be suppressed.”


This means that you can do a conditional output, e.g.:

echo () {
 
  [[ "$SOME_KIND_OF_FLAG" ]] && builtin echo $@
 
}

Set the SOME_KIND_OF_FLAG variable to something non-null, and the overridden echo function will behave like normal echo.

Another alternative would be to use echo for instrumenting (debugging), and printf for the outputs (e.g., for piping purposes). That way, no need for any FLAG. Just disable/enable the echo() { :; } line according to whether you want to instrument or not, respectively.

ID of the bash

$ echo $$
 
11015
 
$ echo $BASHPID
 
11015

IFS - Internal Filed Separator

Variable that splits fields by something other than the default whitespace (space, tab, newline), in this case - a comma ”,”:

Without setting IFS:

$ vals='/mnt /var/lib/vmware/Virtual Machines /dev /proc /sys /tmp /usr/portage /var/tmp'
 
$ for i in $vals; do echo $i; done
 
/mnt
 
/var/lib/vmware/Virtual
 
Machines
 
/dev
 
/proc
 
/sys
 
/tmp
 
/usr/portage
 
/var/tmp


With setting IFS:

$ IFS=$','
 
$ vals='/mnt,/var/lib/vmware/Virtual Machines,/dev,/proc,/sys,/tmp,/usr/portage,/var/tmp'
 
$ for i in $vals; do echo $i; done
 
/mnt
 
/var/lib/vmware/Virtual Machines
 
/dev
 
/proc
 
/sys
 
/tmp
 
/usr/portage
 
/var/tmp
 
$ unset IFS

Read file line after line

The simpliest solution

cat file |
( while read line; do
  echo process $line
done )

Or

while read line; do    
  echo $line
done < file 

Without variable

Code:

c=0; df -tfuseblk|while IFS= read -r;do 
  printf "line %d is -->%s<--\n" $((++c)) "$REPLY" 
done

Output:

$ c=0; df -g | while IFS= read -r;do
  printf "line %d is -->%s<--\n" $((++c)) "$REPLY"
done
line 1 is -->Filesystem           1K-blocks      Used Available Use% Mounted on<--
line 2 is -->/dev/sdb2             60926512  52040560   8885952  86% /media/USB_002<--
line 3 is -->/dev/sdb3             64484908  22401208  42083700  35% /media/USB_003<--
line 4 is -->/dev/sdb1             30876896   6648776  24228120  22% /media/USB_001<-

With variable and here string (ksh93, bash, zsh)

Code:

d=$(df -tfuseblk) c=0
while IFS= read -r; do
  printf "line %d is -->%s<--\n" $((++c)) "$REPLY"
done<<<"$d"

Output:

$ d=$(df -g) c=0
$ while IFS= read -r; do
while>   printf "line %d is -->%s<--\n" $((++c)) "$REPLY"
while> done<<<"$d"
line 1 is -->Filesystem           1K-blocks      Used Available Use% Mounted on<--
line 2 is -->/dev/sdb2             60926512  52040560   8885952  86% /media/USB_002<--
line 3 is -->/dev/sdb3             64484908  22401208  42083700  35% /media/USB_003<--
line 4 is -->/dev/sdb1             30876896   6648776  24228120  22% /media/USB_001<--

With process substitution (ksh93, bash, zsh)

Code:

c=0; while IFS= read -r; do
  printf "line %d is -->%s<--\n" $((++c)) "$REPLY"
done< <(df -g)

Output:

$ c=0; while IFS= read -r; do
while>   printf "line %d is -->%s<--\n" $((++c)) "$REPLY"
while> done< <(df -g)
line 1 is -->Filesystem           1K-blocks      Used Available Use% Mounted on<--
line 2 is -->/dev/sdb2             60926512  52040560   8885952  86% /media/USB_002<--
line 3 is -->/dev/sdb3             64484908  22401208  42083700  35% /media/USB_003<--
line 4 is -->/dev/sdb1             30876896   6648776  24228120  22% /media/USB_001<--

With using read

Code:

{ read 
  while read fs blks used free usepct mnt; do
    printf "dev: %s, size: %d\n" "$fs" "$blks"
  done 
} < <(df -g)

Output:

$ { read; while read fs blks used free usepct mnt; do
  printf "dev: %s, size: %d\n" "$fs" "$blks"
done;}< <(df -g)
dev: /dev/sdb2, size: 60926512
dev: /dev/sdb3, size: 64484908
dev: /dev/sdb1, size: 30876896

Read first 4 lines

#!/bin/bash
# Create dummy input
for i in $(seq 1 10) ; do echo $i >> input-file.txt ; done
 
# Create new file handle 5
exec 5< input-file.txt
 
# Now you can use "<&5" to read from this file
while read line1 <&5 ; do
        read line2 <&5
        read line3 <&5
        read line4 <&5
 
        echo "Four lines: $line1 $line2 $line3 $line4"
done
 
# Close file handle 5
exec 5<&-

Redirect output of script

  • Redirect a script output by command inside of that script:
    #!/bin/bash
     
    exec 1>/tmp/log
     
    log() {
     
     echo `date` : "$@"
     
    }
     
    log "Start"
     
    echo "Process"
     
    ls -l
     
    log "Done Logging"
  • Redirect into two files:
    #!/bin/bash
     
    #Objective : To redirect the stdout & stderr to two different files from within the script and without sacrificing the scripts output
     
     
     
    mkfifo /tmp/out.pipe /tmp/err.pipe
     
     
     
    exec 3>&1 4>&1
     
     
     
    tee /tmp/std.out < /tmp/out.pipe >&3 &
     
    pid_out=$!
     
    exec  1>/tmp/out.pipe
     
     
     
    tee /tmp/std.err < /tmp/err.pipe >&4 &
     
    pid_err=$!
     
    exec  2>/tmp/err.pipe
     
     
     
    log() {
     
     echo "`date` : $@ "
     
    }
     
     
     
    log "Starting the process"
     
    log "Listing directories now..."
     
    ls -l
     
    ls -l Log_to_Stderr
     
    exec 1>&3 3>&- 2>&4 4>&-
     
    wait $pid_out
     
    wait $pid_err
     
    rm /tmp/out.pipe /tmp/err.pipe
     
    log "End of Processing"

Script standard

#!/bin/bash
##
## ----------------------------------------------------------------------------
## "THE BEER-WARE LICENSE"
## <YOUR_EMAIL> wrote this file.  As long as you retain this notice
## you can do whatever you want with this stuff. If we meet some day, and you
## think this stuff is worth it, you can buy me a beer in return.
##                                                                 Jiri Valnoha
## ----------------------------------------------------------------------------
##
 
## VERSION: 20171005
 
 
set -o errexit
set -o nounset
set -o pipefail
 
 
SCRIPTNAME=${0##*/}
 
## Create temporary dir to work in which will be removed after script ends
tmpdir=$(mktemp -d --tmpdir $0_tmp.XXXXXX)
trap 'rm -rf "${tmpdir}"' EXIT
cd "${tmpdir}"

Strings

comparisons

$1=$2 equal
!= not equal
< less then
> greater then
-n s1 string s1 is not empty
-z s1 string s1 is empty

String variable operations

Length of $string

${#string}


Extract substring from $string at $position

${string:position}

Extract $length characters substring from $string at $position

${string:position:length}


Strip shortest match of $substring from front of $string

${string#substring}

Strip longest match of $substring from front of $string

${string##substring}

Strip shortest match of $substring from back of $string

${string%substring}

Strip longest match of $substring from back of $string

${string%%substring}


Replace first match of $substring with $replacement

${string/substring/replacement}

Replace all matches of $substring with $replacement

${string//substring/replacement}

If $substring matches front end of $string, substitute $replacement for $substring

${string/#substring/replacement}

If $substring matches back end of $string, substitute $replacement for $substring

${string/%substring/replacement}



Here is a way to isolate something useful from a large, even multi-line, string. As above, this method relies on enclosing a variable name in curly braces, then aplying a special operator to achieve a particular result.

Operator ”#” means “delete from the left, to the first case of what follows.”

$ x="This is my test string."
 
$ echo ${x#* }
 
is my test string.

Operator ”" means "delete from the left, to the last case of what follows." <code bash> $ x="This is my test string." $ echo ${x* }

string.

</code>

Operator ”%” means “delete from the right, to the first case of what follows.”

$ x="This is my test string."
 
$ echo ${x% *}
 
This is my test

Operator ”" means "delete from the right, to the last case of what follows." <code bash> $ x="This is my test string." $ echo ${x *}

This

</code>

Umount Busy Device

You can use fuser or lsof utilities.

Fuser

Fuser using:

fuser -m /mount/point

 /mount/point/:      29000c

And terminated procces using /mount/point:

fuser -km /mount/point

 /mount/point/:      29000c

Unix Timestamp

Unix time, or POSIX time, is a system for describing instants in time, defined as the number of seconds elapsed since midnight Coordinated Universal Time (UTC) of Thursday, January 1, 1970 (Unix times are defined, but negative, before that date), not counting leap seconds, which are declared by the International Earth Rotation and Reference Systems Service and are not predictable.

Convert Timestamp to Human time:

$ perl -e "print scalar(localtime(1327567923))"

Thu Jan 26 09:52:03 2012

Lsof

Using of lsof utility:

lsof /var

 COMMAND     PID     USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME

 syslogd     350     root    5w  VREG  222,5        0 440818 /var/adm/messages

 syslogd     350     root    6w  VREG  222,5   339098   6248 /var/log/syslog

 cron        353     root  cwd   VDIR  222,5      512 254550 /var -- atjobs


To view the Port associated with a daemon :

lsof -i -n -P | grep sendmail

 sendmail  31649    root    4u  IPv4 521738       TCP *:25 (LISTEN)
  • -i Lists IP sockets.
  • -n Do not resolve hostnames (no DNS).
  • -P Do not resolve port names (list port number instead of its name).

Commands

Command Description
history|awk '{a[$2]++ } END{for(i in a){print a[i] ” ” i}}'|sort -rn|head The most used commands.
cat /dev/null > /var/mail/waldauf Clear /var/mail/waldauf file.
case ${MYVAR} in ””) echo MYVAR is not set;; *) echo MYVAR is set;; esac Test if variable $MYVAR is set
[ -n “$MYVAR” ] && echo is set || echo is NOT set Test if variable $MYVAR is set
exec 1>/dev/null 2>/dev/null Turn off output or redirect to a file.
Navigation
Print/export
Toolbox