Bash Faq - temporary Edition

BASH Frequently Asked Questions

These are answers to frequently asked questions on channel #bash at http://irc.freenode.net

BASH is a BourneShell? compatible shell, which adds many new features to its ancestor. Most of them are available in the KornShell, too. If a question is not strictly shell specific, but rather related to Unix, it may be in the UnixFaq.

If you want to help, you can add new questions with answers here, or try to answer one of the BashOpenQuestions.

---

1. How can I read a file line-by-line?

    while read line
    do
        echo "$line"
    done < "$file"

The read command still modifies each line read, e.g. it removes all leading whitespace characters (blanks, tab characters). If that is not desired, the IFS (internal field separator) variable has to be cleared:

    OIFS=$IFS; IFS=
    while read line
    do
        echo "$line"
    done < "$file"
    IFS=$OIFS

As a feature, the read command concatenates lines that end with a backslash '\' character to one single line. To disable this feature, KornShell and BASH have read -r:

    OIFS=$IFS; IFS=
    while read -r line
    do
        echo "$line"
    done < "$file"
    IFS=$OIFS

Note that reading a file line by line this way is very slow for large files. Consider using e.g. awk? instead if you get performance problems.

---

2. How can I remove the last character of a line?

Using bash and ksh extended parameter substitution:

    var=${var%?}

Remember that ${var%foo} removes foo from the end, and ${var#foo} removes foo from the beginning, of var. As a mnemonic, # appears to the left of % on the keyboard (US keyboards, at least).

More portable, but slower:

    var=`expr "$var" : '\(.*\).'`

or (using sed):

    var=`echo "$var" | sed 's/.$//'`

---

3. How can I insert a blank character after each character?

    sed 's/./& /g'

Example:

    $ echo "testing" | sed 's/./& /g'
    t e s t i n g

---

4. How can I check whether a directory is empty or not?

Despite much dicussion, we never found a good answer to this. What most consider best so far is:

The following idea counts the number of entries in the specified directory (omitting ".." and ".")

    if [ `ls -a "$dir" | wc -l` -le 2 ]
    then
        echo empty directory: $dir
    fi

Most modern systems have an "ls -A" which explicitly omits "." and ".." from the directory listing:

    if [ -n "$(ls -A somedir)" ]
    then
        echo directory is non-empty
    fi

This can be shortened to:

    if [ "$(ls -A somedir)" ]
    then
        echo directory is non-empty
    fi

---

5. How can I convert all upper-case file names to lower case?

# tolower - convert file names to lower case

for file in *
do
    [ -f "$file" ] || continue                  # ignore non-existing names
    newname=$(echo "$file" | tr '[A-Z]' '[a-z]') # lower-case version of file name
    [ "$file" = "$newname" ] && continue        # nothing to do
    [ -f "$newname" ] && continue               # do not overwrite existing files
    mv "$file" "$newname"
done

Purists will insist on using

tr '[[:upper:]]' '[[:lower:]]'

in the above code, in case of non-ASCII (e.g. accented) letters in locales which have them.

This technique can also be used to replace all unwanted characters in a file name e.g. with '_' (underscore). The script is the same as above, only the "newname=..." line has changed.

# renamefiles - rename files whose name contain unusual characters
for file in *
do
    [ -f "$file" ] || continue                  # ignore non-existing names
    newname=$(echo "$file" | sed 's/[^a-zA-Z0-9_.]/_/g')
    [ "$file" = "$newname" ] && continue        # nothing to do
    [ -f "$newname" ] && continue               # do not overwrite existing files
    mv "$file" "$newname"
done

The character class in [] contains all allowed characters; modify it as needed.

---

6. How can I use a logical AND in a shell pattern (glob)?

That can be achieved through the !() extglob operator. You'll need extglob set. It can be checked with:

$ shopt extglob

and set with:

$ shopt -s extglob

To warm up, we'll move all files starting with foo AND not ending with .d to directory foo_thursday.d:

$ mv foo!(*.d) foo_thursday.d

For the general case:

Delete all files containing Pink_Floyd AND not containing The_Final_Cut:

$ rm !(!(*Pink_Floyd*)|*The_Final_Cut*)

By the way: these kind of patterns can be used with KornShell and KornShell93?, too. They don't have to be enabled there, but are the default patterns.

---

7. Is there a function to return the length of a string?

The fastest way, not requiring external programs (but usable only with BashShell? and KornShell):

${#varname}

or

expr "$varname" : '.*'
(expr prints the number of characters matching the pattern .*, which is the length of the string)

or

expr length "$varname"

(for a BSD/GNU version of expr. Do not use this, because it is not POSIX?).

---

8. How can I recursively search all files for a string?

On most recent systems (GNU/Linux/BSD), you would use grep -r pattern . to search all files from the current directory (.) downward.

You can use find if your grep lacks -r:

    $ find . -type f -exec grep -l "$search" '{}' \;

The {} characters will be replaced with the current file name.

This command is slower than it needs to be, because find will call grep with only one file name, resulting in many grep invocations (one per file). Since grep accepts multiple file names on the command line, find can be instrumented to call it with several file names at once:

    $ find . -type f -exec grep -l "$search" '{}' \+

The trailing '+' character instructs find to call grep with as many file names as possible, saving processes and resulting in faster execution. This example works for POSIX find, e.g. with Solaris.

GNU find uses a helper program called xargs for the same purpose:

    $ find . -type f -print0 | xargs -0 grep -l "$search"

The "-print0" / "-0" options ensure that any file name can be processed, even ones containing blanks, TAB characters, or new-lines.

90% of the time, all you need is: Have grep recurse and print the lines (GNU grep):

    $ grep -r "$search" .

Have grep recurse and print only the names (GNU grep):

    $ grep -r -l "$search" .

The find command can be used to run arbitrary commands on every file in a directory (including sub-directories). Replace grep with the command of your choice. The curly braces {} will be replaced with the current file name in the case above. (Note that they must be escaped in some shells, but not in bash)

---

9. My command line produces no output: tail -f logfile | grep 'ssh'

Most standard Unix commands buffer their output if used non-interactively. This means, that they don't write each character (or even each line) as they are ready, but collect a larger number (e.g. 4 kilobytes) before printing it. In the case above, the tail command buffers its output, and therefore grep only gets its input in e.g. 4K blocks.

Unfortunately there's no easy solution to this, because the behaviour of the standard programs would need to be changed.

Some programs provide special command line options for this purpose, e.g.

grep (e.g. GNU version 2.5.1)	--line-buffered
sed (e.g. GNU version 4.0.6)	-u,--unbuffered
awk (some GNU versions)	-W interactive
tcpdump, tethereal	-l

The expect package (http://expect.nist.gov/) has an unbuffer example program, which can help here. It disables buffering for the output of a program.

Example usage:

    $ unbuffer tail -f logfile | grep 'ssh'

There is another option when you have more control over the creation of the log file. If you would like to grep the real-time log of a text interface program which does buffered session logging by default (or you were using script to make a session log), then try this instead:

   $ program | tee -a program.log

   In another window:
   $ tail -f program.log | grep whatever

Apparently this works because tee produces unbuffered output. This has only been tested on GNU tee, YMMV.

---

10. How can I recreate a directory structure, without the files?

With the "cpio" program:

    cd "$srcdir"
    find . -type d -print | cpio -pdumv "$dstdir"

or with "tar":

    cd "$srcdir"
    tar cf - . | (cd "$dstdir"; tar xf -)

Note that the tar solution above actually reads all data (including the files) , writes it to standard output, reads it again from standard input, and writes it to the file system. The cpio solution only reads the directory names (not their content ), and copies them to their final destination. To do the equivalent with GNU tar (more intuitive, but less portable):

    cd "$srcdir"
    find . -type d -print | tar c --files-from - --no-recursion | tar x --directory "$dstdir"

this creates a list of directory names with find, non-recursively adds just the directories to an archive, and pipes it to a second tar instance to extract it at the target location.

---

11. How can I print the n'th line of a file?

The dirty (but not quick) way would be sed -n ${n}p "$file" but this reads the whole input file, even if you only wanted the third line.

The following sed command line reads a file printing nothing (-n). At line $n the command "p" is run, printing it, with a "q" afterwards: quit the program.

    sed -n "$n{p;q;}" "$file"

---

12. A program (e.g. a file manager) lets me define an external command that an argument will be appended to - but i need that argument somewhere in the middle...

    sh -c 'echo "$1"' -- hello

---

13. How can I concatenate two variables?

There is no concatenation operator for the shell. The variables are just written one after the other:

    var=$var1$var2

If the right-hand side contains whitespace characters, it needs to be quoted:

    var="$var1 - $var2"

CommandSubstitution can be used as well. The following line creates a log file name logname containing the current date, resulting in names like e.g. log.2004-07-26:

    logname="log".$(date +%Y-%m-%d)

---

14. How can I redirect the output of several commands at once?

Redirecting the standard output of a single command is as easy as

    date > file

To redirect standard error:

    date 2> file

To redirect both:

    date > file 2>&1

In a loop or other larger code structure:

    for i in $list; do
        echo "Now processing $i"
        # more stuff here...
    done > file 2>&1

However, this can become tedious if the output of many programs should be redirected. If all output of a script should go into a file (e.g. a log file), the exec command can be used:

    # redirect both standard output and standard error to "log.txt"
    exec > log.txt 2>&1
    # all output including stderr now goes into "log.txt"

Otherwise command grouping helps:

    {
        date
        # some other command
        echo done
    } > messages.log 2>&1

In this example, the output of all commands within the curly braces is redirected to the file messages.log.

---

15. How can I run a command on all files with the extention .gz?

Often a command already accepts several files as arguments, e.g.

    zcat *.gz

(One some systems, you would use gzcat instead of zcat. If neither is available, or if you don't care to play guessing games, just use gzip -dc instead.) If an explicit loop is desired, or if your command does not accept multiple filename arguments in one invocation, the for loop can be used:

    for file in *.gz
    do
        echo "$file"
        # do something with "$file"
    done

To do it recursively, you should use a loop, plus the find command:

    while read file; do
        echo "$file"
        # do something with "$file"
    done < <(find . -name '*.gz' -print)

For more hints in this direction, see FAQ #20, below. To see why the find command comes after the loop instead of before it, see FAQ #24.

---

16. How can I remove a file name extension from a string, e.g. file.tar to file?

The easiest (and fastest) way is to use the following:

    $ name="file.tar"
    $ echo "${name%.tar}"
    file

The ${var%pattern} syntax removes the pattern from the end of the variable. ${var#pattern} would remove pattern from the start of the string. This could be used to rename all files from "*.doc" to "*.txt":

    for file in *.doc
    do
        mv "$file" "${file%.doc}".txt
    done

There's more to ParameterSubstitution?, e.g. =${var%%pattern}, ${var##pattern}, ${var//old/new}.

Note that this extended form of ParameterSubstitution? works with BASH, KornShell, KornShell93?, but not with the older BourneShell?. If the code needs to be portable to that shell as well, sed could be used to remove the filename extension part:

    for file in *.doc
    do
        base=`echo "$file" | sed 's/\.[^.]*$//'`    # remove everything starting with last '.'
        mv "$file" "$base".txt
    done

Finally, some GNU/Linux/BSD systems offer a rename command. There are multiple different rename commands out there with contradictory syntaxes. Consult your man pages to see which one you have (if any).

---

17. How can I group expressions, e.g. (A AND B) OR C?

The TestCommand? [ uses parentheses () for expression grouping. Given that "AND" is "-a", and "OR" is "-o", the following expression

    (0<n AND n<=10) OR n=-1

can be written as follows:

    if [ \( $n -gt 0 -a $n -le 10 \) -o $n -eq -1 ]
    then
        echo "0 < $n <= 10, or $n=-1"
    else
        echo "invalid number: $n"
    fi

Note that the parentheses have to be quoted: \(, '(' or "(".

BASH and KornShell have different, more powerful comparison

commands with slightly different (easier) quoting

• ((? for arithmetic expressions, and
• [[? for string (and file) expressions.

Examples:

    if (( (n>0 && n<10) || n == -1 ))
    then echo "0 < $n < 10, or n==-1"
    fi

or

    if [[ ( -f $localconfig && -f $globalconfig ) || -n $noconfig ]]
    then echo "configuration ok (or not used)"
    fi

Note that the distinction between numeric and string comparisons is strict. Consider the following example:

    n=3
    if [[ n>0 && n<10 ]]
    then echo "$n is between 0 and 10"
    else echo "ERROR: invalid number: $n"
    fi

The output will be "ERROR: ....", because in a string comparision "3" is bigger than "10", because "3" already comes after "1", and the next character "0" is not considered. Changing the square brackets to double parentheses (( makes the example work as expected.

---

18. How can I use numbers with leading zeros in a loop, e.g. 01, 02?

As always, there are different ways to solve the problem, each with its own advantages and disadvantages.

If there are not many numbers, BraceExpansion? can be used:

    for i in 0{1,2,3,4,5,6,7,8,9} 10
    do
        echo $i
    done

Output:

00
01
02
03
[...]

This gets tedious for large sequences, but there are other ways, too. If the command seq is available, you can use it as follows:

    seq -w 1 10

or, for arbitrary numbers of leading zeros (here: 3):

    seq -f "%03g" 1 10

If you have the printf command (which is a Bash builtin), it can be used to format a number, too:

    for ((i=1; i<=10; i++))
    do
        printf "%02d\n" "$i"
    done

The KornShell and KornShell93? have the typeset command to specify the number of leading zeros:

    $ typeset -Z3 i=4
    $ echo $i
    004

Finally, the following example works with any BourneShell? derived shell to zero-pad each line to three bytes:

i=0
while test $i -le 10
do
    echo "00$i"
    i=`expr $i + 1`
done |
    sed 's/.*\(...\)$/\1/g'

In this example, the number of '.' inside the parentheses in the sed statement determins how many total bytes from the echo command (at the end of each line) will be kept and printed.

One more addendum: in Bash 3, you can use:

printf "%03d" {1..300}

Which is slightly easier in some cases.

---

19. How can I split a file into line ranges, e.g. lines 1-10, 11-20, 21-30?

Some Unix systems provide the split utility for this purpose:

    split --lines 10 --numeric-suffixes input.txt output-

For more flexibility you can use "sed":

The sed command can print e.g. the line number range 1-10:

    sed -n '1,10p'

This stops sed from printing each line (-n). Instead it only processes the lines in the range 1-10 ("1,10"), and prints them ("p"). sed still reads the input until the end, although we are only interested in lines 1 though 10. We can speed this up by making sed terminate immediately after printing line 10:

    sed -n -e '1,10p' -e '10q'

Now the command will quit after reading line 10 ("10q"). The =-e arguments indicate a script (instead of a file name). The same can be written a little shorter:

    sed -n '1,10p;10q'

We can now use this to print an arbitrary range of a file (specified by line number):

file=/etc/passwd
range=10
firstline=1
maxlines=$(wc -l < "$file") # count number of lines
while (($firstline < $maxlines))
do
    ((lastline=$firstline+$range+1))
    sed -n -e "$firstline,${lastline}p" -e "${lastline}q" "$file"
    ((firstline=$firstline+$range+1))
done

This example uses BASH and KornShell ArithmeticExpressions?, which older Bourne Shells? do not have. In that case the following example should be used instead:

file=/etc/passwd
range=10
firstline=1
maxlines=`wc -l < "$file"` # count line numbers
while [ $firstline -le $maxlines ]
do
    lastline=`expr $firstline + $range + 1`
    sed -n -e "$firstline,${lastline}p" -e "${lastline}q" "$file"
    firstline=`expr $lastline + 1`
done

---

20. How can I find and deal with files containing newlines, spaces or both?

The preferred method is still to use

    find ... -exec command {} \;

or, if you need to handle filenames en masse:

    find ... -print0 | xargs -0 command

for GNU find/xargs, or (POSIX find):

    find ... -exec command {} +

Use that unless you really can't.

Another way to deal with files with spaces in their names is to use the shell's filename expansion ("globbing"). This has the disadvantage of not working recursively (except with zsh's extensions), but if you just need to process all the files in a single directory, it works fantastically well.

This example changes all the *.mp3 files in the current directory to use underscores in their names instead of spaces. (But it will not work in the original BourneShell?.)

for file in *.mp3; do
    mv "$file" "${file// /_}"
done

You could do the same thing for all files (regardless of extension) by using

for file in *\ *; do

instead of *.mp3.

Another way to handle filenames recursively involes using the -print0 option of find (a GNU/BSD extension), together with bash's -d option for read:

unset a i
while read -d $'\0' file; do
  a[i++]="$file"        # or however you want to process each file
done < <(find /tmp -type f -print0)

The preceding example reads all the files under /tmp (recursively) into an array, even if they have newlines or other whitespace in their names, by forcing read to use the NUL byte (\0) as its word delimiter. Since NUL is not a valid byte in Unix filenames, this is the safest approach besides using find -exec.

Finally, here's a helpful tool (which is probably not as safe as the \0 technique):

    Syntax : find_all [path] [command] <maxdepth>

    export IFS=" "
    [ -z "$3" ] && set -- "$1" "$2" 1
    FILES=`find "$1" -maxdepth "$3" -type f -printf "\"%p\" "`
    eval FILES=($FILES)
    for ((I=0; I < ${#FILES[@]}; I++))
    do
        eval "$2 \"${FILES[I]}\""
    done
    unset IFS

This script can recursively search for files with newlines and/or spaces in them. Usually, the "find -print0 | xargs -0" technique is used, but that isn't fit for all purposes. For example, the execution of multiple commands on each element of the result is impossible with xargs and would be quite tedious with :

    find -exec cmd1 {} \; -exec cmd2 {} \; -exec cmd3 {} \; ... etc

Also it would rule out using if ... ; then ; else ; fi type constructs.

Fixme : Bash could crash if the number of files processed, grows very large. This is because the script fills an array with the result of the found files. I would wrather use while and read, because that would process the input stream line by line, and not claim as many memory resources as a for loop.

The default IFS contains '<space>\n\t', but this would influence a newline passed to eval. Therefore we set it to space, and unset it at the end of the script, for clarity.

    export IFS=" "
    ...
    unset IFS

The most important part of the script is

    FILES=`find "$1" -maxdepth "$3" -type f -printf "\"%p\" "`

especially the ' -printf "\"%p\" " '.
If we dissect it, it has these elements :

• \" --> escaped quote, so the result per array element will be "filename w space and?newline" instead of : filename w space and?newline, which would be seen as five different array elements.
• %p --> the filename, see the manual for find, for more about these find-specific specifiers.
• ' ' --> the space at the end, in between \" " will seperate each result of find with a space, like "a" "b" .. etc. so that it is fit to be an array.

So why does this line:

    eval FILES=($FILES)

use 'eval'?

Well, this seems kind of odd, but in order for bash to understand that $FILES is a group of elements, and that each element begins and ends with " and not with a space, this line needs to be executed as if it was a command line, so bash sees all "..." items as one element. A 'parameter' if you will.

This line could have been merged with the line above it in the script like : eval FILES=(`find ...`), but I did not do that so I could discuss this line seperately.

Next is to iterate through all the elements of the array like so:

    for ((I=0; I < ${#FILES[@]}; I++))
    do
        eval "$2 \"${FILES[I]}\""
    done

The variable $I starts at 0, should be smaller than the number of elements in the array $FILES, and $I should be incremented by 1 each iteration. Every iteration, we then execute eval. The outer quotes mark the string eval parses and protect any spaces. $2 is the parameter specifying the command to be executed, and it's parameter, \"${FILES[I]}\" evaluates to "filename", causing it to be passed on to $2 by bash as one parameter, without problems caused by spaces or newlines.

---

21. How can I replace a string with another string in all files?

sed is a good command to replace strings, e.g.

    sed 's/olddomain\.com/newdomain\.com/g' input > output

To replace a string in all files of the current directory:

    for i in *; do
        sed 's/old/new/g' "$i" > atempfile && mv atempfile "$i"
    done

GNU sed 4.x (but no other version of sed) has a special -i flag which makes the temp file unnecessary:

   for i in *; do
      sed -i 's/old/new/g' "$i"
   done

Those of you who have perl 5 can accomplish the same thing using this code:

    perl -pi -e 's/old/new/g' *

Finally, here's a script that some people may find useful:

    :
    # chtext - change text in several files

    # neither string may contain '|' unquoted
    old='olddomain\.com'
    new='newdomain\.com'

    # if no files were specified on the command line, use all files:
    [ $# -lt 1 ] && set -- *

    for file
    do
        [ -f "$file" ] || continue # do not process e.g. directories
        [ -r "$file" ] || continue # cannot read file - ignore it
        # Replace string, write output to temporary file. Terminate script in case of errors
        sed "s|$old|$new|g" "$file" > "$file"-new || exit
        # If the file has changed, overwrite original file. Otherwise remove copy
        if cmp "$file" "$file"-new >/dev/null 2>&1
        then rm "$file"-new              # file nas not changed
        else mv "$file"-new "$file"      # file has changed: overwrite original file
        fi
    done

If the code above is put into a script file (e.g. chtext), the resulting script can be used to change a text e.g. in all HTML files of the current and all subdirectories:

    find . -type f -name '*.html' -exec chtext {} \;

Many optimizations are possible:

• use another sed separator character than '|', e.g. ^A (ASCII 1)
• some implementations of sed (e.g. GNU sed) have an "-i" option that can change a file in-place; no temporary file is necessary in that case
• the find command above could use either xargs or the built-in xargs of POSIX find

Note: set -- * in the code above is safe with respect to files whose names contain spaces. The expansion of * by set is the same as the expansion done by for, and filenames will be preserved properly as individual parameters, and not broken into words on whitespace.

A more sophisticated example of chtext is here: http://www.shelldorado.com/scripts/cmds/chtext

---

22. How can I calculate with floating point numbers instead of just integers?

BASH does not have built-in floating point arithmetic:

    $ echo $((10/3))
    3

For better precision, an external program must be used, e.g. bc, awk or dc:

    $ echo "scale=3; 10/3" | bc
    3.333

The "scale=3" command notifies bc that three digits of precision after the decimal point are required.

awk can be used, for calculations, too:

    $ awk 'BEGIN {printf "%.3f\n", 10 / 3}' /dev/null
    3.333

There is a subtle but important difference between the bc and the awk solution here: bc reads commands and expressions from standard input. awk on the other hand evaluates the expression as part of the program. Expressions on standard input are not evaluated, i.e. echo 10/3 | awk '{print $0}' will print 10/3 instead of the evaluated result of the expression.

This explains why the example uses /dev/null as an input file for awk: the program evaluates the BEGIN action, evaluating the expression and printing the result. Afterwards the work is already done: it reads its standard input, gets an end-of-file indication, and terminates. If no file had been specified, awk would wait for data on standard input.

Newer versions of KornShell93? have built-in floating point arithmetic, together with mathematical functions like sin() or cos() .

---

23. How do I append a string to the contents of a variable?

The shell doesn't have a string concatenation operator like Java ("+") or Perl ("."). The following example shows how to append the string ".2004-08-15" to the contents of the shell variable filename:

    filename="$filename.2004-08-15"

If the variable name and the string to append could be confused, the variable name can be enclosed in braces, e.g.

    filename="${filename}old"

instead of "filename=$filenameold"

---

24. I set variables in a loop. Why do they suddenly disappear after the loop terminates?

The following command always prints "total number of lines: 0", although the variable linecnt has a larger value in the while loop:

    linecnt=0
    cat /etc/passwd | while read line
    do
        linecnt=`expr $linecnt + 1`
    done
    echo "total number of lines: $linecnt"

The reason for this surprising behaviour is that a while/for/until loop runs in a subshell when its input or output is redirected from a pipeline. For the while loop above, a new subshell with its own copy of the variable linecnt is created (initial value, taken from the parent shell: "0"). This copy then is used for counting. When the while loop is finished, the subshell copy is discarded, and the original variable linecnt of the parent (whose value has not changed) is used in the = echo= command.

It's hard to tell when shell would create a new process for a loop:

• BourneShell? creates it when the input or output is redirected, either by using a pipeline or by a redirection operator ('<', '>').
• BASH creates a new process only if the loop is part of a pipeline
• ksh creates it only if the loop is part of a pipeline, but not if the loop is the last part of it.

To solve this, either use a method that works without a subshell (shown below), or make sure you do all processing inside that subshell (a bit of a kludge, but easier to work with):

    linecnt=0
    cat /etc/passwd |
    (
        while read line ; do
                linecnt="$((linecnt+1))"
        done
        echo "total number of lines: $linecnt"
    )

To avoid the subshell completely (not easily possible if the other part of the pipe is a command!), use redirection, which does not have this problem at least for bash and ksh (but still for BourneShell?):

    linecnt=0
    while read line ; do
        linecnt="$((linecnt+1))"
   done < /etc/passwd
   echo "total number of lines: $linecnt"

A portable and common work-around is to redirect the input of the "read" command using exec:

    linecnt=0
    exec < /etc/passwd    # redirect standard input from the file /etc/passwd
    while read line       # "read" gets its input from the file /etc/passwd
    do
        linecnt=`expr $linecnt + 1`
    done
    echo "total number of lines: $linecnt"

This works as expected, and prints a line count for the file /etc/passwd. But the input is redirected from that file permanently. What if we need to read the original standard input sometime later again? In that case we have to save a copy of the original standard input, which we later can restore:

    exec 3<&0    # save original standard input file descriptor "0" as file descriptor "3"
    exec 0</etc/passwd    # redirect standard input from the file /etc/passwd

    linecnt=0
    while read line       # "read" gets its input from the file /etc/passwd
    do
        linecnt=`expr $linecnt + 1`
    done

    exec 0<&3   # restore saved standard input (fd 0) from file descriptor "3"
    exec 3<&-   # close the no longer needed file descriptor "3"

    echo "total number of lines: $linecnt"

Subsequent exec commands can be combined into one line, which is interpreted left-to-right:

    exec 3<&0
    exec 0</etc/passwd
    ...read redirected standard input...
    exec 0<&3
    exec 3<&-

is equivalent to

    exec 3<&0 0</etc/passwd
    ...read redirected standard input...
    exec 0<&3 3<&-

---

25. How can I access positional parameters after $9?

Use ${10} instead of $10. This works for BASH and KornShell, but not for older BourneShell? implementations. Another way to access arbitrary positional parameters after $9 is to use "for", e.g. to get the last parameter:

    for last
    do
        : # nothing
    done

    echo "last argument is: $last"

To get an argument by number, we can use a counter:

    n=12        # This is the number of the argument we are interested in
    i=1
    for arg
    do
        if [ $i -eq $n ]
        then
            argn=arg
            break
        fi
        i=`expr $i + 1`
    done
    echo "argument number $n is: $argn"

This has the advantage of not "consuming" the arguments. If this is no problem, the "shift" command discards the first positional arguments:

    shift 11

    echo "the 12th argument is: $1"

Although direct access to any positional argument is possible this way, it's hardly needed. The common way is to use getopts(3) to process command line options (e.g. "-l", or "-o filename"), and then use either "for" or "while" to process all arguments in turn. An explanation of how to process command line arguments is available here: http://www.shelldorado.com/goodcoding/cmdargs.html

---

26. How can I randomize|shuffle the order of lines in a file?

    randomize(){
        while read l ; do echo "0$RANDOM $l" ; done |
        sort -n |
        cut -d" " -f2-
    }

Note: the leading 0 is to make sure it doesnt break if the shell doesnt support $RANDOM, which is supported by bash, ksh, ksh93 and POSIX shell, but not BourneShell?.

The same idea (printing random numbers in front of a line, and sorting the lines on that column) using other programs:

    awk '
        BEGIN { srand('"$RANDOM"') }
        { print rand() "\t" $0 }
    ' |
    sort -n |    # Sort numerically on first (random number) column
    cut -f2-     # Remove sorting column

This is faster then the previous solution, but will not work for very old AWK implementations (try "nawk", or "gawk", if available).

---

27. How can two processes communicate using named pipes (fifos)?

NamedPipes?, also known as FIFOs ("First In First Out") are well suited for process communication. The advantage over using files as a means of communication is, that processes are synchronized by pipes: a process writing to a pipe blocks if there is no reader, and a process reading from a pipe blocks if there is no writer. Here is a small example of a server process communicating with a client process. The server sends commands to the client, and the client acknowledges each command:

Server

#! /bin/sh
# server - communication example

# Create a FIFO. Some systems don't have a "mkfifo" command, but use
# "mknod pipe p" instead

mkfifo pipe

while sleep 1
do
    echo "server: sending GO to client"

    # The following command will cause this process to block (wait)
    # until another process reads from the pipe
    echo GO > pipe

    # A client read the string! Now wait for its answer. The "read"
    # command again will block until the client wrote something
    read answer < pipe

    # The client answered!
    echo "server: got answer: $answer"
done

Client

#! /bin/sh
# client

# We cannot start working until the server has created the pipe...
until [ -p pipe ]
do
    sleep 1;    # wait for server to create pipe
done

# Now communicate...

while sleep 1
do
    echo "client: waiting for data"

    # Wait until the server sends us one line of data:
    read data < pipe

    # Received one line!
    echo "client: read <$data>, answering"

    # Now acknowledge that we got the data. This command
    # again will block until the server read it.
    echo ACK > pipe
done

Write both examples to files server and client respectively, and start them concurrently to see it working:

    $ chmod +x server client
    $ server & client &
    server: sending GO to client
    client: waiting for data
    client: read <GO>, answering
    server: got answer: ACK
    server: sending GO to client
    client: waiting for data
    client: read <GO>, answering
    server: got answer: ACK
    server: sending GO to client
    client: waiting for data
    [...]

---

28. How do I determine the location of my script? I want to read some config files from the same place.

This is a complex question because there's no single right answer to it. Even worse: it's not possible to find the location reliably in 100% of all cases. All ways of finding a script's location depend on the name of the script, as seen in the predefined variable $0. But providing the script name in $0 is only a (very common) convention, not a requirement.

The simplest answer is "in some shells, $0 is always an absolute path, even if you invoke the script using a relative path, or no path at all", e.g. in BASH. But this isn't reliable across shells; some of them return the actual command typed in by the user instead of the fully qualified path. In those cases, if all you want is the fully qualified version of $0, you can use something like this (POSIX, non-Bourne):

  [[ $0 = /* ]] && echo $0 || echo $PWD/$0

Or the BourneShell? version:

  case $0 in /*) echo $0;; *) echo `pwd`/$0;; esac

However, this approach has some major drawbacks. The most important is, that the script name (as seen in $0) may not be relative to the current working directory, but relative to a directory from the program search path $PATH (this is often seen with KornShell).

Another drawback is that there is really no guarantee that your script is still in the same place it was when it first started executing. Suppose your script is loaded from a temporary file which is then unlinked immediately... your script might not even exist on disk any more! The script could also have been moved to a different location while it was executing. Or (and this is most likely by far...) there might be multiple links to the script from multiple locations, one of them being a simple symlink from a common PATH directory like /usr/local/bin, which is how it's being invoked. Your script might be in /opt/foobar/bin/script but the naive approach of reading $0 won't tell you that.

So if the own name in $0 is a relative one, i.e. does not start with '/', we can still try to search the script like the shell would have done: in all directories from $PATH.

The following script shows how this could be done:

    myname=$0
    if [ -s "$myname" ] && [ -x "$myname" ]
    then                   # $myname is already a valid file name
        mypath=$myname
    else
        case "$myname" in
        /*) exit 1;;             # absolute path - do not search PATH
        *)
            # Search all directories from the PATH variable. Take
            # care to interpret leading and trailing ":" as meaning
            # the current directory; the same is true for "::" within
            # the PATH.

            for dir in `echo "$PATH" | sed 's/^:/.:/g;s/::/:.:/g;s/:$/:./;s/:/ /g'`
            do
                [ -f "$dir/$myname" ] || continue # no file
                [ -x "$dir/$myname" ] || continue # not executable
                mypath=$dir/$myname
                break           # only return first matching file
            done
            ;;
        esac
    fi

    if [ -f "$mypath" ]
    then
        : # echo >&2 "DEBUG: mypath=<$mypath>"
    else
        echo >&2 "cannot find full path name: $myname"
        exit 1
    fi

    echo >&2 "path of this script: $mypath"

Note that $mypath is not necessarily an absolute path name. It still can contain relative parts like ../bin/myscript.

Generally storing data files in the same directory as their scripts is a bad practice. The Unix file system layout assumes that files in one place (e.g. /bin) are executable programs, while files in another place (e.g. /etc) are data files. (Let's ignore legacy Unix systems with programs in /etc for the moment, shall we....)

It really makes the most sense to keep your script's configuration in a single, static location such as $SCRIPTROOT/etc/foobar.conf. If you need to define multiple configuration files, then you can have a directory (say, /var/lib/foobar or /usr/local/lib/foobar), and read that directory's location from a variable in /etc/foobar.conf. If you don't even want that much to be hard-coded, you could pass the location of foobar.conf as a parameter to the script. If you need the script to assume certain default in the absence of /etc/foobar.conf, you can put defaults in the script itself, and/or fall back to something like $HOME/.foobar.conf if /etc/foobar.conf is missing. (This depends on what your script does. In some cases, it may make more sense to abort gracefully.)

---

29. How can I display value of a symbolic link on standard output?

readlink displays the value of a symbolic link. Newer versions of the bash have implemented this feature, but not older versions.

readlink /bin/sh
bash

For older versions add this function to the top of your script:

readlink() {
    local path=$1 ll

    if [ -L "$path" ]; then
        ll="$(LC_ALL=C ls -l "$path" 2> /dev/null)" &&
        echo "${ll/* -> }"
    else
        return 1
    fi
}

---

30. How can I rename all my *.foo files to *.bar?

Some GNU/Linux distributions have a rename command, which you can use for this purpose; however, the syntax differs from one distribution to the next, so it's not a portable answer.

You can do it in POSIX shells like this:

for f in *.foo; do mv "$f" "${f%.foo}.bar"; done

This invokes the external command "mv" once for each file, so it may not be as efficient as some of the "rename" implementations.

If you want to do it recursively, then it becomes much more challenging. This example works (in bash) as long as no files have newlines in their names:

find . -name '*.foo' -print | while IFS=$'\n' read -r f; do
  mv "$f" "${f%.foo}.bar"
done

Another common form of this question is "How do I rename all my MP3 files so that they have underscores instead of spaces?" You can use this:

for f in *\ *.mp3; do mv "$f" "${f// /_}"; done

---

31. What is the difference between the old and the new test command?

"[" ("test" command) and "[[" ("new test" command) are both used to evaluate expressions. Some examples:

    if [ -z "$variable" ]
    then
        echo "variable is empty!"
    fi

    if [ -f "$filename" ]
    then
        echo "not a valid, existing file name: $filename"
    fi

and

    if [[ -e $file ]]
    then
        echo "directory entry does not exist: $file"
    fi

    if [[ $file0 -nt $file1 ]]
    then
        echo "file $file0 is newer than $file1"
    fi

To cut a long story short: "[" implements the old, portable syntax of the command. Although all modern shells have built-in implementations, there usually still is an external executable of that name, e.g. /bin/[. "[[" is a new improved version of it, which is a keyword, no program. This has benefical effects on the ease of use, see below. "[[" is understood by KornShell, BASH (e.g. 2.03), KornShell93, POSIX shell, but not by standard BourneShell?.

Although "[" and "[[" have much in common, and share many expression operators like "-f", "-s", "-n", "-z", there are some notable differences. Here is a comparison list:

Feature	new test "[["	old test "["	Example
string comparison	>	(not available)	-
	<	(not available)	-
	== (or =)	=	-
	!=	!=	-
expression grouping	&&	-a	[[ -n $var && -f $var ]] && echo "$var is a file"
	||	-o	-
Pattern matching	=	(not available)	[[ $name = a* ]] || echo "name does not start with an 'a': $name"

Special primitives that "[[" is defined to have, but "[" may be lacking (depending on the implementation):

Description	Primitive	Example
entry (file or directory) exists	-e	[[ -e $config ]] && echo "config file exists: $config"
file is newer/older than other file	-nt / -ot	[[ $file0 -nt $file1 ]] && echo "$file0 is newer than $file1"
two files are the same	-ef	[[ $input -ef $output ]] && { echo "will not overwrite input file: $input"; exit 1; }
negation	!	~

But there are more subtle differences.

• No field splitting will be done for "[[" (and therefore many arguments need not to be quoted)

        file="file name"
        [[ -f $file ]] && echo "$file is a file"

will work even though $file is not quoted and contains whitespace. With "[" the variable needs to be quoted:

        file="file name"
        [ -f "$file" ] && echo "$file is a file"

This makes "[[" easier to use and less error prone.

• No file name generation will be done for "[[". Therefore the following line tries to match the contents of the variable $path with the pattern /* :

        [[ $path = /* ]] && echo "$path starts with a forward slash /: $path"

The next command most likely will result in an error, because /* is subject to file name generation:

        [ $path = /* ] && echo "this does not work"

"[[" is strictly used for strings and files. If you want to compare numbers, use ArithmethicEvaluation? ((expression)), e.g.

        i=0
        while ((i<10))
        do
            echo $i
            ((i=$i+1))
        done

When should the new test command "[[" be used, and when the old one "["? If portability to the BourneShell? is a concern, the old syntax should be used. If on the other hand the script requires BASH or KornShell, the new syntax could be preferable.

---

32. How can I redirect the output of 'time' to a variable or file?

The reason that 'time' needs special care for redirecting its output is one of those mysteries of the universe. The answer will probably be solved around the same time we find dark matter.

• File Redirection

     bash -c "time ls" > /path/to/foo 2>&1
     ( time ls ) > /path/to/foo 2>&1
     { time ls; } > /path/to/foo 2>&1

• Variable Redirection

     foo=$( bash -c "time ls" 2>&1 )
     foo=$( ( time ls ) 2>&1 )
     foo=$( { time ls; } 2>&1 )

Note: Using 'bash -c' and ( ) creates a subshell, using { } does not. Do with that as you wish.

---

33. How can I find a process id for a process given its name?

Usually a process is referred to using its process id (PID), and the ps command can display the information for any process given its process id, e.g.

    $ echo $$         # my process id
    21796
    $ ps -p 21796
    PID TTY          TIME CMD
    21796 pts/5    00:00:00 ksh

But frequently the process id for a process is not known, but only its name. Some operating systems, e.g. Solaris or some versions of Linux have a dedicated command to search a process given its name, called "pgrep":

    $ pgrep init
    1

Often there is an even more specialized program available to not just find the process id of a process given its name, but also to send a signal to it:

    $ pkill myprocess

If these programs are not available, a user can search the output of the ps(1) command using "grep".

The major problem when grepping the ps output is always, that grep matches it's own ps entry (try: ps aux | grep init). To avoid this, there are several ways:

• Using grep -v at the end

     ps aux | grep name | grep -v grep

will throw away all lines containing "grep" from the output. Disadvantage: You always have the exit state of the grep -v, so you can't e.g. check if a specific process exists.

• Using grep -v in the middle

     ps aux | grep -v grep | grep name

This does exactly the same, beside that the exit state of "grep name" is acessible and a representation for "name is a process in ps" or "name is not a process in ps". It still has the disadvantage to start a new process (grep -v).

• Using [] in grep

     ps aux | grep [n]ame

This spawns only the needed grep-process. The trick is to use the []-character class (regular expressions). To put only one character in a character group normally makes no sense at all, because a "[c]" will always be a "c". In this case, it's the same. grep [n]ame searches for "name". But as grep's own process list entry is what you executed ("grep [n]ame") and not "grep name", it will not match itself.

===BEGIN greycat rant===

Most of the time when someone asks a question like this, it's because they want to manage a long-running daemon using primitive shell scripting techniques. Common variants are "How can I get the PID of my foobard process.... so I can start one if it's not already running" or "How can I get the PID of my foobard proecss... because I want to prevent the foobard script from running if foobard is already active." Both of these questions will lead to seriously flawed production systems.

If what you really want is to restart your daemon whenever it dies, just do this:

#!/bin/sh
while true; do
   mydaemon --in-the-foreground
done

where --in-the-foreground is whatever switch, if any, you must give to the daemon to PREVENT IT from automatically backgrounding itself. (Often, -d does this and has the additional benefit of running the daemon with increased verbosity.) Self-daemonizing programs may or may not be the target of a future greycat rant....

If that's too simplistic, look into daemontools or runit, which are programs for managing services.

If what you really want is to prevent multiple instances of your program from running, then the only sure way to do that is by using a lockfile. And you can't lock a file from a shell script (unfortunately) -- the locking mechanism must be built into the daemon itself. So get out your source code and your favorite editor and start fixing.

===END greycat rant===

---

34. Can I do a spinner in Bash?

Sure.

    i=1
    sp="/-|-\|"
    echo -n ' '
    while true
    do
        echo -en "\b${sp:i++%${#sp}:1}"
    done

You can also use \r instead of \b. You can use pretty much any character sequence you want as well.

---

35. How can I handle command-line arguments to my script easily?

Well, that depends a great deal on what you want to do with them. Here's a general template that might help for the simple cases:

    while [[ $1 == -* ]]; do
        case "$1" in
          -h|--help) show_help; exit 0;;
          -v) verbose=1; shift;;
          -f) output_file=$2; shift 2;;
        esac
    done
    # Now all of the remaining arguments are the filenames which followed
    # the optional switches.  You can process those with "for i" or "$@".

For more complex/generalized cases, or if you want things like "-xvf" to be handled as three separate flags, you can use getopts or getopt. (Heiner, that's your cue....)

---

36. I want all the lines that are in file1 but not in file2 (set subtraction).

  cat file1 file1 file2 | sort | uniq -c |
  awk '{ if ($1 == 2) { $1 = ""; print; } }'

This may introduce an extra space at the start of the line; if that's a problem, just strip it away.

Also, this approach assumes that neither file1 nor file2 has any duplicates in it.

Finally, it sorts the output for you. If that's a problem, then you'll have to abandon this approach altogether. Perhaps you could use awk's associative arrays (or perl's hashes or tcl's arrays) instead.

---