os54 slides unix shell scripting

7/24/2019 OS54 Slides UNIX Shell Scripting

1/75


2/75

We enable you to leverage knowledge

anytime, anywhere! 2008 Infosys Technologies

Ltd.

2008 Infosys TechnologiesLtd .

ConfidentialConfidential 22

Confidential Information

This Document is confidential to Infosys Technologies Limited. This document containsinformation and data that Infosys considers confidential and proprietary (Confidential

Information).

Confidential Information includes, but is not limited to, the following:

Corporate and Infrastructure information about Infosys;

Infosys project management and quality processes;

Project experiences provided included as illustrative case studies.

Any disclosure of Confidential Information to, or use of it by a third party, will be damaging toInfosys.

Ownership of all Infosys Confidential Information, no matter in what media it resides, remains withInfosys.

Confidential information in this document shall not be disclosed, duplicated or used in whole orin part for any purpose without specific written permission of an authorized representative of

Infosys.

This document also contains third party confidential and proprietary information. Such third partyinformation has been included by Infosys after receiving due written permissions and

authorizations from the party/ies. Such third party confidential and proprietary information shall notbe disclosed, duplicated or used in whole or in part for any purpose without specific written

permission of an authorized representative of Infosys.


3/75


4/75


5/75


6/75


7/75


8/75



Ltd.



Introduction to Unix

What is Unix?

Need for Unix in todays world Evolution of Unix

Flavors of Unix

http://www.bell-labs.com/history/unix/

Unix is a very popular multi-user, multitasking, time-sharing operating system.

Unix has become the operating system of choice for various engineering and scientificapplications. The need for Unix could be determined by the different categories ofapplication it suffices. viz. networking, programming, multimedia, high-performancecomputing to name a few.

Evolution of UNIX:1965 : AT&T, GE, IBM and Project MAC joined together to MULTICS (MultiplexedInformation and Computing Service), MULTICS is a time-sharing operating system.

1969 : AT&T Bell Labs was out of MULTICS project in 1969. The initial version of UNICSwas written by Ken Thompson. UNICS was only compatible with PDP-7 machine. UNICSstands for Uniplexed Information and Computing Services. UNICS is later changed to UNIX

1973 : Re-written in high level language C, thus making it machine-independent.

1977-1981 : Unix was widely available at minimal cost and became popular for scientificapplications.

1982 : Unix System III is released.

1984-85 : Unix System V is released.

1992-93 : 4.4 BSD is released.1994 : Linux 1.0 is released.

2001 : Linux 2.4 is released.

Unix is available in many flavors like :

AIX (Advanced IBM Unix), HP-UX ( Hewlett Packard Unix), MINIX (Minimal Unix)

SCO UNIX, SOLARIS, XENIX, SUN OS, LINUX


9/75


10/75


11/75


12/75


13/75


14/75



Ltd.



Operating System Services

Exception: An exception is an unexpected event like reference toan illegitimate address. Exceptions usually occur in the middle of a

process where as interrupts occur between them. Differentalgorithms are used to handle interrupts/exceptions efficiently

Processor execution levels

Based on the priority level of an interrupt the kernel masks certaininterrupts in order that a critical process may proceed without error.

The priority levels in descending order are: Machine error

Clock

Disk

Network Devices

Terminals

Software Interrupts

Memory Management

The concept of a Virtual Machine is central to memory management under Unix. Thecompiler generates addresses for a virtual machine as if no other program willexecute simultaneously on the physical machine. Virtual addresses generated by thecompiler need not be identical to the actual phys ical addresses in the machine. Thememory management coordinates with the machine hardware to set-up a virtual to physicaladdress translation that maps the compiler-generated address to the physical machine

address.


15/75


16/75


17/75


18/75


19/75



Ltd.



How Unix Works

USER TYPES COMMAND

SHELL EXECUTESUTILITY TO CARRYOUT COMMAND

SHELL ASKS FOR A COMMAND

USER INTERACTS WITHUTILITY

SHELL PROMPTS FORNEXT COMMAND

USER TYPES CONTROL-D

LOGOUT

LOGIN

The user connects to the Unix Server using an utility called telnet, and by mentioning the IPaddress of the Unix Server.

The Server requests for the username and password. The user needs to give the validusername and password, for log-in.

Once the username and the password is right, the shell-prompt is displayed, where-in the

user could execute the commands.


20/75


21/75


22/75


23/75


24/75



Ltd.



File Descriptors

Standard Input (0)

This represents the file to which the process is tied to for accepting theinput

Standard Output (1) This represents the file to which the process is tied to for directing the

output

Standard Error (2) This represents the file to which the process is tied to for writing error

messages

Standard Input (0)

Takes the keyboard or f iles as input.

Example

$> mail user1 < s1 #s1 is the name of the file.

The contents of s1 is mailed to user1.

Here the input for the command mail is coming from file user1

Standard Output (1)

This is the default fi le.

Example

$> ls 1> infy #Redirects the output to a file called infy.

Is same as

$> ls > infy

Standard Error (2)

Is used for redirecting the errors.

Example

$> ls abcd11 2> def #Redirects the errors into a file called def

$> cat def

ls: abcd11: No such file or directory


25/75


26/75


27/75


28/75


29/75



Ltd.



Summary

Introduction to the Unix operating system

Unix System Architecture General Commands and Utilities

Wild Cards, I/O redirection and Filters

Regular Expressions


30/75


31/75


32/75


33/75



Ltd.



Types of shells

Bourne Shell Written by Steve Bourne

Bourne Again Shell Brian Fox wrote the first version released in 1988.

Later Chet Ramey joined and is responsible for numerous bug fixand addition of useful features

C Shell Written by Bill Joy

Korn Shell Written by David Korn

TC Shell (tcsh)

Public Domain Korn Shell (pdksh)

Bourne shell (/bin/sh) (written by Steve Bourne of Bell Labs)

This is the original Unix shell and it is available on all UNIX systems.

This shell does not have support of latest user interactive facilities provided by modernshells such as the Korn shell and C shell.

bourne shell does provide an easy to use language to write scripts.

Bourne again shell (/bin/bash) ( Free Software Foundation ).bash provides all the interactive features of the modern shells like Korn shell (ksh) and theC shell (csh) . Bashs programming language is also compatible with the Bourne shell (sh).

C shell (/bin/csh) (written at University of California, Berkeley)

C shell provides a C-like language to write shell scripts.

Korn shell, /bin/ksh, (written by David Korn (Bell Labs) )

Ksh is now the standard shell on all Unix-like systems. Provides all the interactive featuresof the C and TC shells. Kshs shell programming language is also similar/compatible to shellprogramming language of the original Bourne shell.

TC Shell (tcsh )

tcsh provides all the features of the C shell with emacs style editing of the command line.

References: http://unixhelp.ed.ac.uk/


34/75



Ltd.



Welcome to Shell Script ing

Create a file called hello.sh using editor or from command line

Assign execute permissions for yourself

$ chmod u+x hello.sh

Execute it from command prompt

$ ./hello.sh

Creates a sub-shell and execute hello.sh

$ . ./hello.sh

Execute hello.sh in current shell

$ cat hello.sh

#!/bin/bash

date; cal #Di spl ays date f ol l owed by cal endar

echo Welcome to Shell Scripting

hello.sh when run, displays date and calendar for current month, followed by messageWelcome to Shell Scripting

First line of the script #!/bin/bash determines which shell to be used to execute the shellscript. This can be changed to /bin/sh (for Bourne shell), /bin/bash (for bash shell) , /bin/ksh

(for Korn Shell), etc.

Shell treats all characters after # till end of line as comments (ignores them). Comments arevery useful for documenting and elaborating the functionality achieved by command or groupof commands.

Note: While executing shell script using ., it is not required to have execute permissions set

Note: On some systems /bin/sh may refer to POSIX Shell. bash conforms to most of POSIX1003.2 standard. If one is too cautious about portability issues, one can use - -posix optionwhile invoking bash. This would make bash act like a POSIX shell.


35/75


36/75


37/75


38/75



Ltd.



Quotes

Quotes help us in reinforcing the correct interpretation of

things which shell might interpret differently Single quotes

Shell treats everything between pair of quotes literally

We cannot have a single quote inside a pair of single quotes

Double quotes

Enclosed string considered as single string. Single quote has nospecial effect inside.

Characters $, , and \ must be escaped using \

Allow parameter and command substitution, evaluation of arithmeticexpressions

Back quotes (ticks or grave accent) ` `

Enclosed string is treated as a command and shell executes it

Single Quotes:

$ echo Let me study # Let me study

$ echo $var # $var

$ echo *[|>/$\? # *[|>/$\?

Double Quotes:

$ star=*$ echo $star # Displays list of all files in current directory

$ echo $star # Displays *

$ age=29; echo \\$age\=$age # Displays $age=29

$ age=30; echo $age=$age # Displays $age=30

$ varo=Hello vart=World

$ var=$varo $vart

$ echo $var # Displays Hello World

Back Quotes:

$ echo Working Directory is `pwd`

Working Directory is /root/cppdemos

Note: POSIX supports newer construct for command substitution and use of back-quote isdepreciated

$ x=10

$ echo 'echo $x# Displays echo$x

$ echo "echo $x" # Displays echo 10

$ echo `echo $x` # Displays 10


39/75


40/75



Ltd.



test command

Used extensively for evaluating script conditions

It evaluates the condition on its right and if the result istrue, returns an exit status ($?) of zero

[ ] can be used instead of writing test explicitly

n1 eq n2 -Integers n1 and n2 are equal

n1 ne n2 -Integers n1 and n2 are not equal

n1 lt n2 -Integer n1 is < n2

n1 gt n2 -Integer n1 is > n2

n1 le n2 -Integer n1 is = n2

s1 = s2 Strings s1 is same as as s2

s1 != s2 Strings s1 and s2 are not same

-n s1 String s1 is not null (non-zero length)

-z s1 String s1 is null (zero length)

$ x=10 y=20

$ test $x -eq $y; echo $?

1

$ test $x -ne $y; echo $?

0

$ [ $x -ne $y ];echo $?

0

$ [ $x -eq $y ]; echo $?

1

Above list illustrates how test can be used for comparing numbers and strings. Test returntrue when comparison is successful, otherwise false. 0 represents truth while all othervalues (positive and negative) represent false

Note :A space is mandatory after [ and before ]

$ str1="Infosys str2="Infy"

$ test $str1 =$str2 ; echo $?

1 # indicates both the strings are different.

Logical operations: test supports logical and (-a), logical or (-o) and logical not (!).Parenthesis can be used for grouping

$ x=10;

$ test $x -lt 15 -o $x -gt 5; echo $?

0

$ test $x -lt 15 -a $x -gt 5; echo $?

0

$ test ! $x gt 5; echo $?

1


41/75

File test expressions: test expressions can be used to perform file tests as well

-e filename filename exists

-f filename filename exists and it is a regular file # $ test -f $f ; echo $?

-r filename filename exists and is readable

-w filename filename exists and is writable # $ [ ! x $file ]; echo $?

-s filename filename exists and is not empty (size greater than 0)

-d filename filename exists and is a directory

Note: to perform multiple tests on a file, we should use following syntax:

$ test option1 filename option2 filename

For ex: $test r file1 w file1 is used to check whether a file is both readable and writeable.


42/75


43/75



Ltd.



Iteration control

while, until and for loops

# Iterates, if condition is true

while condition; do

statements

done

---------------------------------

# Iterates, if condition is false

until condition; do

statements

done

---------------------------------

# Iterates, if list is not empty

for var in list_of_values; do

statements using $var

done

#!/bin/bash

var=2;

while [ "$var" -le 10 ]; do

echo "Var is $var"

var=$((var+3)) # var= expr $var + 3`

done

------------------------------------

var=2;

until [ "$var" -gt 10 ]; do

echo "Var is $var"

var=$((var+3)) # var= expr $var + 3`

done

------------------------------------

for var in * ; do

echo "$var # Displays all filenames

done

Example: Changes the extension of all .txt file to .doc

for filename in *.txt; do

tmpname=$(basename $filename txt)

mv $filename ${tmpname}doc

done

Example: Finds factorial of number passed from command line

fact=1 num="$1"

while [ "$num" -gt 0 ]; do

fact=$((fact*num))

num=$((num-1))

done

echo "Factorial of $1 is $fact

bash supports non POSIX iteration construct called select. Select generates a menu(formatted by numbers 1, 2, 3 etc) of each item in the list. Asks user to enter a number. Storesthe selected choice number in a built in variable REPLY. Loop terminates when we use break.

#!/bin/bash

select choice in "cal" "who" "exit"; do

case "$REPLY" in

1) echo "$choice"; cal;;

2) echo "$choice"; who;;

3) echo "$choice"; break;;

esac

done


44/75


45/75



Ltd.



I/O redirection and piping data with loops

#!/bin/bash

lineno=1

while read line; do

echo " $lineno $line"

lineno=$((lineno+1))

done < history.txt

----------------------------

#!/bin/bash


echo "$filename"

cat $filename

echo "--- End ---"

done > textfiles.log

for filename in *.dat; do

sort r $filename

done&

--------------------------------------

#!/bin/bash

lineno=1

cat history.txt | while read line; do

echo " $lineno $line"

lineno=$((lineno+1))

done

--------------------------------------

#!/bin/bash


echo "$filename"

done | wc -l

A loop can be executed in the background by adding & to done statement.

First example of I/O redirection reads input from history.txt line by line and displays each lineprefixed by line-number. Second example stores all .txt files in current directory intotextfiles.log after re-formatting. During re-formatting name of file is prefixed to its contentsand message --- End --- is suffixed.

Note:All commands that write to standard output inside loop take part in redirection from

loop. If there is an statement like echo hello >/dev/null inside loop; this would not take partin output redirection from loop.

Note: Some shells take care of piping with loops by creating a sub-shell. Any modificationsmade to variables inside loop, would not be reflected outside iteration control construct thatuses piping. Most of the shells (except few old shells) perform I/O redirection from loopswithout invoking a new sub-shell; hence any modifications to variables inside the loop wouldbe reflected outside the loop as well. Refer to sh, bash and ksh manual for details.

#!/bin/bash

count=5

while [ "$count" -lt 10 ]; do

newcount=$((count*2))echo "$count"

count=$((count+1))

done > count.txt

echo ":$count:$newcount: # Displays :10:18:

If we replace done > count.txt with done | wc l, echo :$count:$newcount: would display5:: (original value of count is not modified)


46/75



Ltd.



Functions

Advantages of using functions Helps us in avoiding repetition of script code if it is required a

number of times

When we run a program that requires a number of functions, wecan store functions in another file and load them into our scriptwhen we wish to use them

Features Arguments passed to function can be accessed using $1, $2 and

so on

Value can be returned from functions return $var

Value returned from a function can be accessed using $?

We can write recursive functions

Functions loaded into memory can be unset $ unset -f function_name

When making standard operations, to avoid having to repeatedly create each line of thescript, we create functions that we call into out scripts.

All functions must be defined before use. A function runs in the current environment and canbe recursive. functions cannot be exported to subshells.

Below example illustrates that $1, $2, etc inside a function refers to arguments passed to

function and not the shell script$ cat >test.sh

#!/bin/bash

argsinfo () {

echo '--argsinfo()--'

echo " $# : $* "

}

echo '--test.sh--'

echo " $# : $*"

shift 2

argsinfo these arguments are different

$ ./test.sh 1 2 3 4 5

--test.sh--

5 : 1 2 3 4 5

--argsinfo()--

4 : these arguments are different


47/75


48/75


49/75



Ltd.



Customizing your environment

Environment variables are set during startup of shell HOME, IFS, PATH, PS1, PS2, PWD, TERM, DISPLAY, etc

are automatically exported to sub-shells Value can be displayed using echo $HOME, echo $PS1, etc

User can also assign new values PS1=Enter a command>

HOME : Users home directory

PS1, PS2 : Primary/secondary prompt

PWD : Present working directory

IFS: Input Field Separator

PATH : List of directories to be searched for command

SHELL : Default shell

EDITOR: Default editor

Export

Makes a variable a part of environment and also be accessible to thechild shell

$ echo $PATH #To display the PATH

/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/home/ajit:.

$ PATH=$PATH:/home #To append to the existing path.

$ echo $PATH

/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/home/ajit:.:/home

A sub-shell has no knowledge of any user defined variables in parent shell.

If exported variables are modified in subshell, it has no effect on original variables in parent shell.Exported variables are not only available in sub-shell but for all processes spawned by thosesubshell.

Example:

$> a=10

$> echo $$ #PID of the shell

6034

$> echo $a

10

$> sh #Creates a child shell

$> echo $$ #PID of child shell

9072$>echo $a

Af ter using export

$> export a

$> sh #Creates a child shell

$>echo $a

10


50/75



Ltd.



Customizing your environment (Contd)

set Used to display all variables (user defined as well as environmental)

Used to set positional parameters Can be used to debug shell scripts as well

alias Shorthand notation for referencing commands

unalias Remove aliases from alias list

unset Used to unset variable or function

Syntax unset [-vf] name

readonly Used to mark variables readonly, after this value of variable cannot be

changed

$ set #Displays all variable and their values

$ set $(date) # runs date command sets it output as positional parameters

$ echo $# # Displays 6

$ echo $1 $2 $6# Displays Mon Jun 2008

$ set -- # Unsets all positional parameters

$ set one two three four

$ echo $# # Displays 3

$ unset v varname # Unsets variable varname

$ unset f func_name # Unsets function func_name

$ readonly # Displays all readonly variables

$ var=100; readonly var # Makes var readonly

$ var=200 # Fails, var is readonly

$ alias # Displays all aliases

$ alias ll=ls l # ll aliased to ls -l

$ ll # executes ls l

$ unalias ll # alias removed

$ unalias -a # Removes all aliases


51/75



Ltd.




Shell start up process

Bourne shell

Locates file .profile in home directory and reads requisite settings from it

.profile is executed only once when you login and not for every invocation ofa subshell

Korn shell

Locates system startup file in /etc/profile, reads it; locates .profile file inhome directory, reads it; finally reads environment file

If a subshell is invoked, only environment file is read

Bash

Locates system startup file in /etc/profile, reads it. Locates .bash_profile inhome directory, reads it

If .bash_profile is not found, bash tries to read .bash_login or .login files inhome directory

.bash_logout is executed every time one logs out

If a subshell is invoked, only .bashrc is read

What is so special about files /dev/tty and /dev/null?

Note: In bourne shell, .profile file contains the settings for the user to customize his shell

When shell starts up, it reads various values from different files; that determine setting for aparticular system. Some of these files are located in our home directory while some may be insystem directories.

/dev/tty and /dev/null are very useful. Any data redirected to /dev/null (bit bucket or trash file) isdiscarded. This is useful if our requirement is to find out the exit status rather than the output.

Example:

if who | grep $username >/dev/null ; thenecho $username is logged in

else

echo $username is not logged in presently

fi

/dev/tty represents a terminal file and cannot store data. Useful for reading passwords.

Example:

#!/bin/bash

echo 'Enter password?'

stty -echo # echo turned off

read varo


52/75



Ltd.




What should shell script do when a signal is raised? Let the default-activity associated with raised signal take over

Ignore the raised signal Specify an user defined action to be performed

trap Used to trap process signals Shell can register a signal handler to catch specified signal Syntax

trap command1; command2; signal_name

Common signals SIGEXIT 0 Normal termination of script SIGHUP 1 Hangup SIGINT 2 Interrupt (ctrl+c) SIGFPE 8 Floating point exception SIGKILL 9 Kill signal

SIGPIPE 13 Broken pipe: write to pipe with no readers SIGTERM 15 Termination signal

When we press Ctrl+c during the execution of a shell script, a signal is sent; normally theprogram is terminated. However, sometimes this is undesirable. One should ensure removalof temporary files and other book-keeping (clean up) must be done before a shell scriptshould terminate. UNIX provides the trap command which allows the user to alter the effectsof certain events that generate signals, which generally tends to halt a process.

#!/bin/bash

trap 'echo SIGINT disabled' INTtrap 'echo $0 terminated' EXIT

trap ' ' HUP # SIGHUP Ignored

echo 'Enter your name?'

read name

echo "Name: $name"

$ ./test.sh

Enter your name?

SIGINT disabled # User pressed ctrl+c

Amit Tiwary # User enters Amit Tiwary

Name: Amit Tiwary

./test.sh terminated # Displayed due to trapping of SIGEXIT

Note: If one ignores a signal, all subshells ignore that signal as well. If one has set trap totake action on receiving the desired signal, the subshell would continue to thake the defaultaction and not the one specified by the user. SIGKILL cannot be trapped or ignored

trap - INT TERM EXIT # Resets the handlers for SIGINT, SIGTERM and SIGEXIT


53/75



Ltd.


ConfidentialConfidential

REGULAR EXPRESSIONS

5353


54/75


55/75



Ltd.



Regular Expressions (Contd)

Atoms

Character This may contain any printable character (alpha-numeric or special)

. (dot) represents any single character except newline

Range/Class matches any one character from the set [ ]

Anchor

Back Reference

Character: $ grep U song.txt matches all lines containing U

Regular expression meta-characters like * . ? $ ^ etc must be preceded by \ to suppress theirmeaning.

$ grep \$ song.txt matches all lines containing $

Dot: $ grep W . . song.txt matches all lines containing W followed by any two characters

Range/Class: $ grep [0-9] song.txt matches all lines containing a digit

A ^ placed as first item of range/class negates the meaning.

[^A-Z] match anything except uppercase letters, [^aeiou] matches anything except vowels, [\^]matches a , [^\^] matches anything except a ^, [A-Za-z0-9] matches an alpha-numeric character,[][-] matches ] or [ or , [0-5\-] matches 0 or 1 or 2 or 3 or 4 or 5 or

Anchor: There are four types of anchors. ^(beginning of line), $ (end of line), \< (beginning of word)and \> (end of word)

$ grep ^F song.txt matches all lines beginning with F, $ grep \^ song.txt matches all linesbeginning with ^ , $ grep e$ song.txt matches all lines ending with e

$ grep \.$ song.txt matches all lines ending with . , $ grep \ song.txt matches all lines having words ending with a comma, $ grep ^$ song.txtmatches all blank lines

$ grep ^.$ song.txt matches all single character lines

Back Reference: Used to restore or replay a pattern saved in buffer. There are nine buffersnumbered 1 to 9. \n restores the pattern from nth buffer. We would be covering more of this whenwell talk about tagged regular expressions under operators.


56/75



Ltd.



Regular Expressions (Contd)

Operators

Alternation | Repetition \{s,e\}

Grouping ( )

Shorthand * + ?

Operators: Operators are used to combine atoms.

Al ternat ion: This is denoted by | and behaves like logical or

$ egrep you|You song.txt matches lines containing either you or You

$ egrep '\|\' song.txt matches lines containing either word this or that

Repetition: is denoted by \{s,e\} and matches previous character s to e times, where s and e are

positive integers.

ab\{2,4\} matches abb abbb and abbb

ab\{2,\} matches abb abbb abbb i.e. a followed by atleast 2 bs

ab\{,2\} matches a ab abb i.e. a follwed by atmost 2 bs

ab\{2\} matches abb i.e. a followed by exactly 2 bs

\{2,4\} matches {2,4} (Why? Repetition is not prefixed by any character)

Grouping: is denoted by () and is used to group two or more regular expresions

Hel(l|lo) matches Hell or Hello

a(bc)\{2\} matches abcbc

Nested Grouping is also possible

(a(bc)\{2\}d)\{2\} matches abcbcdabcbcd

Short Hand Operators: There are three shorthand operators * + and ?

* matches zero or more repeats of previous atom, equivalent to \{0,\}

+ matches one or more repeats of previous atom, equivalent to \{1,\}

? matches zero or one repeat of previous atom, equivalent to \{0,1\}

Note that AA* is equivalent to A+


57/75



Ltd.



Summary of Regular Expressions

. Matches any single character except newline \ Suppress special meaning of meta-characters

[ ] Matches any one character from the set (- indicates range and ^inside [ ] negates the meaning i.e. character not marching thecharacter set is matched)

^ Matches at the beginning of line $ Matches at the end of line \n Replay the nth pattern saved by tagged regular expression  | Matches alternate regular expressions ( ) Groups regular expressions to be matched \{s,e\} Matches occurrences of preceding character s through e times * Matches zero or more occurrences of preceding regular expression + Matches one or more occurrences of preceding regular expression ? Matches zero or one occurrence of preceding regular expression

 Saves the matched pattern enclosed between $ and $ into abuffer

Tagged Expressions (Saving for later replay): Regular expressions may be tagged andsaved in one of the 9 buffers to be replayed later. Symbol used is 

$[A-Z]$\1 matches repeated (twice) upper case letters

$.$ $.$ . \2\1 matches 5 letter palindromes

$UNIX$.*\1 finds two occurrences of word UNIX in a line

$UNIX$\1 finds repeated occurrences of word UNIX in a line

Note: There is no regular expression to mach a newline


58/75



Ltd.


ConfidentialConfidential

ADVANCED SHELL SCRIPTING

5858


59/75


60/75



Ltd.



Advanced I/O redirection (Contd)

Miscellaneous I/O redirectors

file : Use file as both standard input/output

nfile : Use file as both standard input/output for file descriptor n

n>file, n >> file : redirect file descriptor n to file

n& , n&m , nfile : redirect standard output and standard error to file

&- : close the standard input/output

n&- : close input/output from file descriptor n

>|file : force standard output to file even if noclobber option is set

n>|file : force output to file from file descriptor n even if noclobber is set

errorlog # redirects all standard output to file result and all errors to fileerrorlog

$ command >result 2>&1 # redirects all standard output and error to file result since fd2 is

made a copy of fd1 Example: cat song.txt >newsong.txt 2>&1$ command 2>&1 >result # Shell processed redirection left to right. This is not equivalent toprevious command Example: cat song.txt 2>&1 >newsong.txt

$ exec > test.out # test.out is now standard output, all standard output goes to test.out

$ exec > /dev/tty # Redirection standard output to terminal

$ exec 2> /tmp/error.log # Standard error now redirected

$ exec 4< /cppdemos/file # Open new file descriptor 4

$ exec 4


61/75


62/75



Ltd.



Advanced Parameter Expansion

Substitution operators

Pattern matching operators

${var:-word} If var is set (exists and not nul l) , return i ts value; else return word

${var:=word} If var is set (exists and not null), return its value; else set var to word and return word

${var:?word}If var is set (exists and not null), return its value; else display var:word and abortscurrent command or shell script. Omitting word displays parameter null or not set

${var:+word} If var is set (exis ts and not null), return word; else return null

${var%pattern}If the pattern matches the end of the variable's value, delete the shortest part thatmatches and return the rest.

${var%%pattern}If the pattern matches the end of the variable's value, delete the longest part thatmatches and return the rest.

${var#pattern}If the pattern matches at the beginning of the variable's value, then deletes theshortest part that matched and return the rest.

${var##pattern}If the pattern matches the beginning of the variable's value, delete the longest partthat matches and return the rest.

Substitution operators are used test existence and substitution of default value of variables.

$ count= ; echo ${count:-5} # Displays 5

$ echo $count # Displays blank line

$ count=2; echo ${count:-5} # Displays 2

$ echo $count # Displays 2

$ count= ; echo ${count:=5} # Displays 5$ echo $count # Displays 5

$ count=2; echo ${count:+5} # Displays 5

$ echo $count # Displays 2

$ count= ; echo ${count:?undefined} # Displays count:undefined

Note: pattern refers to shell wildcard patterns

Note: ${#var} can be used to count the no of characters stored inside var.

$var=hello world; echo ${#var} # displays 11

$ var="thinktwice"

$ echo ${var#th} # Displays inktwice

$ echo ${var#*i} # Displays nktwice

$ echo ${var##*t} # Displays wice

$ echo ${var##*t*i} # Displays ce

$ echo ${var%t*i} # Displays thinktwice

$ echo ${var%i*e} # Displays thinktw

$ echo ${var%%i*e} # Displays th

$ echo ${var#tt} # There is no match so displays thinktwice


63/75



Ltd.



eval

The eval command can be used to execute a given string as it is ashell command

Syntax $ eval command_string Command_string is first processed by the shell and then passed to eval

for execution

Examples $ count=| wc l ; $ ls l $count # Error, use eval ls l $count $ pro=Perl; pre=pro; echo $pre # Displays pro $ eval echo \$$pre # Displays Perl $ great=>; $ echo Perl $great out # Displays Perl > out $ great=>; $ eval echo Perl $great out # Sends Perl to out $ cat < *.c # Error $ eval cat \< *.c # Displays all .c files Displaying the value of last command line argument

$ echo $$# $ echo \$$#

$ echo $\$# $ eval echo \$$# # Ok


64/75



Ltd.



Evaluation order

1. split into tokens; if single quote go to step 11; if double quote go to step 5

2. 1st token keyword, read next command and go to step 1

3. 1st token alias, expand alias and go to step 1

4. tilde(~) expansion

5. variable/parameter substitution

6. command substitution

7. arithmetic expression substitution; has double quotes go to step 11

8. word splitting

9. pathname/wildcard expansion

10. command lookup: special built-in, function, built-in, executable file in $PATH

11. run command (if eval is used, output of step 11 is passed to step 1. bash performsbrace expansion between steps 3 and 4)

Attempt writing a function cd that should not only change directory but alsodisplay current working directory as part of command prompt

Every line read from standard input is called pipeline (containing one or more commandsseparated by zero or more | characters). Shell breaks each pipeline into commands, shellthen sets up I/O for the pipeline and then follows above mentioned 11 step process for eachcommand.

Example: Assume nerds home directory is /home/nerd, present working directory is/home/nerd/utils and there are files named data5, test.txt, test.sh, test.c in the current

directory$ alias li=ls i

$ var=5000

$ li $(pwd)/main.c ~nerd/cppdemos/*$(($var/1000))

ls i $(pwd)/main.c ~nerd/cppdemos/*$(($var/1000))

ls i $(pwd)/main.c /home/nerd/cppdemos/*$(($var/1000))

ls i $(pwd)/main.c /home/nerd/cppdemos/*$((5000/1000))

ls i /home/nerd/util/main.c /home/nerd/cppdemos/*$((5000/1000))

ls i /home/nerd/util/main.c /home/nerd/cppdemos/*5

ls i /home/nerd/util/main.c /home/nerd/cppdemos/data5

/usr/bin/ls i /home/nerd/util/main.c /home/nerd/cppdemos/data5

Examples of special built-in: break, continue, eval, exit, readonly, return, export, et, trap, etcExamples of shell built-in: cd, wait, test, etc

Note: command builtin bypasses shells inclusion of functions in the search for commandsto run

Example: $ command cd /home/nerd


65/75


66/75



Ltd.



Non portable features from bash (Contd)

Extended substitution and pattern matching operators

${var:offset:length}

Substring operation. Returns substring starting at offset and uptolength characters. Offset starts at 0 (first character). If length isommited, all characters form offset and onwards are returned.

echo ${var:5} # Displays twice

echo ${var:2:3} # Displays ink

${var/pattern/string}

First longest match of pattern in $var is replaced with string. Ifpattern begins with #, pattern should match at the beginning of$var. If pattern begins with %, pattern should match with the endof $var

echo ${var/*k/act} # Displays acttwice

${var//pattern/string}

Every longest match of pattern in $var is replaced with string. Ifpattern begins with #, pattern should match at the begining of$var. If pattern begins with %, pattern should match with the end

of $varecho ${var//t?i/X} # Displays XnkXce

echo ${var//i?/X} # Displays thXktwXe

echo ${var/%i*e/X} # Displays thX

var=thinktwice


67/75



Ltd.



Non portable features from bash (Contd)

Arrays

Index always starts from 0 Array name refers to first element of array at index 0

# Declaring an array

$ declare -a array

$ array[0]="Amit"

$ array[2]="Tiwary"

$ array[1]="Kumar"

$ array[5]="Linux

# Other ways of declaring and initializing

$ friends=( Bill Arnold Randal Eric John )

$ friends=( $(cat namelist) )

Note: array is spare whereas friends is

continuous

$ echo ${array[0]} ${array[2]}

Amit Tiwary

$ echo $array[0] $array

Amit[0] Amit

# echo ${array[*]}

Amit Kumar Tiwary Linux

# echo ${#array[*]}

4 # Total no of elements

# echo ${#array[2]}

6 # Length of element at 2

Note: behavior of *, @, * and @ in arrays is in line with behavior of $*, $@, $* and $@respectively

Values can be deleted using unset

$ unset array # Deletes array

$ unset array[3] # Deletes element and index 3


68/75


69/75



Ltd.



Debugging shell scripts

Steps involved

Try to understand the error message Figure out location of error in the script

Fix the error

Using set options -n : read commands but not execute

-v : display shell input lines

-x : execution trace

-t : exit after reading one command and executing it

-e : exit if any command terminates with non zero status

-u : treat unset variables as error condition

$ nl test.sh

1 #!/bin/bash

2 rem=$($1%2) # rem=$(($1%2))

3 if [ "$rem" -eq 0]; then # space missing before ]

4 echo "Even"

5 else

6 echo "Odd"

7 fi

$ sh -v ./test.sh 5

#!/bin/bash

rem=$($1%2)

$1%2

./test.sh: line 2: 5%2: command not found

if [ "$rem" -eq 0]; then

echo "Even"

else

echo "Odd; fi

./test.sh: l ine 3: [: miss ing `]'

Odd

--- After correcting the errors ---

$ sh -x ./test.sh 5

+ rem=1

+ '[' 1 -eq 0 ']'

+ echo Odd

Odd


70/75


71/75


72/75


73/75


74/75


75/75

os54 slides unix shell scripting

Documents