1 korn shell script writing & hpc science & technology support group high performance...
TRANSCRIPT
1
Korn Shell Script Writing & HPC
Science & Technology Support Group
High Performance Computing
OSC (Ohio Supercomputer Center)
1224 Kinnear Road
Columbus, OH 43212
2
Table of Contents• Introduction
– What is a shell script?– Script Programming Language– Why use shell scripts?– Korn shell scripts– References
• The Evolution of a Script– Sampling of Script Features– A simple script– A more versatile Script– Creating Files within a Script– A documented Script
• Script Basics– Creating a shell script– Executing a shell script– Echoing script commands
• Script Variables– User-defined Variables– Storing a Value into a Variable
• Assignment Operator =• Read Command• Command Substitution
– Null Variables– Special Shell Variables– Positional Parameters– Referencing Variables– Parameter Substitution– Exercises
• Decision-Making statements– Flow Control– Exit Status– If-Then-Else command – Conditional Tests– Logical Operators– Illustrative Examples– Case statement– Exercises
3
Table of Contents• Looping Statements
– for Statement– select Statement– while Statement– Until Statement– Exercises
• Working with Integers– Arithmetic Operations– Operator Precedence– Numerical Bases– String vs Integer Variables– Exercises
• Script I/O– Shell Redirection Operators– "Here-is" Files– print– read – Quoting– Exercises
• Communicating with Running Scripts
– Sending Signals– How Scripts Receive Signals
• Script Functions– Why Use Functions?– Syntax & Use– Auto-Loading– Function Definition Location– Command Precedence
• HPC Designed Scripts– Timing Loops– Debugging Options– Batch Jobs– Performance Tool
Preprocessing– Multiple Tools runs– Multiple Tool Reports– File Renaming
4
Introduction• What is a shell script?• Script Programming Language• Why use shell scripts?• Korn Shell scripts• References
5
What is a shell script?• A shell script is a collection of Unix commands entered into
a file. When the file (script) is executed, the commands are executed in order.
• Running a shell script is EXACTLY equivalent to interactively entering each command
• For example, say a user always does the same four steps before printing out a file: making the file owner- writable, using vi to check/change the file, printing it out, and then making the file unwritable (for safety reasons)
• An interactive session of a user following this procedure for the file comp.txt is shown on the next page
6
Interactive editing and printing $ chmod u+w comp.txt
$ vi comp.txt Here is a table summarizing OSC computers as of today.
Supercomputer # Processors Proc Speed Ideal Calc. ------------ ----------- --------- ------------- Itanium Cluster 124 900 MHz 223.2 GFLOPS Cray SV-1 16 300 MHz 19.2 GFLOPS Origin 2000 64 300 MHz 38.4 GFLOPS IA-32 Intel 128 550 MHz 70.4 GFLOPS Sun Fire 6800 24 900 MHz 43.2 GLOPS "comp.txt" 11 lines, 535 characters $ lpr -Pps1 comp.txt
$ chmod u-w comp.txt
7
Making the viprint script• To make the previous editing and printing procedure into a
script, all the user has to do is type the same commands they use interactively into a file
• Say the file is called viprint, it looks like this: $ cat viprint chmod u+w comp.txt vi comp.txt lpr -Pps1 comp.txt chmod u-w comp.txt• After viprint is made executable, whenever the user wants
to execute these four commands they simply type viprint as if it was a “normal” Unix command
$ viprint
8
Script Programming Language• The real power is writing shell script comes from using
special shell commands and variables that essentially make your script act like a program
• In this course, you will learn about the programming statements which can be used in scripts, including:– Initializing and using your own script variables. Both single valued
and arrays– Performing operations with variables– Decision-making statements (if-then-else, etc)– Looping statements (for,etc)– Defining and using functions in scripts– Script I/O commands and procedures
• Above bullet reads like a “table of contents” for a text on a “traditional” programming languages such as Fortran or C, although …
• Script programming has less capabilities and commands
9
Why use shell scripts?• Convenience: Run a number of Unix commands by typing a single
script name
• Command log: Script can permanently contain a detailed set of commands with numerous options. (If interactive, complex command is gone)
• Usefulness: Write a script with programming commands to give the script a number of capabilities/options
• Take “full” use of what the shell has built-in to offer you
• Considering a script as a program, no compiler, loader, or libraries needed to make it executable
• Portability: If a script works in the shell on your machine it will work on the same shell on a different machine
• Easy to debug: Small number of programming statements which are relatively simple
• Easy to modify, expand, and customize: Just type in the new commands
• Run-type interaction: Can send signals to running scripts
10
Korn shell scripts• Several types of Unix shells exist, each with their own script
programming languages: – Bourne (sh), Born Again shell (bash), Korn shell
– C shell (csh), TENEX C shell (tcsh)
• Features you would expect from a well-written shell:– Convenient interactive features: command history, command-line editing,
and filename completion
– Powerful scripting capabilities including reliability and efficiency
• This course will discuss Korn shell scripting– It has all the convenient interactive features
– It is backward compatible with the Bourne shell (the first shell)• Most scripts are sh
• Start-up sequence for ksh same as sh only enhanced
• Unix utilities and administration commands depend on sh
– It does not have the scripting bugs that the C shell has
11
Korn shell scripts• What if I already use a different shell on my machine?
– Could switch to Korn shell (use the chsh command if you can)
– Keep your existing shell as your interactive shell and write your scripts as Korn shell scripts (Easily done: one line needed in the script)
12
References• Internet URLs
– nacphy.physics.orst.edu/rubin/melanie/node144.html (Tutorial)
– gonzo.tamu.edu/csh.whynot.html (Bugs in C shell scripting)
• Usenet Newsgroups– comp.unix.shell
– comp.unix.questions
• Anonymous ftp– athos.rutgers.edu/~patra/vi.html (shell-101.BetaA.Z)
– rtfm.mit.edu:/pub/usenet/news.answers/unix-faq/faq (Unix FAQ)
• OSC Unix-Series workshops (oscinfo.osc.edu)– Basic Unix
– Intermediate Unix
– Introduction to Perl
13
The Evolution of a Script• Sampling of Scripting Features• A Simple Script• A More Versatile Script• Creating Files within a Script• A Commented Script
14
Sampling of Scripting Features• Next few slides will act as a course overview as we take a simple
script and keep extending its capabilities
• The more advanced scripts will demonstrate some of the capabilities of Korn Shell scripting
• Don’t worry about understanding all the details at this point. A complete explanation of the scripting commands used is given in later chapters
15
A Simple Script• In the script called triangle, a C program is compiled and then
executed. The program calculates the area of the triangle and reads in the lengths of the sides from a file and outputs the result to another file.
• Here is an example of a user running the script: $ cat triangle cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o area area.c area < sides.in > area.dat cat area.dat rm area $ ls -l triangle -rwxr--r-- 1 dje appl 108 May 3 12:09
triangle $ cat sides.in 8 3 7 $ triangle Area of Triangle is 10.392305
16
A Simple Script• Limitation of the simple triangle script
– Script only works for the one area program. What if you want to run other triangle-related programs with this same script ?
17
A More Versatile Script• Scripts can take arguments on the command line when they are
run. The first argument is passed to a variable with a special name: $1. We can use this feature to make the triangle script work for any program.
• Here is the resulting new script in action: $ cat triangle cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1 $1.c $1 < sides.in > $1.dat cat $1.dat rm $1 $ triangle area Area of Triangle is 10.392305 $ triangle perim Perimeter of Triangle is 18.000000
18
A More Versatile Script• Limitation of versatile script:
– Scripts should be as independent of possible of “outside” existing files (for portability and ease of use). Since the sides.in file is so short, we should create the file within the script
19
Creating Files Within a Script• The easiest method for creating a small “internal” file is to use
“Here-is” Unix redirection. After it is used, this temporary file is removed:
$ cat triangle if [[ $2 = debug ]]; then cc -g -o $1 $1.c print "$1 meant to be debugged" else cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1 $1.c cat << EOF > sides.in 8 3 7 EOF $1 < sides.in > $1.dat cat $1.dat rm $1 sides.in fi $ triangle perim Perimeter of Triangle is 18.000000
20
A Commented Script• A bad script writing style has been shown in all
incarnations of the triangle script: the lack of comments. Comments are critical to another user (or you) in understanding what the script does.
• The special “comment” on the first line of script actually specifies the shell to be used for the script commands. This is required for users whose interactive and script shells differ.
• On the next page is the timing script just discussed but now with proper commenting
21
A Commented Script $ cat triangle #!/bin/ksh # Compiling C code found in first parameter cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1 $1.c # Using redirection to make input file cat << EOF > sides.in 8 3 7 EOF # Using while loop to run the executable 17 times let i=0 while (( i < 17 )); do $1 < sides.in >> $1.dat let i=i+1 done # Checking output cat $1.dat # Clean up files no longer needed rm $1 sides.in
22
Script Basics• Creating a Shell Script• Executing a Shell Script• Echoing Script Commands
23
Creating a Shell Script
• A shell script is an executable file which is executed by the shell line-by-line. It can contain the following: – UNIX commands – shell programming statements – comments
• Create using editor of choice • Can include a #! construct in first line of script to override
login shell – #!/bin/ksh uses Korn shell to execute script – #!/bin/csh uses C shell to execute script – Recommend always put this in every Korn shell script. Makes it
clear what the file is. "file" Unix command will even identify it as a Korn Shell script
• Name should be short and descriptive
24
Executing a Shell Script
There are 3 ways to execute a shell script:
1."dot" method
$ . scriptname
2."just the name" method $ scriptname
3.in the background $ scriptname &
25
Executing a Shell Script
• Method 1 runs the command as if you typed them in on the command line – No ksh subprocess created
• Note that methods 2 and 3 require: – execute permission for scriptname
chmod +x scriptname – current directory (.) must be in PATH or else must use
./scriptname
26
Executing a Shell Script • Demonstration of execution methods:
$ ps PID TTY TIME CMD 31612 ttyq5 0:00 sh 31631 ttyq5 0:00 ps $ ls -l showsh -rw-r--r-- 1 dje appl 43 Nov 25 1997 showsh $ cat showsh print Executing the shell script showsh ps $ . showsh Executing the shell script showsh PID TTY TIME CMD 31612 ttyq5 0:00 sh 31628 ttyq5 0:00 ps
27
Executing a Shell Script $ chmod u+x showsh $ showsh Executing the shell script showsh PID TTY TIME CMD 31578 ttyq5 0:00 ps 31612 ttyq5 0:00 sh 31635 ttyq5 0:00 sh
28
Echoing Script commands• For understanding or debugging scripts it is often
convenient to see the script commands shown on the monitor along with the output they produce.
• Accomplished using the set -x command at the beginning of the file
• To turn off the echoing, make the command set +x (counter-intuitive?)
• Also useful in batch job files and - on occasion - interactively
• On the next page a demonstration of the power of set -x is shown
29
Echoing Script commands $ see /homea/dje/ksh_script boba.osc.edu:0.0 a_opt.txt cba_opt.txt item.set.3.txt pid.txt b_opt.txt fant4.txt item.set.4.txt showsh.txt c_opt.txt item.intmod.txt no_opt.txt trap.txt $ cat see #!/bin/ksh pwd echo $DISPLAY ls *.txt $ cat see2 #!/bin/ksh set -x pwd echo $DISPLAY ls *.txt
30
Echoing Script commands $ see2 + pwd /homea/dje/ksh_script + echo boba.osc.edu:0.0 boba.osc.edu:0.0 + ls a_opt.txt b_opt.txt c_opt.txt cba_opt.txt
fant4.txt item.intmod.txt item.set.3.txt item.set.4.txt no_opt.txt pid.txt showsh.txt trap.txt
a_opt.txt cba_opt.txt item.set.3.txt pid.txt
b_opt.txt fant4.txt item.set.4.txt showsh.txt
c_opt.txt item.intmod.txt no_opt.txt trap.txt
31
Script Variables• User-defined Variables• Storing a Value into a Variable
– The assignment operator =– The read command– Command Substitution
• Null Variables• Special Shell Variables• Positional Parameters• Referencing Variables• Parameter Substitution• Exercises
32
User-defined Variables• The Korn shell includes the following types of variables:
– User-defined variables – Special shell variables
• User-defined variables can be initialized, used and changed from the command line or from within a shell script. – Not "declared" like in a programming language. Just type the name
of the variable– Korn shell scripts use string and integer variables
• A variable name can consist of the following: – letters, digits, and the underscore character – first character of a variable name must be a letter or an underscore
character
• A variable can be made read-only: once it is assigned a value, the value cannot be changed– "Archaic" method: readonly variable_name– Alternate method: typeset -r variable_name
33
Storing a Value into a Variable• There are several methods for variables to obtain values. The assignment operator =• Enter the name that you have chosen for the variable
followed by an equal sign and then the value that you want to store in the variable.
• To use the a variable's value, put the $ operator in front of it's name. (Read the $ as "contents of")
• The typeset command can also be used for variable assignment. More up-to-date procedure, typeset has many options.
• To assign a value to an integer variable, the assignment command must be proceeded by the word let
• On the next page, a script is shown that demonstrates all these approaches
34
Storing a Value into a Variable $ cat assign street=Elm print street <-- Most common error when using variables print $street my_name="John Smith" echo $my_name neph=Chris Sellgren print $neph typeset card=ace; print $card let num=57; echo $num let num=num+3; print $num $ assign street Elm John Smith assign2.ksh[9]: Sellgren: not found
ace 57 60
35
Storing a Value into a Variable Second method: read in a value for a variable
• So far have used print and echo (archaic) commands for printing out all the strings following them and then a carriage return
• There is a complimentary command read which input a value into variables. The user type in as many strings as there are variables on the read line, and then hits carriage return
• On the next page is a demo script indicating how the read command works.
36
Storing a Value into a Variable $ cat input print Please enter a ship name read ship print Please enter your full name read fname lname print Captain $lname, welcome aboard the $ship echo May I call you $fname? print Enter an integer; read num let num=2*num echo Twice your number is $num $ input Please enter a ship name Redoubt Please enter your full name Horatio Hornblower Captain Hornblower, welcome aboard the Redoubt May I call you Horatio? Enter an integer 27 Twice your number is 54
37
Storing a Value into a Variable Use command substitution. The string(s) that a Unix
command returns may be put into a script variable
• The following syntax is as follows: var_name=$(UNIX command)
• As with all the script commands, command substitution can be done interactively as well
• On the next page is a demo script illustrating this technique
38
Storing a Value into a Variable 136:oscb$ cat command dir=$(pwd) print I am in the directory $dir weekday=$(date +%A) print It is $weekday files=$(ls -lt | wc -l) print There a $files files in this directory me=$(whoami) print I am $me computer=$(hostname) print My computer is called $computer logged_on=$(who | cut -f1 -d' ') print These people are also on $computer: $logged_on 137:oscb$ command I am in the directory /home/dje It is Tuesday There a 277 files in this directory I am dje My computer is called oscb These people are also on oscb: osu2796 opr006 paul ysu039 osu2796 jimg
mmittal jimg mmittal osu2224 osu2224 srb cls038 osu1938 jeff dje
39
Null Variables
A variable can be set to a null value, even if previously assigned, using any of the following methods. There should be no spaces preceding or following the equal sign. If the contents of null character variables are printed out, a blank appears. If the contents of a “nullled” integer variable is outputted, a 0 is displayed.
$ name=
$ name=''
$ name="”
$ let num=
$ unset varname
40
Null Variables
All variables that don't exist are assumed null unless set -o nounset is used. Then the shell will indicate an error when an undefined variable is encountered
$ unset name $ set -o nounset
$ print $name
ksh: name: parameter not set
$ set +o nounset
41
Special Shell Variables
• In addition to your own user-defined variables, other variables defined by the shell for the user are called special shell variables and are most useful
• These variables are set and updated automatically by the shell. Their values cannot be changed, but they may be referenced.
• The special shell variables of most use are identified by the following symbols: $#
$-
$?
$$
$!
$0
$*
42
Special Shell Variables
• The variable $# contains the number of arguments typed on the command line.
$ cat numargs print The number of arguments is $#
$ numargs
The number of arguments is 0
$ numargs 1 2 3 4 5
The number of arguments is 5
$ numargs "Hello World"
The number of arguments is 1
43
Special Shell Variables
• The variable $- contains the shell flags (options) of the current shell.
$ print $-
isum
$ set +u
$ echo $-
ism
44
Special Shell Variables
• The variable $? contains the exit status of the last command. If the exit status is 0, the command ran successfully. If it is non-zero, there was a problem
$ ls
file1 data account.txt
$ rm file1
$ print $?
0
$ rm dataa
dataa: No such file or directory
$ print $?
2
• We will see exit status again. It will be used to define true and false. Also, the user can set the exit status in their script by using the exit command
45
Special Shell Variables
• The variable $$ contains the Process ID (PID) of the current shell process.
$ cat pid #!/bin/ksh ps print $$ $ pid PID TTY TIME CMD 34738 ttyq5 0:00 ps 34744 ttyq5 0:00 sh 34765 ttyq5 0:00 pid 34765 $ . pid PID TTY TIME CMD 34743 ttyq5 0:00 ps 34744 ttyq5 0:00 sh 34744
46
Special Shell Variables
• The variable $! contains the process ID number of the last command sent to the background.
$ sleeper & [1] 29144 $ ps PID TTY TIME CMD 29144 ttyq5 0:00 sleeper 29150 ttyq5 0:00 sh 29160 ttyq5 0:00 ps 29167 ttyq5 0:00 sleep $ compress moby_dick & [2] 29136 $ print $! 29136
47
Special Shell Variables
• The variable $0 contains the name of the script currently being execute
$cat old_script print The name of this script is $0 $old_script The name of this script is old_script $mv old_script new_script $new_script The name of this script is new_script
• The variable $* contains all the arguments on the command line.
$ cat args print The arguments are: $*
$ args bob dave
The arguments are: bob dave
48
Positional Parameters
• For a script, the command line arguments are also called positional parameters. Each positional parameters refers to one individual argument on the command line. The ten positional parameters available to the script writer are referenced as follows:
$1 $2 $3 $4 $5 $6 $7 $8 $9 • The parameter $1 contains the first argument, $2 the second
argument, and so on. Consider the following script:$ cat parms
print Arg 1 is: $1
print Arg 2 is: $2
print Arg 3 is: $3
print Arg 4 is: $4
print Arg 5 is: $5
49
Positional Parameters
$ parms 2001 A Space Odyssey
Arg 1 is: 2001
Arg 2 is: A
Arg 3 is: Space
Arg 4 is: Odyssey
Arg 5 is:
$ parms "Space, the final frontier"
Arg 1 is: Space, the final frontier
Arg 2 is:
Arg 3 is:
Arg 4 is:
Arg 5 is:
50
Positional Parameters
Use the set command to change positional parameters. It replaces existing positional parameters with new values
$ cat newpos print starting args are $*
print number of args is $#
print arg 1 is $1
print arg 2 is $2
set NCC 1701 Enterprise
print new args are $*
print number of args is $#
print arg 1 is $1
print arg 2 is $2
print arg 3 is $3
51
Positional Parameters
Here is an example of the use of the newpos script:
$ newpos 76 trombones
starting args are 76 trombones
number of args is 2
arg 1 is 76
arg 2 is trombones
new args are NCC 1701 Enterprise
number of args is 3
arg 1 is NCC
arg 2 is 1701
arg 3 is Enterprise
52
Positional Parameters
• What if there are more than 10 positional parameters?
$ cat ten_args print arg 10 is $10
$ ten_args a b c d e f g h i j
arg 10 is a0
• Use the shift command to perform a shift of the positional parameters n positions to the left.
(Default n=1)• The shell variables $# and $* also change when the shift
command is used (along with the values of the positional parameters.
• Once a default shift is performed, the first parameter is discarded unless it is saved in a user-defined variable
53
Positional Parameters
Examples of shift usage: $ cat shift_it print $#: $0: $*
shift
print $#: $0: $*
shift
print $#: $0: $*
shift 3
print $#: $0 $*
$ shift_it It was the best of times, it was the worst of times.
12: shift_it: It was the best of times, it was the worst of times
11: shift_it: was the best of times, it was the worst of times
10: shift_it: the best of times, it was the worst of times
7: shift_it: times, it was the worst of times
$ cat ten_args
arg1=$1
shift
print $arg1 $*
$ ten_args 1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
54
Referencing Variables• So far we have seen that the syntax for referencing a
variable (obtaining its contents) is $var_name • An alternate syntax which has more capabilities is $
{var_name} This syntax:•
Allows a script to easily use more than 10 positional parameters Allows for more general use of a variable contents Allows for the number of characters in the variable’s string contents
to be determined Allows for parameter substitution: replacement of the value of a
variable with an different value based on specific conditions and events.
55
Referencing Variables Greater than 10 positional parameters:
$ cat moreargs #!/bin/ksh print arg 10 is $10 print arg 10 is ${10} print arg 11 is $11 print arg 11 is ${11}
$ moreargs what rough beast, its time come at last, slouches toward Bethlehem
arg 10 is what0 arg 10 is toward arg 11 is what1 arg 11 is Bethlehem
56
Referencing Variables More general use $ cat copy set -x file=lone cp $file $filewolf cp $file ${file}wolf ls lone* $ copy + file=lone + cp lone Insufficient arguments Usage: cp [-aDfirRp] [-b size] [-tP -[e size]] f1 f2 cp [-aDfirRp] [-b size] [-tP -[e size]] f1 ... f2 dir + cp lone lonewolf + ls lone lonewolf lone lonewolf
Length of a Variable’s contents $ cat length book="Pillars of the Earth" print length of book= ${#book} $ length length of book= 20
57
Parameter Substitution• Misnamed: should be variable substitution
• Four syntaxes for parameter substitution
${var_name: - value} ${var_name: = value} ${var_name: + value} ${var_name: ? value}
• The meanings and typical usages of these four methods are shown on the following pages
58
Parameter Substitution ${parameter:-value} • If parameter exists and isn't null, return its present
contents; otherwise return value. • Purpose: returning a default value if the variable is
undefined. • If parameter is null, the value returned is temporary. The
null value of the variable is not permanently altered• Interactive Examples:
$ count=""
$ print You have ${count:-5} cards
You have 5 cards
$ print $count
$ FOO=/usr/local/bin/xx
$ print edit file ${FOO:-$HOME/xx}
edit file /usr/local/bin/xx
$ print $FOO
/usr/local/bin/xx
59
Parameter Substitution
• As the name implies, this particular syntax of parameter substitution can be used positional parameters.
• Consider the following script that searches for a file:
$ cat fnddemo #!/bin/ksh set -x find ${2:-$HOME} -name $1 -print $ fnddemo bk.txt.Z . + find . -name bk.txt.Z -print ./bk.txt.Z $ fnddemo dale11.ps + find /homea/dje -name dale11.ps -print /homea/dje/dale11.ps
60
Parameter Substitution ${parameter:=value} • If parameter exists and isn't null, return its present
contents; otherwise set it to value and then return value • Purpose: Setting a variable to a default value if it is
undefined. • This form of parameter substitution cannot be used with
positional parameters because their values can only be changed with the set command
• Interactive Example:
$ count="" $ print You have ${count:=5} cards
You have 5 cards
$ print $count
5
61
Parameter Substitution ${parameter:+value} • This substitution format acts in an opposite manner to the
previous two.• If parameter exists and is not null, return value; otherwise
return null. • Purpose: Testing for the existence of a variable • Interactive Example: $ cat trace set -x trace=$1; print ${trace:+"Trace mode on"} $ trace y + trace=y + print Trace mode on Trace mode on $ trace2 + trace= + print
$
62
Parameter Substitution ${parameter:?message} • If parameter exists and isn't null, return its contents; otherwise
print scriptname: parameter: message • and abort the current script. Omitting message produces the
default system message. • Purpose: Catching errors that result from variables being undefined
which need to be.• Examples: $ cat fnd2 find ${2:-$HOME} -name ${1:?"No file specified"} -print $ fnd2 fnd2: 1: No file specified $ cat fnd3 find ${2:-$HOME} -name ${1:?} -print $ fnd3 fnd3: 1: parameter null or not set
63
Parameter Substitution
• NOTE: In all forms, the colon (:) is optional. If omitted, the operator tests for existence only, i.e., change "exists and isn't null" to "exists" in each definition.
64
Advanced Pattern Matching• Korn shell enhancements to the set of pattern-matching
operators:
Operator Meaning
${variable#pattern} If the pattern matches the beginning of the variable's value, delete the shortest
part that matches and return the rest.
${variable##pattern} If the pattern matches the beginning of the variable's value, delete the longest
part that matches and return the rest.
${variable%pattern} If the pattern matches the end of the variable's value, delete the shortest part that
matches and return the rest.
${variable%%pattern} If the pattern matches the end of the variable's value, delete the longest part that
matches and return the rest.
Mnemonic: # matches the front because number signs precede numbers; % matches the rear because percent signs follow numbers.
65
Advanced Pattern Matching
• Typical use of the above pattern-matching operators is stripping off components of pathnames such as directory prefixes and filename suffixes.
• Example: Assume the variable TEST has the value /home/job/src/long.file.name then:
Expression Result
${TEST##/*/} long.file.name
${TEST#/*/} job/src/long.file.name
${TEST} /home/job/src/long.file.name
${TEST%.*} /home/job/src/long.file
${TEST%%.*} /home/job/src/long
66
Exercises1. Write a script called lsc which executes the command ls -C. To
execute this command you must give the full path name to your lsc script. Make the lsc shell script executable and run it.
2.Write a script called b which changes your current directory to /bin, displays the current directory, and then executes the lsc script created above. Make the b script executable and run it. What is your current directory when the b script is finished executing? Why?
3.Write a script called ex_on, which turn on execute permission on the first argument on the command line.
4.Modify ex_on to turn on execute permission on all arguments on the command line.
5.Write a script called 12 that prints the twelfth argument on the command line.
67
Exercises6. Suppose you keep a file of your home coin collection that keeps track
of how many coins you have in a given category. Lines in this file look like:
62 U.S. proofs
11 U.S. pennies (1850-1908)
36 U.S. pennies (1909-1950)
9 U.S. nickels (1861-1938)
Write a script that prints the N types of coins of which you have the most. The default for N is 10. The program should take one argument for the name of the input file and an optional argument for how many lines to print.
68
Decision Making Commands• Flow Control• Exit Status II• if/then/else Command• Condition Tests• Logical Operators• Negation Operator• Illustrative Examples• Case Statement• Exercises
69
Flow Control• Flow control gives a programmer the power to specify that
only certain blocks of shell commands are executed
• Korn shell supports the following decision making constructs:
if/else Execute a list of statements if a certain condition is/is not true
case Execute one of several lists of statements depending on the value of a variable
70
Exit Status II• For decision making, we will look at the exit status in a
different way then we did before• The constructs if, while, until and the logical AND (&&)
and OR (||) operators use exit status to make logical decisions:
A 0 value is a logical "true" (success)
A nonzero is a logical "false" (failure)
• There are built-in true and false commands which you can use. The true command returns an exit status of 0, the false command returns an exit status of 1.
71
Exit Status of a Script• A script, like any other process, sets an exit status when it
finishes executing. Shell scripts will finish in one of the following ways:
Abort - If the script aborts due to an internal error, the exit status is that of the last command (the one that aborted the script).
End - If the script runs to completion, the exit status is that of the last command in the script
Exit - If the script encounters and exit command, the exit status is that set by that command
• Syntax for the exit command is exit [num]. When the exit command is encountered the script ends right there and the exit status is set to num
72
if / else
• Simplest type of decision making construct is the conditional embodied in Korn shell's if statement.
• Syntax:
if condition then
statements
[elif condition <-- can use multiple elif clauses
then statements . . .] [else
statements]
fi <-- must be by itself on the final line of the construct
73
if / else
• The if statement uses an exit status to determine whether or not to execute the commands.
• Statements are executed only if the given condition is true.
• If one or more elifs are used, the last else clause is an "if all else fails" part.
74
Condition Tests
• The if construct can only test exit status but that doesn't limit you to checking only whether commands ran properly or not.
• Using the [[ ]] operator, many different attributes can be tested: – string comparisons– file attributes– arithmetic conditions
• [[ condition ]] just returns an exit status that tells whether condition is true or not
• [[ ]] surround expressions that include various types of relational operators.
75
Condition Tests: Strings
String comparison operators:
Relational Operator True if ...
str = pat str matches pat.
str != pat str does not match pat.
str1 < str2 str1 is less than str2.*
str1 > str2 str1 is greater than str2.*
-n str str is not null (has length greater than 0).
-z str str is null (has length 0).
*based on the ASCII value of their characters
str refers to an expression with a string value, and pat refers to a pattern that can contain wildcards.
76
String Comparison Examples $ cat compstr x=castle; y=cast* if [[ $x = $y ]] then print $x matches $y else print $x does not match $y fi $ compstr castle matches cast* $ cat compstr1 x=hello; y=helper if [[ $x > $y ]] then print "$x is > $y" else print "$x is not > $y" fi $ compstr1 hello is not > helper
77
String Comparison Examples $ cat compstr2 if [[ -n $1 ]] then print First argument is $1 fi
if [[ -z $1 ]] then print Where is the first argument? fi $ compstr2 Logan First argument is Logan $ compstr2 Where is the first argument?
78
Condition Tests: FilesFile attribute checking:
Relational Operator True if ...
-a file file exists
-d file file is a directory
-f file file is a regular file
-r file there is read permission on file
-s file file is not empty
-w file you have write permission on file
-x file you have execute permission on file, or directory search permission if it is a directory
-O file you own file
-G file your group id is same as file's
file1 -nt file2 file1 is newer than file2
file1 -ot file2 file1 is older than file2
There are 21 such operators in total.
79
File Checking Demo Directory• Consider a directory that contains the following files and
subdirectories
$ls -lrt -rwxr--r-- 1 dje appl 5 Jun 27 2000 file -rw-r--r-- 1 dje appl 5 Jun 27 2000 my_prog -rw-r--r-- 1 dje appl 5 Jun 27 2000 my_scr.1 drwxrwsr-x 2 dje appl 512 Jul 3 2000 scripts -rw-rw-r-- 1 dje appl 5 Sep 4 2000 slide1.ps drwxr-sr-x 2 dje appl 512 May 18 11:54 probs drwxr-sr-x 2 dje appl 512 May 18 11:54 solutions -rw-r--r-- 1 dje appl 0 May 18 11:54 court.msg -rw-r--r-- 1 dje appl 0 May 18 11:55 heel -rw-r--r-- 1 dje appl 0 May 18 11:57 greed -rwxr-xr-x 1 dje appl 80 May 18 12:01 file.c -rwxr-xr-x 1 dje appl 80 May 18 12:01 file.f90 -rw-r--r-- 1 dje appl 0 May 18 12:04 typescript
80
File Checking Examples $ cat fexist if [[ -a $1 ]] then print File: $1 does exist else print File: $1 does not exist fi $ fexist sort.c File: sort.c does not exist $ cat ffull if [[ -s $1 ]] then print $1 has something in it else print $1 is empty fi $ ffull heel heel is empty $ cat fx if [[ -x $1 ]] then print I can execute $1 else print I cannot execute $1 fi $ fx file I can execute file
81
Condition Tests: Integers• Need to use a new test operator for integer comparisons (( … )) instead of [[ … ]]
Relational Operator Meaning
< less than
> greater than
<= less than or equal
>= greater than or equal
== equal
!= not equal
82
Integer Comparison Examples $ cat intcomp if (( $1 < $2 )) then print $1 is less than $2 fi if (( $1 <= $2 )) then print $1 is less than or equal to $2 fi if (( $1 == $2 )) then print $1 is equal to $2 fi if (( $1 > $2 )) then print $1 is greater than $2 fi if (( $1 >= $2 )) then print $1 is greater than or equal to $2 fi if (( $1 != $2 )) then print $1 not equal to $2 fi
83
Integer Comparison Examples
$ intcomp 3 89 3 is less than 89 3 is less than or equal to 89 3 not equal to 89
$ intcomp 13 13 13 is less than or equal to 13 13 is equal to 13 13 is greater than or equal to 13
84
Logical Operators
• Two special symbols allow the user to combine relational tests into more complex conditions
&& is a logical AND
|| is a logical OR
• How they work:
statement1 && statement2
statement2 will execute only if statement1 was successful (returned an exit status of 0)
statement1 || statement2
statement2 will execute only if statement1 was not successful (returned a nonzero exit status).
85
Logical Operators• The logical operators && and || are commonly used with
if/else commands • Examples: $ ls -l -r-xr-xr-x 1 dje appl 365 May 18 15:13 intcomp -rw-r--r-- 1 dje appl 209 May 18 15:12 length.txt $ cat setw #!/bin/ksh if [[ -f $1 && -x $1 ]] then print You are getting write permission for $1 chmod u+w $1 fi $ setw length.txt $ setw intcomp You are getting write permission for intcomp $ ls -l -rwxr-xr-x 1 dje appl 365 May 18 15:13 intcomp -rw-r--r-- 1 dje appl 209 May 18 15:12 length.txt
86
Logical Operators $ cat gone #!/bin/ksh if [[ $1 = quit || $1 = bye ]] then print You have closed correctly else print Improper closing word exit 11 fi $ gone bye You have closed correctly $ echo $? 0 $ gone quit You have closed correctly $ echo $? 0 $ gone end Improper closing word $ echo $? 11
87
The Negation Operator• If an exclamation point ! is put before any of the conditions
discussed, it reverses the “truth or falseness” of the condition.
• [[ ! -d probs ]] returns true (exit status 0) if probs is not a directory
• [[ ! Bob = ?o? ]] returns false (exit status 1) even though Bob matches the pattern
88
Combining Different Types of Conditions
• If you want to combine an integer comparison with file or string comparison within the same condition, you must do two things:– Use [[ … ]] for the test operator– Use the integer relational operators shown below
Relational Operator Comparison
-lt less than
-le less than or equal
-eq equal
-ge greater than or equal
-gt greater than
-ne not equal
89
Combining Conditionals Example $ cat fileint #!/bin/ksh let num=$2 if [[ $1 < tact || num -ge 3 ]]; then print Pippin else print Merry fi $ fileint ace 45 Pippin $ fileint vice 2 Merry
90
Illustrative Examples
Test operator run interactively $ [[ -z "" ]]
$ print $?
0
$ [[ -z foo ]]
$ print $?
1
Treating integers strings as strings: compressed format
$ if [[ 6 > 57 ]] ; then print huh?;fi
huh?
91
Illustrative Examples
The following script sets user execute permission on an ordinary, file that is not a directory:
if [[ -f $1 && ! -d $1 ]]; then chmod u+x $1
fi
92
Illustrative Examples
The following script removes the first file if it's older than the second file and the variable KEY is non-null:
if [[ $1 -ot $2 && -n $KEY ]] then
/bin/rm $1
fi
93
Illustrative ExamplesThe following script compares two files and if no differences are
found, removes the second file. Also helpful USAGE programming style:
USAGE="Usage:\t $0 file1 file2"
if [[ $# -ne 2 ]]; then
print ${USAGE}
exit 1
fi
diff $1 $2 > /dev/null
if [[ $? -eq 0 ]]; then
/bin/rm $2
print $2 removed, $1 kept
else
print $1 and $2 differ
fi
94
Case command• Provides a multiple choice decision structure. When
executed one of several statement blocks will be run.• Replacement for the “elif-ladder”• Lets you test strings against patterns that can contain
wildcard characters.
• Syntax: case expression in pattern1)
statements ;;
pattern2)
statements ;;
...
esac
95
Case Operation
• If expression matches one of the patterns, its corresponding statements are executed
• If there are several patterns separated by pipes, the expression can match any of them in order for the associated statements to be run ( | acts as logical or )
• Patterns are checked in order for a match; if none is found, nothing happens
• When the double colons (;;) are encountered at the end of the one of the statement lists, the case terminates
96
Case Examples
Here's a simple script which moves C and Fortran source files to one directory and object code files to another:
case $file in
*.c|*.f)
/bin/mv $file ${HOME}/src ;;
*.o )
/bin/mv $file ${HOME}/obj;;
* )
print $file not moved ;; (Default value or USAGE)
esac
(Could have also used *.[cf] wildcard construct above)
97
Case Examples• The case statement is often used to specify options for a
shell script. • Here is a shell script called dt_fmat that allows the user to
enter options that affect the way the date is displayed: case $1 in -d) print -n Date: date +"%a %h %d" ;; -t) print -n Time: date +"%T" ;; -w) print -n Weekday: date +"%a" ;; -y) print -n Year: 20 date +"%y" ;; -m) print -n Month: date +"%h" ;; *) print -n Date: date +"%a %h %d" print -n Time: date +"%T" ;; esac
98
Case Usage $ date Sat May 19 12:01:57 EDT 2001 $ dt_fmt -d Date:Sat May 19 $ dt_fmt -t Time:12:02:05 $ dt_fmt -w Weekday:Sat $ dt_fmt -y Year: 2001 $ dt_fmt -m Month:May $ dt_fmt Date:Sat May 19 Time:12:02:54
99
“Elif-Ladder” vs Case• The following script shows and “elif-ladder” and a case
statement that do the exact same thing. Note how the case statement has a simpler syntax
if [[ $1 = cat || $1 = dog ]]; then print You have a normal pet elif [[ -z $1 ]]; then print You have no pets else print You have a wierd pet fi
case $1 in cat|dog) print You have a normal pet ;; '') print You have no pets ;; *) print You have a wierd pet ;; esac
• Advantage of “elif-ladder”: not limited to string comparisons
100
Exercises
1.Write a script called char that checks a single character on the command line, c. If the character is a digit, digit is displayed. If the character is an upper or lowercase alphabetic character, letter is displayed. Otherwise, other is displayed. Have the script print an error message if the argument c is more than one character in length.
2.Write a script called mycopy that copies a source file to a directory. Add a check to see if the source file exists. If the source file does not exist, print an error message.
3.Write a script called mydir that prints the message File is a directory if the file is a directory.
101
Exercises
4.Write a script called ver that accepts one argument on the command line indicating the name of a file. The file is copied to another file with the same name with the addition of the extension .v2. Also, the line #Version 2 is added to the top of the file.
5.Execute the ver script on itself, creating a new version of the file called ver.v2.
6.Rewrite ver.v2 to accept a possible second argument. If two arguments are entered, the file specified by the first argument is copied to a file with the name of the second argument. If no second argument is entered, the file is copied to another file with the
same name, adding the extension .v2. In either case, the line #Version2 is added to the top of the file.
102
Looping Commands• for• select• while • until• Exercises
103
for Loop Command• The for loop allows a section of commands to be run a fixed
number of times• During each time through an iteration, the loop variable is set to
a different value. • Syntax:
for name [in list] do
statements that can use $name
done
where— name is the loop variable which can be called anything — list is a list of strings (defaults to $*) separated by spaces
• Operation: name is set to each string in list, in order, for each iteration; the number of iterations equals the number of names in list.
104
for Loop Examples
Outputting a Phrase $ cat simple for word in This is a test
do
print $word
done
$ simple
This
is
a
test <-- Omitting the in This is a test phrase
and instead running the script as
$ simple This is a test
would yield the same output.
105
for Loop Examples
The list can be the contents of a local variable:
Check to see who is logged on the machines listed in the variable
SYSTEMS="myrtle gull sandy newport daytona":
for sys in $SYSTEMS
do
finger @$sys
done
106
for Loop Exampleslist can contain shell wildcards and command substitution as well: $ cat file1 for i in *
do
print $i
done
$ file1
file1
file2
$ cat file2
for i in $(ls)
do
print $i
done
$ file2
file1
file2
107
for Loop Examples• Can explicitly make the list all the command line arguments (also
notice the alternate syntax):
$ cat upper #!/bin/ksh for bug in $* ; do typeset -u day=$bug print $day done $ upper Robson Bond Green ROBSON BOND GREEN
108
select Command
• select allows you to generate simple interactive ASCII menus easily.
• Syntax: select name [in list] do
statements that can use $name
done
• This is the same syntax as the for loop except for the keyword select. As with for, in list defaults to $* if omitted.
109
select Operation• Each item in list is displayed as the choice in the menu with a
number preceding it. At the bottom of the menu the PS3 prompt is displayed and script is waiting for the user to select a choice by entering its number. (A default value for the PS3 prompt is provided by the system but the user can change it)
• The selected choice is stored in name and the selected number in a system-predefined variable called REPLY
• The statements in the body of the do loop are run
• Repeats the process forever; exit loop with break statement (or user can issue ctrl-d)
110
select ExampleThe following script termselect allows you to select a terminal
setting:
PS3='terminal? ' select term in vt100 vt220 xterm
do
if [[ -n $term ]]; then
TERM=$term
print TERM is $term
else
print invalid choice
fi
done
111
select Example
$ termselect 1) vt100
2) vt220
3) xterm
terminal? 4
invalid choice
terminal? 3
TERM is xterm
112
select Example $ cat travel select i in Past Present Future; do print You have picked choice $REPLY print "You will go to the $i" break done $ travel 1) Past 2) Present 3) Future #? 1 You have picked choice 1 You will go to the Past $ travel 1) Past 2) Present 3) Future #? 3 You have picked choice 3 You will go to the Future $
113
while Command• Allows a section of code to be run repetitively while a certain
condition holds true. • Syntax:
while condition do
statements ...
done
• Operation: while condition remains true (exit status 0) the body of the while loop is run
• [[ ]]operator can be used here as with if to specify the condition
• Beware of creating an infinite loop (condition must become false at some point).
• If condition is false when a while loop is first encountered, the loop will not be executed even once
114
while Example
Example: print out positional parameters: $ cat args while [[ $# -ne 0 ]]; do
print $1
shift
done
$ args There are 7 “deadly sins”
There
are
7
deadly sins
115
while Example
Example: Using a while as a counting loop:
$ cat whileint print $#: $3 let i=1 while (( i <= 5 )); do shift let i=i+1 done print $#: $3 $ whileint 1 3 5 7 9 11 13 15 17 19 10: 5 5: 15
116
until Command• Allows a section of code to be run repetitively until a certain
condition becomes true. • Syntax:
until condition do
statements ...
done
• Operation: while condition remains false (exit status nonzero) the body of the until loop is run
• [[ ]]operator can be used here as with if to specify the condition
• Typically, condition is false when an until loop is first encountered; thus the loop will not be executed at least once
• Just about any until can be converted to a while by simply negating the condition
117
until Example• Classic example: checking user input $ cat prompt #!/bin/ksh until [[ $ans == no || $ans == yes ]]; do print -n 'Continue? (yes or no): ' read ans done $ prompt Continue? (yes or no): y Continue? (yes or no): sure Continue? (yes or no): ok Continue? (yes or no): yes $ prompt Continue? (yes or no): n Continue? (yes or no): nope Continue? (yes or no): no way Continue? (yes or no): no $
118
Exercises
1.Write a script called lis that uses a for loop to display all files and directories in the current directory.
2. Write a script called dirs that displays only the names of the directory files in your current directory.
3. Using the select command, write a script that creates a menu of the abbreviations for all the months of the year. If a certain month abbreviation is chosen, the script should print out the full name of the month and and number of days in it. Change the PS3 prompt to something appropriate.
4. Using a while loop, write a script that displays a guessing game. The player is to guess a number between 0 and 59. If the guess is too low, the message “Higher” is printed. If the guess is too high the message “Lower” is printed. Guesses continues in until the player guesses the correct number. The number the user is trying to guess is taken from the seconds of the date command.
119
Exercises 5. Modify the script from problem 4 to keep track of the number of
guesses it takes for the user to discover the correct number. Have the script print the total number of guesses that it took.
6. Rewrite the first while example script shown in this chapter (printing out the positional parameters) using the until statement instead.
120
Working with Integer Variables• Arithmetic Operations• Operator Precedence• Numerical Bases• String Variables vs Integer Variables• Exercises
121
Arithmetic Operations• So far we have seen how to give values to integer variables and
compare them (in two different ways!)
• In this section, integers will be used in arithmetic operations
• Below is a list of the most common operations performed with integers
• Arithmetic Operator Meaning
+ addition
- subtraction
* multiplication
/ division with truncation
% modulus division
• The shell interprets symbols surrounded by $(( and )) as arithmetic expressions.
• Variables in arithmetic expressions do not need to be preceded by dollar signs.
•
122
Arithmetic Examples Demonstration of the integer divisions: $ cat intop let x=31 let y=8 print The integer division of $x by $y is $((x/y)) print The modulus division of $x by $y is $((x%y)) $ intop The integer division of 31 by 8 is 3 The modulus division of 31 by 8 is 7
Parentheses can give operations higher precedence: $ cat paren #!/bin/ksh let x=$(( $1 + $2 * $3 - $4 )) print $x let x=$(( ($1 + $2) * ($3 - $4) )) print $x $ paren 2 4 6 3 23 18
123
Different Numerical Bases Possible to represent integers in alternate bases: $ cat base #!/bin/ksh print -n $(( 57 )) print -n , print -n $(( 16#39)) print -n , print -n $(( 8#071)) print -n , print -n $(( 2#00111001)) print $ base 57,57,57,57 $
124
String Variables vs Integer Variables Arithmetic does not work with string variables: $ cat intstr print As integers ... if (( $1 < $2 )) then print $1 is less than $2 fi let x=$2+4; print $2+4 is $x print As strings ... if [[ $1 > $2 ]] then print $1 is greater than $2 fi x=$2+4; print $2+4 is $x $ intstr 3 15 As integers ... 3 is less than 15 15+4 is 19 As strings ... 3 is greater than 15 15+4 is 15+4
125
Exercises
1. Write a script which finds the sum of all the positional parameters.
2. Write a script to determine if the first positional parameter is prime.
3. Write a script to perform the factorial of the first positional parameter
4. Write a script that will output positive and negative integers.
5. Write a script that will convert the first positional parameter from base 10 to base 5.
6. Write a script that reads in a time in seconds and outputs as hours:minutes:sec.
126
Script Input / Output• Input / Output Redirection• “Here-is” documents• print• read• Quoting• Exercises
127
Input / Output Redirection• The basic I/O redirectors >, < and | are good enough for most
tasks.
• The Korn shell supports 16 redirectors in total.
• Here are some other useful ones:
Redirector Function
>> file Direct stdout to file, appending to file if it already exists
>| file force stdout to file even if noclobber is set
<< label “here-is” document (see below)
n> file direct file descriptor n to file
n<file set file as file descriptor n
>&n duplicate stdout to file descriptor n
<&n duplicate stdin from file descriptor n
128
Input / Output Redirection
Example:
This script called start which can be used to start a different script (entered as positional parameters) in the background and save both stdout and stderr in a single log file.
$* > logfile 2>&1 &
If we want to do the same as above but to the screen as well:
$* 2>&1 | tee logfile &
The command tee takes its stdin and copies it to stdout and to the file given as argument.
129
“Here-is” documents
• Special form of inline input redirection
• Syntax:
command << label input for command
....
label
• The shell performs command and variable substitution where appropriate unless label is surrounded by quotes
130
print• We have already seen many examples of the print command.
• Here are some useful escape sequences and options; enclose escape sequences in quotes (either double or single):
Escape Sequence Character printed
\a bell
\b backspace
\r return
\t tab
Option Function
-n omit the final newline
-r ignore any escape sequence
-s print to history file
-un print to file descriptor n
131
print Example $ cat escape #!/bin/ksh print "Call me \t Ishmael" print -r "Call me \t Ishmael" $ escape Call me Ishmael Call me \t Ishmael $ cat print.u3 #!/bin/ksh print -u3 "My errors:" 3>errors $ ls errors Cannot access errors: No such file or directory $ print.u3 $ cat errors My errors:
132
read
• We have seen simple examples of read.
• Shell scripts with read are useful for certain tasks such as reading in lines from a file
• Suppose we have the file /etc/nicks : myrtle beach_bum
killian brewster
daytona speedster
...
which is a list of hostnames followed by nicknames. We want to assign the variable NICKNAME to the appropriate nickname depending on which machine you log into.
133
read
• The following code will do it:
machine=$(uname -n) while read host nick; do
if [[ $host = $machine ]]; then
NICKNAME=$nick
print NICKNAME is $NICKNAME
break
fi
done < /etc/nicks <-- read exits with status 1 when there is
nothing left to read
134
Quoting• A source of confusion for novice and experienced Korn
shell script writers is when to use single quotes ‘ ‘ and when to use double quotes “ “
• Throughout the workshop we have learned that certain symbols have special meanings: $name indicates the contents of the variable name, $(command) indicates the output of a command, etc.
• Single quotes are strong quotes: they prevent the special meaning symbols from performing their tasks. The symbols are treated just like normal characters.
• Double quotes are weak quotes: they allow the following actions to take place:– variable and positional parameter substitution– command substitution– arithmetic calculations
135
Quoting Demonstration
$ cat quote #!/bin/ksh money=franc print "Nonsense words ${money} $1 $(date +%T) $((3*75))" print 'Nonsense words ${money} $1 $(date +%T) $((3*75))' $ quote raven Nonsense words franc raven 16:23:30 225 Nonsense words ${money} $1 $(date +%T) $((3*75))
136
Exercises
1.Write a shell script tfile which creates a test file called junk. Do not use a cp or mv command to accomplish this. File junk contains the following:
This is a test
abcdefghijklmnopqrstuvwxyz
0123456789
end of test
2. Create three files called data0, data1, and data2 which contain the month, day, and year respectively of your birthday. Write a single script that reads from all three of these files and then outputs your birthday to the screen
137
Communicating with Running Scripts• Sending Signals• Signal Handling
138
Signal Handling• We want to have control over our shell scripts even when they are
interrupted by signals.
• Signals come in the form of control keys, the kill command and user-initiated events like logging out.
• Programs can be set up to "trap" specific signals and process them in their own way.
• Syntax of trap command:
trap command sig1 sig2 ...
when any of sig1, sig2, etc., are received, run command, then resume execution
•
139
Signal Handling
• After command finishes, the script resumes execution just after the command that was interrupted.
• sig1, sig2, ... can be given by name or number (name is more portable).
• trap without arguments lists traps that have been set
140
Signal Handling
Simple example - shell script loop : trap 'print You hit ctrl-c!' INT
while true: do
sleep 60
done
ctrl-c will not kill this program but rather print the message and start another sleep; type ctrl-\ to stop it (then rm core).
141
Signal Handling
Enhancing the script to handle background execution (trap the kill command):
trap 'print You hit ctrl-c!' INT
trap 'print No kill I!' TERM
while true: do
sleep 60
done
if run as loop &, kill %loop won't stop it; need kill -KILL %loop
Note: kill -l will give the numbers associated with given signals (defined in
/usr/include/sys/signal.h).
142
Signal Handling Examples: • If any of the signals listed (0, 1, 2, 3, or 15) are received
by the program, file temp$$ will automatically be removed. Without the exit command, program execution would
continue from the point where the signal was received:
trap 'rm -f temp$$; exit 0' 0 1 2 3 15
• The command listed for trap is null. This specifies that the signals listed are to be ignored, and the program will try to continue where it left off:
trap "" 0 1 2 3 15
• Using the null command (:) specifies that the signals listed are to be ignored for the next command only:
trap : 0 1 2 3
143
Signal Handling
• What if you want a program running in the background to continue even after you log out?
Shell sends the HUP (hangup) signal to all of your background jobs when you log out.
• Use the command nohup to put a job into the background which will ignore HUP and QUIT (ctrl-\) signals:
nohup $* > logfile 2>&1 &
144
Signal Handling
Most common signals:
Signal Name Note
1 HUP Hangup
2 INT ctrl-c
3 QUIT ctrl-\
9 KILL Cannot be trapped
15 TERM default kill signal
24 TSTP ctrl-z
145
Script Functions• Why Use Functions?• Syntax & Use• Auto-Loading• Function Definition Location• Command Precedence• Exercises
146
Functions
• Improves shell's programmability– already in memory (unless autoloaded)– modular programing
• Syntax: function functname { shell commands
}
or,
functname () {
shell commands
}
147
Functions
• Delete a function definition with
$ unset -f functname
• Display all defined functions:
$ functions
148
Functions • Two important differences between functions and shell
scripts run by name (Method 2):– functions do not run in a subprocess; behave more like a script run
by Method 1 – functions have precedence over scripts
• Where to put them?
– enter on command line (no longer available if you log out and log back in)
– .profile (only available in login shell) – $ENV file (available to all child shells) – in $FPATH directory (Korn shell will use autoload feature)
• Function names should begin with a letter or underscore • The closing brace (}) must appear on a line by itself
149
Functions
Example: Run a program and view the output
$ myrun ()
> {
> cd $WORKDIR
> a.out < data.in
> more data.out
> /bin/rm data.in
> mv data.out data.in
> }
150
Exercises
1.Write a script which sets the PAGER environment variable. The script should accept input values from:
• command-line arguments: • -p pager or -ppager
• shell environment variable PAGER
• default setting: more -s
Include a usage line. (Hint: You should try using the pattern matching operators introduced in this section)
2. Write a script that implements a simple version of the shell's built-in whence command. It should check each of the directories on your PATH for a command given as an argument, and print the full pathname if the command is found. Only files that are executable by you (and are not directories!) should be returned.
151
Exercises
3. Design functions pushd and popd which implement a "stack" of directories (as in the C shell).
• The stack is just a list of directories, stored in an environment variable we'll call DIRSTACK.
• pushd should "push" a specified directory onto the stack, i.e., prepend the directory name to DIRSTACK.
• popd "pops" a directory off the stack, i.e., cds to the top directory on the stack and then removes that directory from the stack.
• Your functions should: – Allow the use of "." to specify the current directory. – Return an error message if pushd is invoked without an argument, or if the
specified directory does not exist. – popd should return an appropriate message and do nothing if the stack is
empty.
• Why are these implemented as functions rather than scripts?
• What modifications (if any) should you make to your ~/.profile in order to use them?
152
Exercises
4. Write a script that will change the suffix of all the files in the present directory from .old to .new
153
HPC Designed Scripts• Repeating Program Execution• Debugging Options• Batch Jobs• Performance Tool Preprocessing• Multiple Tool Runs• Multiple Tool Reports• File Renaming
154
Repeating Program Execution• It is often desired to run a program several times. Even a serial programs
may have to run repeatedly to debug intermittent problems, find logical errors, etc.
• In HPC parallel processing arena, there are even more reasons to write a script that will execute code many times:
– Each run is with a different number of processors so that the traditional speedup graph data can be collected (and compared with Ahmdal’s Law)
– To perform repeated timing of the entire or partial code. • Several parallel machines do not allow dedicated use of processors• A severe load imbalance may come and go depending on how concurrently the various
processors actually work• Over synchronized program’s times can vary due to the order of arrival of the processors at
barriers
– To gather data from program output for statistical analysis. One can also gather timing data for similar reasons.
– To confirm that a parallel program is preventing race conditions which can arise when one processor is interrupted by another when the former is trying to read, update, and write back to a shared memory location
• Repeated runs is easily done in our introductory triangle script by using a while loop. Here the script will run the timearea code which times how long the program takes to calculate the area
155
Timing Script $ cat triangle cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1 $1.c cat << EOF > sides.in 8 3 7 EOF let i=0 while (( i < $2 )); do $1 < sides.in >> $1.dat let i=i+1 done cat $1.dat rm $1 sides.in
156
Timing Script Usage & Output $ triangle timearea 17 1.215377 1.227504 1.197454 1.193690 1.220033 1.190733 1.194406 1.221519 1.201336 1.187614 1.197026 1.220640 1.217389 1.218919 1.192417 1.218107 1.228289
157
Debugging Options• To use debugging software (dbx, Totalview, Xbrowse, etc)
programs need to be compiled with certain options.
• The debugging option is traditionally –g but has been improved lately to include various levels of optimizations while the debugging is being performed (G0,G1,…)
• For all debugging options, the compiler will make a symbol table which connects a program identifier (variable name, etc.) with a memory address. This allows the debugger to track the values of any variables as the program is executed line by line
• On next page we show a variation of the introductory triangle script in which a decision is made- through the use of a second positional parameter- if debugging will be implemented
158
Debugging Script $ cat triangle if [[ $2 = debug ]]; then cc -g -o $1 $1.c print "$1 ready to be debugged" else cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1
$1.c $1 < sides.in > $1.dat cat $1.dat rm $1 fi $ triangle area Area of Triangle is 10.392305 $ triangle area debug area ready to be debugged
159
Batch Jobs• There are a number of popular batch processing software
that are heavily used on HPC machines: NQE, PBS, and LSF to name three.
• At this point in the course, we now realize that the “batch files” we are used to writing are actually just shell scripts. The header lines have the syntax and are treated as comments if the batch script is not submitted to the batch software.
• But since these batch scripts have designed purpose – to run the UNIX commands in them and send a log file back-we can tailor them to reflect it.
• For example, it is recommended that the shell command echoing feature be turned on so that the commands appear in the log file preceding their output. An example is shown on the next slides.
160
A Batch Script• Here is a simple batch job (file sp_gravity.job):
#QSUB -r grav#QSUB –lT 3000#QSUB –lM 10Mw#QSUB -oeset -x # echo commandscd gravity/batchls -lrtf77 -o spg sp_gravity.f./spg < region.rspls –lrt
161
A Log file (grav.o3613)+ ls -lrttotal 1512-rw-r----- 1 dje G-541 46098 May 18 12:22 sp_gravity.f-rw-r----- 1 dje G-541 637236 May 18 12:22 newbedrx.cray-rwxr----- 1 dje G-541 299 May 18 12:31 region.prm-rw-r--r-- 1 dje G-541 16 May 18 12:31 region.rsp-rw-r--r-- 1 dje G-541 125 May 18 12:55 sp_gravity.job+ f77 -o spg sp_gravity.fsp_gravity.f: MAIN xy2glqr:"sp_gravity.f", line 346: Warning: local variable "zero" never used gauleg:+ ./spg+ 0< region.rsp Parameter file: Gravity output file: region.out 3-D ORTHOGONAL COLLOCATION ON FINITE ELEMENTS ********** (GAUSS-LEGENDRE QUADRATURE) ... Integration over source region in progress . . .
162
Log file (cont.)+ ls -lrt
total 1712
-rw-r----- 1 dje G-541 46098 May 18 12:22 sp_gravity.f
-rw-r----- 1 dje G-541 637236 May 18 12:22 newbedrx.cray
-rwxr----- 1 dje G-541 299 May 18 12:31 region.prm
-rw-r--r-- 1 dje G-541 16 May 18 12:31 region.rsp
-rw-r--r-- 1 dje G-541 125 May 18 12:55 sp_gravity.job
-rwxr-xr-x 1 dje G-541 67228 May 18 12:58 spg
-rw-r--r-- 1 dje G-541 2128 May 18 12:58 region.coeff
-rw-r--r-- 1 dje G-541 14502 May 18 13:00 region.out
163
qsub Command with options• The realization that a batch job is really just a normal shell
script, is clear when the script only consists of the Unix commands.
• The “header” section is replaced by options to the qsub command:qsub –r grav –lT 3000 –lM 10Mw –eo < grav.ksh
• Where grav.ksh is the remaining Unix commands:set -x cd gravity/batchls -lrtf77 -o spg sp_gravity.f./spg < region.rspls –lrt
• This is the alternative ( not recommended) method for running a batch job. It will return the same log file shown on the previous slides.
164
Advanced Batch Scripting• Since a batch script is – just that – a script, we can using
any part of the scripting language in it.
• Consider our “classic” compile and run script we have been using in the previous slides. If the compilation is a failure, there is no point in proceeding with the script’s commands since there will be no executable to work with.
• In the advanced batch script shown on the next slide we use the if/then shell command to check to see if there were no errors produced in the compilation. If there was, we exit the script (and the job,BTW).
• The log file returned after submitting this script is also shown. An error was deliberately inserted in the source code.
165
Error-Checking Batch Scriptset -x
cd SV1/dje/gravity
f90 -o spg sp_gravity.f
if [[ $? -ne 0 ]]; then
exit 33
fi
./spg < region.rsp
166
Error-Checking Log File+ cd SV1/dje/gravity+ f90 -o spg sp_gravity.f
do i=1, ^f90-197 f90: ERROR XY2GLQR, File = sp_gravity.f, Line
= 408, Column = 14 Unexpected syntax: "operand" was expected but found
"EOS".
Cray Fortran: Version 3.5.0.4 (u15f51p42138g41109a18)Cray Fortran: Mon Jun 10, 2002 17:13:34Cray Fortran: Compile time: 0.3487 secondsCray Fortran: 1130 source linesCray Fortran: 1 errors, 0 warnings, 0 other messages,
0 ansiCray Fortran: "explain f90-message number" gives more
information about each message+ [[ 1 -ne 0 ]]+ exit 33
167
Performance Tool Preprocessing• Almost all of the sophisticated software tools that exist on
HPC machines to profile your code and aid in serial, vector, and parallel optimization require the user to run the code and generate a raw data file. The tool subsequently analyzes the raw data under the user control (usually interactively)
• For some HPC software tools the time to gather the raw data can be long, especially if the code is complex with many functions and subroutines. Or the syntax required might be different than ever other optimization tool on that machine.
• Because of these reasons, it is often useful to perform the generation of raw data as a shell script.
• The script shown on the next slide, generates the raw data file for the prof tool on the Cray SV1.
168
Using the prof preprocessing scriptoscb:$ cat profraw.ksh
#! /bin/ksh
f90 –G1 –o $1 –lprof $1.f
./$1 < $1.in
# Creates the prof.data file
prof –x –m prof.data $1 > $1.prof.raw
# User can pick the raw output filename
rm $1
oscb:$ profraw.ksh poisson
oscb:$ profview poisson.prof.raw
169
Multiple Tool Runs• For HPC tools that access Hardware Counters on the actual
vectorizing processors or commodity Microprocessors (Pentiums, Itantiums, Ultra-Sparcs, MIPS R12K, etc.) there are too many counter chip to gather all the information with just one run of an optimization tool.
• Therefore, to access all the hardware use information, your code must be run several times. For Cray machines, 4 runs of hpm and perf are required. For Microprocessor-based MPP machines there a typically 64 counters only two of which can be read out in a single run of the appropriate tool (ssrun for the Origin 2K, collect for the Sun Fire 6800, etc.)
• Compiling and running an executable, with different options chosen to access various counters is an ideal task for shell programming.
• In the next slide, a script is shown that gathers all the hardware use info for Cray machines with the hpm and perf tools installed
170
total_hw.ksh#! /bin/ksh# First parameter is the source code “prefix”cc –O4 -TARG:madd=on -o $1 $1.c hpm –g0 ./$1 > $1.hpmhpm –g1 ./$1 >> $1.hpmhpm –g2 ./$1 >> $1.hpmhpm –g3 ./$1 >> $1.hpm# Recompile with perf optionscc –F –lperf –o $1 –O4 –TARG:madd=on $1.ctypeset –x PERF_GROUP=0 PERF_DATA=p0.raw ./$1typeset –x PERF_GROUP=1 PERF_DATA=p1.raw ./$1typeset –x PERF_GROUP=2 PERF_DATA=p2.raw ./$1typeset –x PERF_GROUP=3 PERF_DATA=p3.raw ./$1cat p0.raw p1.raw p2.raw p3.raw >perf.datarm $1
171
Saving Multiple Tool Reports• A powerful feature of HPC Optimization software tools is
that they provide information on a variety of performance topics with just one run of the tool. A certain tool might have a report of subroutine time-profiling, a caller-callee tree, a summary report, etc.
• When interactively using such a tool, a user can pick which report to read, and then pick another and so on.
• It is often desirable to just save a select set of the possible reports. This can be done by running the tool several times but with a different option each time. The option determines what report will be saved in that run. Again, a natural job for a shell script.
• In the next slide, a script is shown that saves several reports from the flowview tool used on Cray PVP machines. (12 reports are available from this tool)
172
flowview report selection scriptoscb:$ cat flowchoice.ksh
#! /bin/ksh
# Script assumes raw data file flow.data exists
# First parameter is prefix of the source code
pre=$1
shift
for opt in $x
do
flowview –L$opt >> pre.flow
done
oscb$ flowchoice.ksh neuralnet m h u k
173
Renaming Files• Specifically, renaming the prefixes or suffixes of a large
number of files. Many HPC parallel programs will produce hundreds of files which contain data at certain time steps, or spatial extent. Data visualization files are one example of this type of output.
• Say the user needs to put a new suffix or prefix on a large set of files in order that they not be overwritten by a subsequent data set.
• After this course, one’s mind should naturally think “ I bet a script could do this for me”. You would win that bet.
• On the next slide is shown a script that will change the suffix of all files whose name has the syntax strain001.txt, strain002.txt to strain500.txt to the same prefix but with “txt” replaced by “old”. The script can easily be made more general purpose as well.
174
Renaming scriptoscb$ cat suffix.ksh
#set –x
for file in $*
do
mv $file ${file%.*}.old
done
oscb$ suffix.ksh strain*.txt