shell introduction
TRANSCRIPT
Shell introduction
More features and gory details in ‘man sh’—hard to follow forbeginners, but hopefully much better after these notes.
I stick to old, basic Bourne shell (author’s name was Bourne) ‘sh’in this course for more fundamental understanding.
Most systems default to Bourne Again shell (a pun) ‘bash’. Manymore features and cursor editing.
There are others, e.g., csh, tcsh, zsh, fish.
Your homework submissions and test/exam answers must workunder the very basic sh. You may use other nicer shells otherwise.
1 / 41
CommentsA comment begins with ‘#’ and extends until end of line. Can bewhole line or begin from middle of line.
ls -l # this is a comment
2 / 41
VariablesType is string.
Set value: var=abcNote: No space around ‘=’
Read value: $var or ${var}(If uninit: get empty string.)
Why ‘${var}’ syntax provided: Try this example:
v=xxxv0=yyyecho $v0 # yyyecho ${v}0 # xxx0
3 / 41
“Simple” Commands“simple command” = command name, arguments, optionally [file]redirection (next slide).
Example (without redirection): tr -d 123
Command name has 4 cases, not apparent from syntax:
I Shell built-in command, e.g., ‘cd’I Shell function (user-defined).I Shell alias (user-defined) (omitted, but dead simple).I Program name, e.g., ‘tr’.
4 / 41
[File] RedirectionExample: tr -d 123 < infile > outfile
tr
stdin stdout
infile outfile
‘>’ overwrites. Use ‘>>’ instead if you want to append.
Redirect stderr: 2>
Redirect both stdout and stderr to same file:command > file 2>&1
5 / 41
PipelineE.g., sort | uniq
sort
stdin stdout
uniq
stdin stdout
Operator precedence: Lower than redirection.
6 / 41
[Command] List—Sequential CompositionMultiple commands can be put on multiple lines (especially in shellscript files).
Or, a single line but separated by semicolons. Example:
cd B09 ; ls ; ls -l ; cd ..
Either way, known as “list” or “command list”, sequentiallyexecuted: wait for one to finish before running the next.
Operator precedence: Lower than ‘|’.
Example: ls ; sort file1 | uniqRun ‘ls’, then run ‘sort file1 | uniq’.
7 / 41
Success, Failure, Exit CodeMost commands declare success or failure:
I Most commands fail upon non-existent pathnames.I ‘grep’ fails if nothing matches at all.
Conveyed by “exit code”. 0 for success, non-0 for failure. Become“true” and “false” for logical operators and conditional branching.
In C, recall “int main(...)”, that return value is exit code!
$? is most recent command’s exit code.
Beware: ‘echo $?’ is also a command! And it succeeds. Exercise:What does it print if you run it twice consecutively?
8 / 41
Logical AND, OR, NOTmkdir foo && cp myfile fooSequential execution, but stop upon first “false”.
mkdir foo1 || mkdir foo2 || mkdir foo3Sequential execution, but stop upon first “true”.
(So they are short-circuiting.)
! mkdir fooLogical not: turn 0 to 1, non-0 to 0.
Operator precedence:
‘;’ lower than ‘&&’, ‘||’‘&&’, ‘||’ same precedence, lower than ‘!’‘!’ lower than ‘|’
9 / 41
GroupingWhen operator precedence doesn’t work for you, write
{ list ; }
for explicit grouping. (Recall “[command] list”.)
Example:{ grep foo file1 ; ls ; } > file2
Again, may use newline instead of ;
Easy to miss: after last command, before }, you need newline or ;
10 / 41
Operator Summary And PrecedenceFrom highest to lowest precednece:
description operatorgrouping {}redirection < > >>pipeline |not !and, or && ||command list ; newline
11 / 41
Conditional Branchingif list1 ; then
list2elif list3 ; then
list4elselist5
fi
Easy to miss:
I before “then”, need ; or newlineI “elif”, not “else if”
Exercise: “else if” is not wrong, but what is annoying about it?
12 / 41
While-Loopwhile list1 ; do
list2done
May use ‘break’ and ‘continue’.
Easy to miss: before “do”, need ; or newline
13 / 41
Test CommandsA whole suite of shell builtin “[ expression ]” commands to douseful tests and give you exit codes for booleans.
File tests:
I [ -e path ]: existsI [ -f path ]: exists and regular fileI [ -d path ]: exists and directoryI [ -r path ]: exists and readableI [ -w path ]: exists and writableI [ -x path ]: exists and executableI [ -nt path1 path2 ]: both exist and path1 is newerI [ -ot path1 path2 ]: both exist and path1 is older
Example usage: [ -d lab02 ] || echo sadface
14 / 41
Test CommandsString comparisons:
I [ s1 = s2 ]: two strings equalAlso ‘!=’, ‘<’, ‘>’
I [ -n string ]: string not emptyI [ -z string ]: string empty
Number comparisons (parsing strings to numbers):
I [ n1 -eq n2 ]: two integers equalAlso ‘-ne’, ‘-gt’, ‘-ge’, ‘-lt’, ‘-le’
15 / 41
Test CommandsLogical connectives, by example:
I [ $x -eq 5 -a $y -eq 6 ]: andI [ $x -eq 5 -o $y -eq 6 ]: orI [ ! -e path ]: not
Parentheses also supported.
16 / 41
Test Commands in if/whileif [ $x = $y ] ; then...
fi
while [ $x = $y ] ; do...
done
Easy to miss: Still need ; or newline, even though ] “ends thecondition”.
Recall the syntax isif list ; thenhere, “[ $x = $y ]” is that list.
17 / 41
For-loopfor var in word1 word2 ... ; do
listdone
In the loop, use ‘$var’ to read the variable.
for i in Albert Frank do ; doecho $imkdir $i
done
May use ‘break’ and ‘continue’ in the loop.
Easy to miss: Need ; or newline before do to mark end of words.Lest computer thinks your do is one of the words, like above.
18 / 41
Patterns (to match filenames)
I ‘*’ matches any string (but doesn’t cross directory boundaries)Example: ls a2/*.pyAll python files in directory a2
I ‘?’ matches one characterI ‘[ace]’ matches “a” or “c” or “e”I ‘[0-9]’ matches a digitI ‘[!0-9]’ matches a non-digit
Important: Shell expands pattern to multiple pathnames beforehanding to program. ‘ls’ never saw “a2/*py”; it saw “a2/foo.py”,“a2/bar.py”, etc.
Important: If no match, the pattern stays as itself.
For-loop may use pattern too:for i in *.py ; do echo $i ; done
19 / 41
Escaping And QuotingShell syntax is full of special-meaning characters like ‘*’, ‘[’, ‘$’, ‘#’,‘(’, ‘;’, even space. . . You have seen them in special notations.
What if you want those characters themselves? For example howto safely print “x* #” (2 spaces in between)?
I Use backslash ‘\’ to “escape”:echo x\*\ \ \#Note: So ‘\’ is also special! Use ‘\\’ for backslash itself.
I Use single quotes:echo ’x* #’
I Use double quotes:echo "x* #"Note: ‘$’ still special, for a good reason shown later.
Likewise when setting variables: v=’x* #’
20 / 41
Example Uses of Escaping And Quoting
I Filenames containing spaces.ls -l ’Harry Potter.pdf’
I Some operators in test commands:[ ! ’(’ aaa ’<’ abc -o aaa ’>’ abc ’)’ ]
I Regular expressions for ‘grep’.grep ’<[a-z]*>’ lab.html
21 / 41
Example Uses of Escaping And Quoting
I Filenames containing spaces.ls -l ’Harry Potter.pdf’
I Some operators in test commands:[ ! ’(’ aaa ’<’ abc -o aaa ’>’ abc ’)’ ]
I Regular expressions for ‘grep’.grep ’<[a-z]*>’ lab.html
21 / 41
Example Uses of Escaping And Quoting
I Filenames containing spaces.ls -l ’Harry Potter.pdf’
I Some operators in test commands:[ ! ’(’ aaa ’<’ abc -o aaa ’>’ abc ’)’ ]
I Regular expressions for ‘grep’.grep ’<[a-z]*>’ lab.html
21 / 41
Variables in Double QuotesHave a filename “Harry Potter.pdf” that contains a space.
Set this: f=’Harry Potter.pdf’
What happens when: ls -l $f
Explanation: ls -l Harry Potter.pdftwo filenames.
Solution: ls -l "$f"
This is why $ is still special in double quotes.
One more example:v=’x* #’echo "$v"
22 / 41
Variables in Double QuotesCommon mistake when checking whether $v is non-empty:[ -n $v ]
No! If $v is empty, shell sees [ -n ], which makes no sense.
Solution: [ -n "$v" ]
Exercise: Older generation used[ x != x$v ]When does it work? When does it break?
23 / 41
CaseConditional branching on pattern matching.case "$var" in
*.py)rm "$var";;
*.c | *.sh | myscript)echo w00t "$var";;
*)echo meh "$var"
esac
24 / 41
Shell ScriptsPut your commands in a file, call it “myscript” say. You can run itwith
sh myscript
More savvy users go one step further:
I Put as first line: #!/bin/shI Set executable flag on the file:chmod u+x myscript
I Run it with ./myscript
Next: How to access arguments? E.g.,‘./myscript foo’how can the script see that “foo”?
25 / 41
Positional ParametersIf I run your script as ‘./myscript foo bar xyz’:
I $# has 3, the number of argumentsI $1 has “foo”I $2 has “bar”I $3 has “xyz”I $* has “foo bar xyz”"$*" expands to one single word “foo bar xyz”
I $@ has “foo bar xyz”"$@" expands to 3 words “foo”, “bar”, “xyz”
26 / 41
shift
Shift positional parameters. E.g., starting from the previous slide,one shift causes:
I $# has 2I $1 has “bar”I $2 has “xyz”I $* has “bar xyz”"$*" expands to one single word “bar xyz”
I $@ has “bar xyz”"$@" expands to 2 words “bar”, “xyz”
Convenient for looping over arguments!
27 / 41
Small Example Script (Pg 1/2)#!/bin/shdryrun=verbose=while [ $# -gt 0 ]; do
case "$1" in-n)
dryrun=y;;
-v)verbose=y;;
*)break
esacshift
done
28 / 41
Small Example Script (Pg 2/2)for f in "$@" ; docase "$f" in*.py)[ -n "$verbose" ] && echo "deleting $f"[ -z "$dryrun" ] && rm "$f";;
*)[ -n "$verbose" ] && echo "not deleting $f"
esacdone
Code file: smallscript.sh
29 / 41
Small Example Script: ExplanationGo through arguments (meant to be filenames), delete those thatare Python files.
But if there are ‘-n’ and/or ‘-v’ at the beginning:-n means dry-run—don’t actually delete-v means verbose—say what is happening to each filename
Page 1 detects ‘-n’ and ‘-v’.
After that, ‘$@’ is left with the filenames.
Page 2 can use a for-loop over ‘$@’ to process each filename.
Exercise: Why didn’t I use a for-loop in page 1?
30 / 41
ExitMay use the command ‘exit’ to terminate the shell script and/orthe shell process.
Not required if your script just runs from start to finish normally.
But handy for:
Early termination (when inside loops, functions, etc.)
Controlling exit code, e.g., ‘exit 1’.
(Default exit code is whatever the last executed command gives.)
31 / 41
FunctionsExample function definition:
myfunction() {echo $1echo $@
}
Example function call:
myfunction foo bar xyz
Inside a function, positional parameters become functionarguments.
May return from function early, or specify exit code, with ‘return’or e.g., ‘return 1’.
(Default exit code is from the last executed command.)
32 / 41
Local Variables in FunctionsExample:
myfunction() {local x yx=1# y deliberately unchanged , see belowecho $x $y
}
Local as in: Does not change outer variables.
Initial values copied from caller.
y=foomyfunction
prints “1 foo”.
33 / 41
Dot command: Execute stuff in current shellThis reads commands from cmds.sh, executes them in currentshell:
. ./cmds.sh
The command name is a single dot “.”
Contrast: sh cmds.sh runs in newly spawned shell process.
Use case: If cmd.sh defines functions, set variables, or uses cd,then
I . ./cmds.sh does them in current shell.I sh cmds.sh does them in new shell process, which then
quits, much ado about nothing../cmd.sh ditto.
34 / 41
Here DocumentTo feed multi-line hardcoded text into stdin of a command:
cat << EOFHello I’m Albert.You can use variables tooE.g., \$x=$xEOF
The first time I said “EOF”, shell takes note. Second time, shellknows I’m marking the end.
“EOF” is not a keyword, you may choose another word, just don’tclash with your actual text!
35 / 41
Here Document: One More ThingIf you declare your end-marker in quotes:
cat << ’EOF’Hello I’m Albert.Now $x is $xEOF
cat << "EOF"Hello I’m Albert.And $x is still $xEOF
then $ is no longer special.
36 / 41
Command SubstitutionRun a command, capture its stdout, insert output data in-place:$(command)
or‘command‘
(both backticks, not single quotes)
The data is split into words.
for i in $(echo aaa bbb ccc) ; doecho "$i"
done
But if inside double-quotes, not splitted. (And other technicalitiesnot shown here.)
for i in "$(echo aaa bbb ccc)" ; doecho "$i"
done
37 / 41
Environment VariablesEvery process (shell or otherwise) has a collection of “environmentvariables”, as part of process state.
Names are strings, values are strings too. Convention: Names inall caps, e.g., PATH, HOME, TZ, LC_ALL (these are standard Unixones), CLASSPATH (specific to Java).
Initialized by copying from launcher (done by kernel): If p launchesq, q gets a copy of p’s. But independently changeable otherwise.
Program ‘printenv’ prints the environment variables you currentlyhave. It works because at startup it gets a copy of yours! Now itjust has to print what it has.
38 / 41
Environment Variables in ShellShell downplays difference between shell variables andenvironment variables. Only convention: shell variable names arein lowercase.
Both read by same syntax: $x, $LC_ALL
Both changeable by same syntax:x=CLC_ALL=C
Both erasable by same ‘unset’ command.
How to mark a variable as environment variable:export MYENVVAR=foo
or two commands:MYENVVAR=fooexport MYENVVAR
39 / 41
Environment Variables in ShellTo run a program but give it different environment variables(existing or new) without changing your own:
LC_ALL=C MYNEWENV=foo printenv
This is why the following two commands mean different things:
x=’foo bar’x=foo bar
40 / 41
Some Standard Environment VariablesHOME: Home directory.
TZ: User timezone preference. (Can be absent.)
PATH: Colon-separated list of directories. Searched when youlaunch a program, if program name does not contain any slash.
Example: AssumePATH=/usr/local/cms/jdk1.8.0_31/bin:/usr/bin:/bin
javac Foo.javafound in /usr/local/cms/jdk1.8.0_31/bin
printenvfound in /usr/bin
shfound in /bin
41 / 41