Practical 02Bash & shell scripting

1

iMac lab

• login: maclab• password: 10khem

1.use the Finder to visually browse the file system (single click opens)

2.find the /Applications folder3.open the Utilities folder4.open the Terminal application

2

Input/Ouput redirection >, >>, <

cat < FILEcat FILE1 FILE1 > FILE3cat FILE1 FILE2 >> FILE3

What happens withcd p01cat 1AKE.pdb 1AKE.pdb > tmp/new.pdbcat 1AKE.pdb 1AKE.pdb >> tmp/new.pdbcat 1AKE.pdb 1AKE.pdb > tmp/new.pdb

Use ls -l ...

3

Counting file contents: wc

wc [options] FILEwc < FILEwc -l FILE

Count the lines in 1AKE.pdb and tmp/new.pdb

4

Counting file contents: wc

wc [options] FILEwc < FILEwc -l FILE

Count the lines in 1AKE.pdb and tmp/new.pdb

$ wc 1AKE.pdb 4424 50152 358344 1AKE.pdb

lines words characters

4

Counting file contents: wc

wc [options] FILEwc < FILEwc -l FILE

Count the lines in 1AKE.pdb and tmp/new.pdb$ wc -l 1AKE.pdb 4424 1AKE.pdb$ wc -l tmp/new.pdb 8848 tmp/new.pdb

$ wc 1AKE.pdb 4424 50152 358344 1AKE.pdb

lines words characters

4

curl http://becksteinlab.physics.asu.edu/pages/courses/2013/SimBioNano/01/1AKE.pdb -O

Downloading with curl

• Instead of using a web browser, use

curl URL -O [name]

5

curl http://becksteinlab.physics.asu.edu/pages/courses/2013/SimBioNano/01/1AKE.pdb -O

Downloading with curl

• Instead of using a web browser, use

curl URL -O [name]

• “cat URL”

curl URL > namecurl URL | wc -l

5

Variables

BASEURL="http://becksteinlab.physics.asu.edu/pages/courses/2013/SimBioNano/"PRACTICAL=01FILE=1AKE.pdb

echo "Practical ${PRACTICAL}: "echo "downloading file ${FILE}"

URL="${BASEURL}/${PRACTICAL}/${FILE}"

curl -O "${URL}"

6

Editing files

• vi editor (“vim”)

vi hello.sh

7

A simple shell script

#!/bin/bash# usage: hello.sh NAME# Greets NAME.

NAME="$1"

echo "Hello ${NAME}!"

Run the script:

bash ./hello.sh your_name

8

-rwxr-xr-x 1 oliver staff 91 Jan 14 14:15 hello.sh

Executable scripts

Run the script:

chmod a+x ./hello.sh

./hello.sh your_name

A script can only be directly executed if the “executable” flag is set:

Check with ls -la ./hello.sh:

9

Argument processing

• white space splits arguments: –./hello.sh your name–./hello.sh “your name”

• individual positional arguments can be accessed as $1, $2, $3, ..., ${10}, ...

• all arguments: $*• optional arguments (-h, -f FILE, ...): use the

–getopts (shell builtin) –getopt (GNU, not always available)

functions (advanced)

10

Write your own course downloader “cdl”

• Should work like this:

cdl.sh 01 vim_basics.txt

downloads the file http://becksteinlab.physics.asu.edu/pages/courses/2013/SimBioNano/01/vim_basics.txt

#!/bin/bash# usage: cdl.sh PRACTICAL FILENAME# downloads FILENAME for PRACTICAL# example:# cdl.sh 01 vim_basics.txt

• Start your script with a standard header

11

#!/bin/bash# usage: cdl FILENAME# downloads FILENAME for practical 01

“Course DownLoad” script cdl$ vi bin/cdl

12

Download script

• make executable chmod a+x cdl

• store in your ~/bin directory mkdir ~/bin cp cdl ~/bin

• try it ~/bin/cdl 01 1AKE.pdb ~/bin/cdl 01 vimqrc.pdf

13

Environment variables

• The shell sets a number of variables with special meaning.

• See all of them with

env | less• The PATH contains a list of directories (separated by

“:”) where the shell looks for commands (“executables”):echo $PATH.:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin

14

Environment variables

• Add your own ~/bin directory to the PATHPATH=${PATH}:~/binexport PATH

• Make this change permanent by adding the lineexport PATH=${PATH}:~/bin

to the end of the shell startup file ~/.bashrc

• try it: open a new terminalwhich cdlcdl 01 1AKE.pdbcdl 02 extra.txt instructions for writing a

more challenging script

15

Pipes: |

• Unix commands can be joined:

standard output →standard input ...

•command | command | ...

cat FILE | wc -lcat 1AKE.pdb tmp/new.pdb | wc -l

16

Filters: grep and egrep

• egrep == grep -E• prints lines matching a “pattern” (“regular

expression”)

cat FILE | egrep PATTERN

cat 1AKE.pdb | egrep '^ATOM'egrep '^ATOM.* A *[0-9]+' 1ake.pdb \ > 1ake_A.pdb

17

Filters: grep and egrep

• Count the atoms in 1AKEcat 1AKE.pdb | egrep '^ATOM'| wc -l

• Count the residues in 1AKE (each residue contains a single “CA” atom)cat 1AKE.pdb | egrep '^ATOM.*CA'| wc -l

18

Filters: sort and uniq

• alphabetical sort

cat FILE | sort• numerical sort

cat FILE | sort -n

• select “fields” for sorting (white space separated)

cat FILE | sort -n -k2,2

• only keep unique entries

cat FILE | sort | uniq

• count unique entries (“histogram”)

cat FILE | sort | uniq -c

19

cut -f N,M -d ' ‘

Filters: cut

• Cut fields or columns from a FILE or input stream

cat FILE | cut OPTIONScut OPTIONS FILE

• data from columns N-M and X-Y

cut -c N-M,X-Y FILE

Example: get all residue names from a PDB file:

• data from fields N-M separated by white space

cat 1AKE.pdb | egrep ‘^ATOM’ | \ cut -c 18-20

20

cat FILE | sed 's/SEARCH/REPLACE/g'

sed and awk

• sed is a “stream editor” (similar to some of the commands used in vi)

• common use is search and replace:

cat FILE | \ awk '/REGEX/ {awk-command; \ awk-command; ...}'

• awk applies a small program to each input line (it is its own programming language)

• common use: working on separated column data:

cat /etc/passwd | awk -F ':' '$1 ~ /^[a-z].*/ {print $1}'

21

More...

• see Introduction to Unix – http://becksteinlab.physics.asu.edu/pages/unix/IntroUnix/

p01_UNIX.html

• or as PDF file– cdl IntroductiontoUNIX.pdf

• assignment (will be posted on Blackboard)

22