Introduction to Unix

Daniel PriceHonours Computing Course, March 2009

What is Unix?

• UNIX was an proprietary operating system developed at AT&T’s Bell labs in the 70’s. Many companies developed their own “flavours” of Unix in the 80’s.

• written in C, so Unix and C share many characteristics. Previously operating systems had been written in assembler (i.e., not portable).

• “Unix” refers to a collection of components (kernel, commands, documentation) that are usually packaged together.

• Various attempts to define what defines “Unix”, e.g. IEEE POSIX specification (1980’s). Now defined by the “Single Unix Specification”. Other variants referred to as “Unix-like”.

• Modern Unix or Unix-like operating systems include: Linux/GNU, Mac OS/X, Solaris

Unix concepts

• My definition of Unix philosophy: “a bunch of small, independent utilities, each designed to do one job and do it well, but which can be combined to perform more powerful tasks”.

• Every process has an associated Process ID (PID) and is independent of other processes. It is also associated with a particular user via the user-ID (UID).

• Users have restricted “permissions” (=no viruses).

GNU/Linux

• Linus Torvalds: wrote “Linux” in 1991. Linux refers to the kernel - the part of the OS that schedules processes, handles interrupts, interfaces between hardware and software etc.

• Richard Stallman: Founder of the GNU project - goal is to create a “complete UNIX-like operating system which is free software”

• Wrote the GNU General Public License

• Many modern Unix programs are from the GNU project

GNU GPL

• You are allowed to make copies, but must give away source code

• If you modify the code, must release it under the same license

“When we speak of free software, we are referring to freedom, notprice. Our General Public Licenses are designed to make sure that youhave the freedom to distribute copies of free software (and charge for

this service if you wish), that you receive source code or can get itif you want it, that you can change the software or use pieces of it

in new free programs; and that you know you can do these things.”

Unix from a users perspective

• Command-line interpreter: the “shell” (like DOS). Can be sh, bash, tcsh, ksh, etc...

• Windowing system: usually “X-Windows” or “X11” (like MS Windows). One advantage is that X-windows can be launched from remote machines.

• Desktop environment/window manager: Gnome or KDE.

The unix terminal or “shell”

• The most powerful part of unix

• Some commands can be replicated via gui interfaces (e.g. copying/moving files), but you miss some of the most powerful features of unix - the ability to string commands together.

• example:

Exercise 1: Shell basics: pwd, ls, man

• login/open a terminal

• find what directory you are in: “pwd”

• list the files in the current directory “ls”

• get help on the ls command: “man ls”

• work out how to show all of the files, including “hidden files” that begin with a dot.

• work out how to list all files in “long format” to get the file size, owner and date of last modification.

Exercise 2: Working with files, redirecting output

• use the unix “>” redirect to output the contents of ‘ls’ into a file: “ls > file”

• make a copy of the file: “cp file file_backup”

• make a new directory: “mkdir myverylongdir”

• move the file into the new directory: “mv file myverylongdir” (use tab completion)

• change to the new directory: “cd my<tab>”

• delete the file “rm file”

• leave the directory “cd ..”

• remove the directory “rmdir mydir”

• try rmdir on a directory that is not empty. Use “rm -r mydir” instead.

Summary of navigation/file management commands

ls list the contents of a directory

ls -l list contents in “long format”

ls -a list all files, including “hidden” files starting with a dot (.)

pwd find out what directory you are in

cd mydir change to the directory “mydir”

cd .. move back one directory in the heirarchy

cd - go back to the previous directory you were in

cd ~/mydir go to the directory mydir one level down from your “home space”

mkdir newdir make a new directory called “newdir”

rmdir newdir remove a directory (must be empty)

cp file file_backup copy a file

cp mydir/file . copy a file from the directory “mydir” to the current directory “.”

mv file_backup file move a file

rm file delete a file

rm -r mydir recursively delete all files in a directory (very dangerous!)

rm -i -r mydir recursively delete all files in a directory (with prompts)

rm -rf mydir recursively delete all files in a directory, overriding permissions (very very dangerous!)

Important places in a Unix file system

• Unix uses a system of directories to categorise particular types of files. The ones you should know about are:

/bin/ where files executable by the user generally go

/home/ where users home directories lie

/dev/ hardware devices (important one is /dev/null, the null device)

/tmp/ a place for temporary files (usually wiped when you reboot)

/lib/ where system libraries needed for the binaries in /bin/ go

/mnt/ where temporarily mounted file systems (e.g. your usb stick) will appear

/var/ variable files such as system logs (and your email, in /var/mail)

/usr/ “user data”, where most applications and utilities will be, divided into:

/usr/bin/ non-essential executable files, to be accessed by all users

/usr/X11R6/ or /usr/X11/ where the windowing system (X-windows) lies

/usr/include/ where c “header files” go

/usr/lib/ where libraries for the binaries in /usr/bin go

Redirecting output and Unix pipes

cmd > file redirect the output of cmd to file

cmd >& file redirect *all* the output (including error messages) of cmd to file

cmd 2> file redirect only the error messages to file

cmd | nextcmd redirect the output of cmd to nextcmd

cmd | nextcmd > file redirect the output of cmd to nextcmd, then put the output in a file

cmd `newcmd` run cmd using the output of newcmd as the argument

cmd > /dev/null run cmd but throw away the output (send it to the “null device”)

cmd >> file append the output of cmd to file

cmd < myfile use myfile as the input to cmd

• unix (and c) has two output streams - one for “normal” messages (stdout), and one for error messages (stderr).

• you can capture and redirect one or both of these.

• the output of one command can also be used as the input to the next command (this is called “piping”).

Exercise 3: output redirection and pipes

• redirect the output of the ls command into a file

• type “ls myfile” where myfile does not exist. Pipe the output to a file (“ls myfile > newfile”)

• use “cat” to see what is in the file “cat newfile”

• now repeat the ls command, redirecting both the output and the error stream. Now see what is in the file (use cat)

• repeat the ls command, but silence all the output

Unix text editors

• Whilst there are many GUI text editors now available for editing basic text files, Unix/X-Windows has a few “classics” which at some point you will need to be aware of. We will briefly discuss vi and emacs and nedit.

• You need to know about vi and emacs because they let you edit text files from inside the terminal (without launching windows). Occasionally, if you are

connected to a remote machine, you will need to do this when your favourite GUI editor is not available. vi and emacs will nearly always be available in any unix distribution on any machine.

• Personally, I use nedit as it is very easy to use and vi for in-terminal editing. Emacs has an archaic system of commands though it is very powerful once you know how to use it.

vi: the essentials

• :q

• or maybe :q!

Exercise 3: vi

• start vi

• quit vi

A vi quick reference

i insert text

esc exit “insert mode” and go back to “command mode”

x delete character

r replace character

dd delete current line

5dd delete 5 lines

:w write changes to file

:wq write changes to file and exit

:q, :q! quit without saving changes, really quit without saving changes

yy copy (“yank”) a line to the clipboard

5yy copy (“yank”) 5 lines to the clipboard

p paste contents of clipboard

ctrl-f page down

ctrl-b page up

/string search for “string” in the file

Emacs: the essentials

• ctrl-x ctrl-c

• emacs -nw to run “in-terminal”

Exercise 4: emacs

• start emacs

• quit emacs

An Emacs quick reference

ctrl the emacs “control” key, abbreviated C-

alt or esc the emacs “meta” key, abbreviated M-

ctrl-x ctrl-c quit

ctrl-x ctrl-s save file

ctrl-x ctrl-f open a file

ctrl-a go to start of line

ctrl-e go to end of line

ctrl-k delete to end of line

ctrl-_ undo

ctrl-s string search for “string”

ctrl-h emacs help

ctrl-g stop whatever weird thing emacs is doing

ctrl-x b switch buffer (i.e., go back to what you were doing)

meta-x doctor seek psychotherapy

meta-x tetris play tetris

Exercise 5: making your life simpler (aliases)

• find the files called .profile and .bashrc: this is where you can customise your local shell environment. In general it is least confusing to just use .bashrc and make sure this is also read by .profile.

• open .profile in your favourite editor (we will use nedit): “nedit .profile &”. The & is important (what happens if you don’t use it?).

• add the line “source ~/.bashrc”. This means “execute the contents of ~/.bashrc”.

• use the alias command to add an alias for ‘ls -l’ to this file, i.e. add the line: alias ll=`ls -l’

• save and exit, then type “source ~/.bashrc” to execute these commands

• type `ll’ to use your new command

Environment variables (bash)

printenv print all the currently set environment variables

MYVAR=’blah’ set the variable MYVAR to ‘blah’

export MYVARexport the setting of the MYVAR variable to all child processes of the current process

export MYVAR=’blah’ shortcut for the above two commands

export MYVAR=$MYVAR:’blah’ append ‘blah’ to the current setting of the environment variable MYVAR

echo $MYVAR print the current setting of the variable MYVAR

echo “my variable = $MYVAR”print the current setting of the variable MYVAR with some text (NB: must use double quotes for this).

unset $MYVAR “unset” the variable MYVAR

• Environment variables can be used to change the run-time behaviour of programs, or to make basic information about your system available to programs at run time.

Exercise 6: Environment variables

• set an environment variable called MYNAME to your name

• print your name (using the environment variable) to the screen

• print your name as part of the sentence “my name is ....”

• set the environment variable with the output of the `whoami’ command and repeat the above

• find the environment variable that tells you which shell is running (i.e. bash/sh/tcsh)

Checking what is running and killing it: top, jobs, ps, kill

• finding jobs running from the current terminal: “jobs”

• killing job number 1 in the current terminal: “kill %1”

• finding processes run from any terminal: ps or top (“man ps” to get options)

• killing jobs run from other terminals: “kill 123” where 123 is the PID.

• finding all processes running on the current machine: “top”

Exercise 7: foreground and background processes

• start nedit in the foreground: “nedit”

• ctrl-c to quit

• start nedit in the background “nedit &”

• type “jobs” to find the job id

• kill the nedit job - e.g. kill %1

• restart nedit in the background “nedit &”

• open a new terminal and use the “ps” command to find the PID of the job, then kill the job from the other terminal using “kill PID”

• start nedit in the foreground, then use ctrl-z and “bg” to move it to the background

Exercise 8: killing processes

• type “yes `Daniel is awesome’”

• stop this using ctrl-c

• type “sleep 100 &”

• get the job id for this process: “jobs”

• kill the job: “kill %1”

• type “yes ‘Daniel is awesome’ &”

• open another terminal, use ps to find the PID, and kill it using the “kill” command.

Summary of process control/user information

jobs list all the jobs running from the current shell

kill %1 kill job number 1

kill -9 %1 axe murderer kill, die, destroy job number 1

top list all the things running on your machine

top -o cpu list all the things running on your machine, ordered by cpu usage (Mac OS/X)

kill 11123 kill the job with PID 11123 (get the PID from top or ps -u)

ps -u myuser lists all the processes being run by user “myuser” on the current machine

last | head list the last few logins to your machine

finger myuser tells you what myuser is up to

finger @machine tells you who is logged in to a particular machine

whoami answers existential questions about yourself

who tells you who is currently logged in to your machine

ctrl-c kill the job you are in

ctrl-z move the job you are in to the background

fg bring the last-backgrounded job to the foreground

bg send the current (suspended) job to the background

Viewing and editing files: cat, more, less, head, tail

cat file view contents of file

more file view contents of file, a screen at a time

cat file | more same effect as “more file”

less file less=more

head file view first few lines of file

head -100 file view first 100 lines of file

tail file view last few lines of file

tail -100 file view last 100 lines of file

tail -f file view last few lines of file, and keep updating it if the file gets updated

paste file1 file2 paste file1 and file2 side-by-side

diff file1 file2 print the differences, if any, between file1 and file2

wc file count the number of words in file

Exercise 9: cat, more, head, tail

• use “wget” to download the example latex file from: http://users.monash.edu.au/~dprice/honourscomputing/latex/mclusterRT.tex

• spit out the contents of this file using cat. Pipe the output to more to break up the pages.

• use wc to count the number of words in this document.

• use “head” and “tail” to isolate the abstract text and count the number of words in the abstract alone.

• use the search function in more to see if any lines contain the word “Monash”

Text processing: echo, grep, sed, cut, paste

echo “hello world” print hello world

echo -e “hello \nworld”print hello world with a line break in the middle (\n is the line break escape character)

grep string file print lines in file that contain `string’

grep string file* print lines containing `string’ in all files with name beginning with `file’

grep -i StRiNg file case insensitive search for ‘string’ in file

grep -v string file print all lines in file which *do not* contain `string’

cat file | grep -v string same as above

cat file | sed ‘s/string/newstring/g’ print contents of file with ‘string’ replaced by ‘newstring’

cat file | cut -d’ ’ -f 4 print fourth column of file where columns are separated by a single space

Exercise 10: grep, sed

• use grep to find the lines containing the word “Monash” in the previous example.

• use the sed “replace” function to replace “Monash” with “The Empire”. Put the output in a new file (e.g. mclusterRT_new.tex).

• use diff to see the changes between the two files

Pattern matching: Regular expressions

• Regular expressions are one of the strangest features of Unix, but also the most powerful. They define how to match particular patterns. For example:

expression meaning example

* match any string/series of characters ls *.jpg, matches all jpeg files

? matches any individual character ls file?.jpg, matches file1.jpg, filez.jph etc.

[a-z] matches any character in the range a-z ls file[a-z], matches filea,fileb,filec,...

[1-9] matches any character in the range 1-9 ls file[1-9], matches file1,file2,file3,...

[xyz] matches any one of the characters x,y or z ls file[123], matches file1, file2 or file3 only

^ matches the start of a line

$ matches the end of a line grep ^$ file, gets all the blank lines in the file

[xyz1-3] matches x, y or z or numbers in the range 1-3

[a-zA-Z] matches any character a-z in upper or lower case

• because these characters (*,?,[,^,$...) have special meanings, to use them in their usual sense in a string, you need to either surround them by (single) quotes (e.g. ls `file?.jpg’) or preface them by a “\”, e.g. ls file\?.jpg

Exercise 11: more grep, regular expressions

• download “splash” from http://users.monash.edu.au/~dprice/splash/splash.tar.gz

• count the total number of lines of fortran in this code (i.e., all .f90 files)

• find how many of these lines are blank, how many are comments (start with !)

• using grep, construct a list of all the subroutine names and the files they are in.

• use the “cut” command to extract only the subroutine names (not the files they belong to)

Unix permissions

• chmod: change the permissions on a file. Examples:

• chown: change the owner of a file

• chgrp: change the “group of users” to which the file belongs

chmod u+x myfile make the file executable for the current user

chmod g+x myfile make the file executable for the current group of users

chmod a+x myfile make the file executable for all users

chmod u-w myfile make the file read-only for the current user (remove write permission)

chmod a-w myfile make the file read-only for all users

chmod a-r myfile do not let anyone read the file

chmod u+r myfile allow the current user to read the file

chmod -R a+r * make files readable to all users, for the current directory and *all subdirectories*

• type the following into a file, e.g. called “myscript”:

• make the file executable “chmod a+x myscript”

• run it. What does it do? Why?

• now retry the script, replacing /bin/rm with /bin/bash

• you’ve written your first shell script!

Exercise: A completely useless shell script

#/bin/rmecho ‘hello world’

Bash loops

• with bash it is very easy to perform an operation on a whole series of files using a loop, e.g. to list all jpeg files:

for x in *.jpg; do echo $x; done

• or to print a list of the files with the extension .jpg changed to .png

for file in *.jpg; do echo ${file/.jpg/.png}; done

• where ${x/.jpg/.png} means replace .jpg with .png in the string. So you could perform a convert operations

for x in *.jpg; do convert $x ${x/.jpg/.png}; done

Bash scripts

• to put the previous commands into an executable file, we would use:

#/bin/bash

for x in *.jpg;do convert $x ${x/.jpg/.png};done

• then make the file executable to be able to run it

Testing whether or not a file exists: bash if/then/else

• A script to test if a file exists or not:

#/bin/bashif [ $# -ne 1 ]; then echo “Usage: $0 filenameelse file=$1; if [ -e $file ]; then echo “$file exists”; else echo “$file does not exist”; fifi

this script takes a command line argument, so check that one is present

$0 refers to the “0th argument”, which is always the name of the executable being

run (in this case, the script name)

$1 is the first command line argument

-e is the test for “file exists”

Exercise: bash functions

• a very powerful feature of bash is the ability to define functions in your .bashrc file, e.g. we define a “hello” command by adding the lines:

• Exercise: make a new command called “exists” in your .bashrc that checks whether or not a file exists based on the previous example

hello (){ echo “hello $USER”; echo “Is $USER the same as `whoami` ?”;}

Bash select case

#/bin/bashfor x in $@; docase $x in sheep) sound=’baa’; cow) sound=’moo’; horse) sound=’neigh’; dog) sound=’woof’; *) sound=’an unknown sound’;esac;echo “$x says $sound”;done

cycle over all of the command line arguments

default case

• Exercise: modify this script so that instead of taking command line arguments it takes the animal names from a file called `animals’ which has one animal name per line

Advanced scripting languages

• bash scripts are useful, but they are limited to a few simple tasks.

• there are now quite a few fully-fledged scripting languages (that is, you could write basically any program in them). You should learn at least one of them. Perl and/or Python are perhaps the two most useful (ruby is also becoming popular).

• scripting languages are often called “interpreted” languages, because the commands are interpreted on-the-fly. It is not a good idea to try to do numerically intensive tasks with them (except with specialised libraries, e.g. numpy) because in general they will be *much slower* than compiled languages like C and Fortran.

• links to tutorials on Perl & Python are on the web page, as well as some of my own example scripts

Querying available disk space: du, quota, df -H

• How do we check how much space files are using?

• type “man du” to find out about the du command.

• find out how big the file created in the last exercise using “du -s -k -h myfile”. See what each of the flags does.

• go back to the previous directory “cd ..”

du -s -k * find out how much space all of the files and directories in the current folder are using

df -H find out how much space is available on the disk

quota -v find out about any file system quotas you may have, and how much you are using

Compression/archiving: tar, gzip

• to put a bunch of files on the web or send them to another user, it is useful to be able to package them up into a neat, small package. To do this we use the unix “tape archive” command.

• literally this used to mean concatenating all the files together on magnetic dat tape. In practise now it is a useful file format (hence tar cv`f’ - the f refers to making files instead of actually writing to a tape).

tar xvf file.tar extract the contents of the tar file “file.tar”, do this verbosely (v)

tar cvf mydir.tar mydir/ bundle up the contents of the directory mydir into a file called mydir.tar

gzip mydir.tar compress the file mydir.tar (becomes mydir.tar.gz)

gunzip mydir.tar.gz uncompress the gzipped tar file mydir.tar.gz

zip mydir.tar compress the file mydir.tar using pkzip (becomes mydir.tar.zip)

unzip mydir.tar.zip uncompress the pkzipped tar file mydir.tar.zip

Soft links

• like “shortcuts” in windows

• Exercise: create a soft link to a file, e.g. “ln -s myfile newfile”. Now use nedit or another editor to edit the contents of “newfile”. Save your changes, exit and look at the contents of myfile (using cat or more).

ln -s ~/mydir/file creates a shortcut (called `file’) in the current directory, pointing to ~/mydir/file

ln -s ~/mydir/file myfile creates a shortcut (called `myfile’) in the current directory, pointing to ~/mydir/file

Remote login / remote file operations: ssh, scp

• Part of the power of the unix shell is that it is very easy to perform command-line operations on a remote machine

• the command for doing so is “ssh” (secure shell). Remote copy operations are performed with “scp” (secure copy). sftp can also be used. Obviously you need an account on the remote machine.

• Importantly, you can also forward X11 connections from remote machines. To do so, use “ssh -Y” when you log in. Any X-window launched on the remote machine will then pop up on your machine. Examples:

ssh user@maths.maths.monash.edu.au remote login to your maths account

ssh -Y user@maths.maths.monash.edu.au remote login to your maths account, forwarding X connections

scp file user@maths.maths.monash.edu.au: copy a file into your home directory in your maths account

scp user@maths.maths.monash.edu.au:file . copy a file from the maths account to your local machine

Backing up files: rsync

• rsync is like scp, but only copies files that have changed

• this is perfect for e.g., backing up your hard drive

• use “man rsync” to get the possible options, examples below

rsync -avzu /Users/dprice/Documents dprice@maths.maths.monash.edu.au:

copy the contents of my Documents folder to the Documents folder in my maths home space. Skip files which are newer in the maths space (-u)

rsync -avzu --exclude=.DS_Store --exclude=non-work/ /Users/dprice/Documents dprice@maths.maths.monash.edu.au:

as above, but excluding files called .DS_Store and excluding everything in the subdirectory “non-work/”

rsync -avz --delete /Users/dprice/Documents/scripts dprice@maths.maths.monash.edu.au:Documents;

sync the Documents/scripts directory to the Documents/scripts directory in my maths home space. Also *delete* files remotely which have been deleted locally (use with caution!).

rsync -avzu dprice@maths.maths.monash.edu.au:/home/dprice/research/ /Users/dprice/research/;

sync (copy) from the remote machine to the local machine (without deleting any files)

A neat backup command

find ~/Documents \! -type d \( -size -3000c -or -name "*.[f,F,f90,F90]" \) -print | tar -T - -c -v -z -f $HOME/backup-docs.tgz;

finds all the files under 3k in size (this covers most important files like scripts and text files), or with names ending in .f, .f90., .F or .F90 (i.e. Fortran programs) and

bundles them into a tar file in my home directory

Random other tidbits

wgetdownloads files from the internet, e.g. wget “http://users.monash.edu.au/~dprice/teaching/unix.html”

a2ps myfile.txt `pretty print’ a text file ready for printing

fortune prints a fortune cookie or pithy quote (often disabled by system administrators)

banner hello try it

yes a completely useless command

open myfile Mac OS/X only: opens myfile using the default application

talk userid open a “talk session” with another user

cal prints a calendar. Try “cal 1752” or “cal sep 1752”.

uptime tells you how long since last reboot, cpu load averages