week 9 - nov 7, 2005 1 week 9 agenda i/o redirection i/o redirection pipe pipe tee tee

Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005 11

Week 9 AgendaWeek 9 Agenda I/O redirectionI/O redirection pipepipe teetee

22 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Redirection and PipingRedirection and Piping

Most “processes” in UNIX:Most “processes” in UNIX:

send their output to the terminal screensend their output to the terminal screen

get their input from the keyboardget their input from the keyboard

send any error messages to the terminalsend any error messages to the terminal

33 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005



send their output to the terminal screensend their output to the terminal screen

get their input from the keyboardget their input from the keyboard

send any error messages to the terminalsend any error messages to the terminalDo you

remember what a “process” is?

44 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

A process is the execution of a A process is the execution of a command by the UNIX system.command by the UNIX system.UNIX assigns a unique process UNIX assigns a unique process identification (PID) number at the identification (PID) number at the creation of each process. creation of each process. As long as a process is in existence, it As long as a process is in existence, it keeps the same PID number. keeps the same PID number.

55 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

RedirectionRedirection


send their to thesend their to theoutputoutput terminal screenterminal screen

inputinput keyboardkeyboardget their from theget their from the

send any to thesend any to theerror messageserror messages

terminal screenterminal screen

66 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

RedirectionRedirectionUNIX allows us to UNIX allows us to

““redirect” redirect” where where

processes send their outputprocesses send their output

wherewhere

processes get their input processes get their input

and and

where where

processes send any error messages.processes send any error messages.

77 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

88 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

I/O Channel 0 - standard input (stdin)

I/O Channel 1 - standard output (stdout)

I/O Channel 2 - standard error (stderr)

These “channels” are called file descriptors

99 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

stdinstdout

stderr

1010 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

I/O Channel 0 – stdin -default is the terminal keyboard

I/O Channel 1- stdout - default is the terminal screen

I/O Channel 2- stderr - default is the terminal screen

These “channels” are called file descriptors

Let’s learn to redirect

”stdout” first

1111 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Standard Input, Standard Input, Standard Output, Standard Output, Standard Error.....Standard Error.....

not a piece of not a piece of hardwarehardware

not a keyboard, a not a keyboard, a screen or a card in screen or a card in your computeryour computer

Actually “virtual Actually “virtual files”...opened by the files”...opened by the kernel for every kernel for every command command

RememberRemember

1212 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Redirecting the output Redirecting the output of a command to a fileof a command to a file

Use the Use the greater thangreater than symbol symbol >>

this redirects the “standard output” or this redirects the “standard output” or “channel 1” of a command “channel 1” of a command

this is the same as this is the same as 1>1> but you don’t but you don’t need to type the “1” - it is need to type the “1” - it is “understood”.“understood”.

1313 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Redirecting the output of a Redirecting the output of a command to a filecommand to a file

““The Rules”The Rules”

a command always comes before the > symbola command always comes before the > symbol

a filename always comes after the > symbola filename always comes after the > symbol

command > file

always a command before the >

always a file name after the >

You can think of “>” symbol as being an arrow You can think of “>” symbol as being an arrow pointing from a command to a filepointing from a command to a file

1414 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005


The output of this command is displayed on the screen by

default

Nothing is displayed on the

screen – the output of the date command is

redirected to the file “today”

Issue the command w >users

1515 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Now, look at the contents of the file named users


1616 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Redirecting the output Redirecting the output of a command to a fileof a command to a file

Use the double Use the double greater thangreater than symbol symbol >>>>

this appends the “standard output” or this appends the “standard output” or “channel 1” of a command to a file“channel 1” of a command to a file

You can think of the symbol as a double You can think of the symbol as a double arrow from the command to the file, arrow from the command to the file, meaning 'add the output from the meaning 'add the output from the command to the END of the file'.command to the END of the file'.

1717 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

We can also redirect the We can also redirect the outputoutput

of one commandof one command to be to be inputinput

to another commandto another command


1818 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Piping - redirecting the output Piping - redirecting the output from a command to a commandfrom a command to a command

Use the pipe symbol to “connect” Use the pipe symbol to “connect” commandscommandsThe “pipe” character is a broken vertical line The “pipe” character is a broken vertical line on the keyboardon the keyboardMost fonts display it as a single line that Most fonts display it as a single line that looks like this: “ | ”looks like this: “ | ”Data goes in the start of a pipe and flows Data goes in the start of a pipe and flows out the endout the end

command1 | command2

1919 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Using pipesUsing pipes

Q. How many files are in my current directory?

ls | wc -w

2020 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Using pipesUsing pipes

ls | wc -w

The output of the ls

command is a list of file

names

The wc command takes the output of the ls command as

its input and displays a count of

the words

2121 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Using PipesUsing Pipes

ls | wc –w > file_count

The wc command sends its output to a file called

file_count

The spaces before and after

a pipe are optional

(you will find your commands easier to read if you use them)

The spaces before and after a file redirection symbol (>, >>) are also optional

2222 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005


UNIX commands were designed to do UNIX commands were designed to do “one thing very well”“one thing very well”We use a series of pipes between We use a series of pipes between simple UNIX commands to create simple UNIX commands to create complex commands linescomplex commands linesMuch of the power and flexibility of Much of the power and flexibility of the UNIX command line is built upon the UNIX command line is built upon “pipelines” of commands! “pipelines” of commands!

2323 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005


Command aCommand a | | Command bCommand b | | Command cCommand c

The stdout of command ' The stdout of command ' Command aCommand a' is ' is connected to the stdin of 'connected to the stdin of ' Command Command b', b',

and the stdout of ' and the stdout of ' Command bCommand b' is ' is connected to the stdin of ' connected to the stdin of ' Command cCommand c'.'.

2424 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

“Always-Remember-and-Never-Forget”...

RULE #1 – Use file redirection symbols such

as > and >> before a RULE #2 – Use a pipe before a

filenamecommand

2525 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005


RULE #1 ...

RULE #2 ...

2626 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005


RULE #3

Don’t use pipes when you don’t need them!

example:

cat users |more

more users

WRONG!

CORRECT!

2727 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005


RULE #3

Don’t use pipes when you don’t need them!

example:

cat users |head -5

head -5 users

WRONG!

CORRECT!

2828 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Skill Testing Question #1Skill Testing Question #1

On the command line what comes On the command line what comes AFTER a > or >> symbol?AFTER a > or >> symbol?

2929 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005


On the command line what comes AFTER a On the command line what comes AFTER a | symbol?| symbol?

3030 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005


When should you NOT use file redirection When should you NOT use file redirection OR piping?OR piping?

3131 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005




We’ve done this!

Now let’s do this!

Redirection and Piping

3232 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Redirecting the input of a command Redirecting the input of a command

by default commands get their “input” by default commands get their “input” from the terminal keyboardfrom the terminal keyboard

you can redirect “stdin” so that it you can redirect “stdin” so that it comes from a file or from another comes from a file or from another commandcommand

3333 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Redirecting the input of a command Redirecting the input of a command

Use the Use the less thanless than symbol symbol <<

this redirects the “standard input” or “channel 0” this redirects the “standard input” or “channel 0” of a command to come from a file instead of the of a command to come from a file instead of the keyboardkeyboard

this is the same as this is the same as 0<0< but you don’t need to type but you don’t need to type the “0” - it is “understood”.the “0” - it is “understood”.

You can also think of the 'You can also think of the '<<' symbol as being an ' symbol as being an arrow pointing from the file to the command, arrow pointing from the file to the command, meaning that the information will flow from the meaning that the information will flow from the file into the input of the command.file into the input of the command.

3434 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Redirecting the input of a commandRedirecting the input of a command

Examples:Examples:

cat < tempcat < temp

cat < temp > temp2cat < temp > temp2

mail –s “week 9 Notes” mail –s “week 9 Notes” [email protected]@senecac.on.ca < week9.html < week9.html

3535 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Redirecting the input of a commandRedirecting the input of a command

Are these the same?Are these the same?

cat < tempcat < tempcat tempcat temp

Are these the same?Are these the same?

cat < temp >temp2cat < temp >temp2 cp temp temp2cp temp temp2

and and

and and

3636 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

The < symbol redirects the input of The < symbol redirects the input of a command to come from a file a command to come from a file

Q. What if you want the input of a command Q. What if you want the input of a command to come from another command?to come from another command?

A. When you place a command after a pipe you are redirecting its input to come from a command instead of from the keyboard

3737 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005




We’ve done this!

We’ve done this!

Redirection and Piping

Let’s do this!

3838 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Error RedirectionError RedirectionBy default, stderr goes to the terminal screenBy default, stderr goes to the terminal screen

The “standard error” of a command can be The “standard error” of a command can be redirected to a file using the redirected to a file using the 2>2> symbol symbol

Source: Ling ZhuSource: Ling Zhu

3939 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Error RedirectionError Redirection

cp file1 file2 2cp file1 file2 2>> error.logerror.log

This would This would overwrite overwrite the file error.logthe file error.log

cp file1 file2 2cp file1 file2 2>>>>error.logerror.log

This would This would appendappend the error message of the cp the error message of the cp command to the file error.logcommand to the file error.log


Warning:There is no space in between

4040 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

A Special Place of UNIX/LinuxA Special Place of UNIX/Linux

/dev/null/dev/null Can be called bit bucketCan be called bit bucket Can be called black holeCan be called black hole


4141 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Use > or >> to redirect output to a Use > or >> to redirect output to a filefile

Use a | to redirect output to a Use a | to redirect output to a commandcommand

Q. What if you want to do both?Q. What if you want to do both?(this is on the test!)(this is on the test!)

A. The tee command

4242 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

The tee command duplicates the The tee command duplicates the standard output of a command standard output of a command

you can then redirect one “copy” of the you can then redirect one “copy” of the output to a file and one “copy” of the output to a file and one “copy” of the output to a commandoutput to a command

example:example:

ls –l | tee listing | morels –l | tee listing | more

the filename you are sending one copy of the output of the ls –l

command to

the command you are sending one copy of the output of the ls –l

command to

4343 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

The tee command duplicates the The tee command duplicates the standard output of a command standard output of a command

Example:Example:

ls –l | tee listing | morels –l | tee listing | more

if this file exists it will be overwritten

To append to an existing fileuse tee –a filename

ls –l | tee –a listing | more

4545 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Let’s review some commands then Let’s review some commands then practice “redirection” with them...practice “redirection” with them...

1.1. sortsort

2.2. grepgrep

4646 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Some Special symbols when Some Special symbols when you you grepgrep a pattern a pattern

^̂ - beginning of a line - beginning of a line $$ - end of a line - end of a line

4747 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

sortsort and and grep grepbelong to a group of UNIX utilities belong to a group of UNIX utilities

called “filters”called “filters”

Filters are utilities that:Filters are utilities that:

1.1. read from stdinread from stdin

2.2. write to stdoutwrite to stdout

3.3. do not change the contents of the original do not change the contents of the original filefile

4848 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

How do you learn a new How do you learn a new command?command?

1.1. Learn its purposeLearn its purpose

2.2. Learn its “syntax” including common Learn its “syntax” including common optionsoptions

3.3. Learn where it gets input and where it Learn where it gets input and where it sends its outputsends its output

4949 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Other UNIX utilities that are Other UNIX utilities that are “filters”“filters”

headhead

tailtail

wcwc

teetee

5050 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Filters can be used between pipes Filters can be used between pipes in a complex command linein a complex command line

Example:Example:

grep happy diary >childhoodgrep happy diary >childhood

sort mymarks | tee sortedmarks | moresort mymarks | tee sortedmarks | more

5151 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005

Commands that are Commands that are not not filters include:filters include:

cd cd

chmodchmod

cpcp

rmrm

datedate

echoecho

ls ls

mvmv

pwdpwd

who....who....

Warning!Warning!

You can not put these

commands between pipes!

Moral of the Story:

ALWAYS be careful about “what” you

put in a pipe!

week 9 - nov 7, 2005 1 week 9 agenda i/o redirection i/o redirection pipe pipe tee tee

Documents