Unix Talk #2Unix Talk #2

(sed)(sed)

22

You have learned…You have learned…

Regular expressions, grep, & egrepRegular expressions, grep, & egrep grep & egrep are tools used to search for grep & egrep are tools used to search for

text in a filetext in a file AWK -- powerfulAWK -- powerful What about SED?What about SED?

33

Things in common between awk and Things in common between awk and sedsed

They are invoked using similar syntaxThey are invoked using similar syntax Stream-oriented (hence Stream-oriented (hence streamstream editor) editor) Use regular expressions for pattern Use regular expressions for pattern

matchingmatching Allow the user to specify instructions in a Allow the user to specify instructions in a

scriptscript

44

Common syntaxCommon syntax

Command [options] script filenameCommand [options] script filename ScriptScript

– You tell the program what to do, which is called You tell the program what to do, which is called instructionsinstructions

– Each instruction has two partsEach instruction has two parts PatternPattern Procedure (actions)Procedure (actions)

– (Sound familiar?)(Sound familiar?)

55

SEDSED

A tool usually designed for a short line A tool usually designed for a short line substitution, deletion and printsubstitution, deletion and print

Allows for automation of editing process Allows for automation of editing process similar to those you might find in vi or ex (or similar to those you might find in vi or ex (or ed)ed)

Non-destructiveNon-destructive Reads in a stream of data from a file, Reads in a stream of data from a file,

performs a set of actions, & outputs the performs a set of actions, & outputs the resultsresults

66

SEDSED

SED reads the data one line at a time, make SED reads the data one line at a time, make a copy of the input line & places it in a buffer a copy of the input line & places it in a buffer called the pattern spacecalled the pattern space

Modifies that copy in the pattern spaceModifies that copy in the pattern space Outputs the copy to standard outputOutputs the copy to standard output

NOTE: 1.The pattern space holds the line of NOTE: 1.The pattern space holds the line of text currently being processedtext currently being processed

2. You don’t make changes to the original file2. You don’t make changes to the original file3. If you want to capture this output, what do 3. If you want to capture this output, what do

you do?you do?

77

SED SyntaxSED Syntax

OptionsOptions– nn

only prints matchesonly prints matches

– ff scriptfilescriptfile run commands in scriptfilerun commands in scriptfile

– ee allows multiple instructions on a single lineallows multiple instructions on a single line

Sed syntaxSed syntax– Sed [option] ‘instruction’ file(s)Sed [option] ‘instruction’ file(s)– Sed –f scriptfile file(s)Sed –f scriptfile file(s)

Must give sed instruction or scriptfileMust give sed instruction or scriptfile Can use file redirection to create a new fileCan use file redirection to create a new file

88

Sed optionsSed options

-e-e– Only needed when you supply more than one Only needed when you supply more than one

instruction on the command lineinstruction on the command line– sed –e ‘script’ –e ‘script’ filesed –e ‘script’ –e ‘script’ file

99

Print CommandPrint Command

cat /home/fac/pub/fruit_pricescat /home/fac/pub/fruit_prices

sed –n sed –n ''pp'' fruit_prices fruit_prices

What does it do?What does it do?

Try with out the –n. What happens?Try with out the –n. What happens?

1010

Print CommandPrint Command

Can specify zero, one or two addresses to print, the address Can specify zero, one or two addresses to print, the address can be a line number or the pattern for matchingcan be a line number or the pattern for matching– sed –n sed –n ''1p1p'' fruit_prices fruit_prices

The line counter does not reset for multiple input The line counter does not reset for multiple input filesfiles

– sed –n sed –n ''$p$p'' fruit_prices #prints last line fruit_prices #prints last line– sed –n sed –n ''6,8p6,8p'' #prints lines 8-10 #prints lines 8-10

sed –n sed –n ''/^$/p/^$/p'' fruit_prices fruit_prices sed –n sed –n ''1, /^$/p1, /^$/p'' fruit_prices # range fruit_prices # range sed –n sed –n ''/App*/p/App*/p'' fruit_prices fruit_prices sed –n sed –n ''/^[0-1]/p/^[0-1]/p'' fruit_prices fruit_prices sed –n sed –n ''/[^7-9]$/p/[^7-9]$/p'' fruit_prices fruit_prices

What happens if you remove the caret?What happens if you remove the caret?

1111

Print CommandPrint Command

sed –n sed –n ''/\$1\./p/\$1\./p'' fruit_prices fruit_prices sed –n sed –n ''/1./p/1./p'' fruit_prices fruit_prices

Need to use –n when printing, Need to use –n when printing, otherwise you get multiple otherwise you get multiple copies of linescopies of lines

1212

Read data into a variableRead data into a variable

Create a script:Create a script:read –p “Enter fruit name: ” fruitNameread –p “Enter fruit name: ” fruitNamesed –n “/$fruitName/p” fruit_pricessed –n “/$fruitName/p” fruit_prices

Always surround you patterns with “” or ‘’ to Always surround you patterns with “” or ‘’ to prevent problemsprevent problems

1313

Delete CommandDelete Command

sed sed ''/^A/d/^A/d'' fruit_prices fruit_prices

Cat the file after you have run Cat the file after you have run the command. Is the line gone?the command. Is the line gone?

1414

Delete CommandDelete Command

sed ‘1d’ fruit_pricessed ‘1d’ fruit_prices sed ‘$d’ fruit_pricessed ‘$d’ fruit_prices sed ‘/^$/d’ fruit_pricessed ‘/^$/d’ fruit_prices sed‘1,/^$/d’fruit_prices.txt > sed‘1,/^$/d’fruit_prices.txt > newfilenewfile

1515

SubstituteSubstitute To change one pattern to anotherTo change one pattern to another SyntaxSyntax

– s/pattern/replacement/flagss/pattern/replacement/flags FlagsFlags

– n: A number (1 to 512) indicating that a n: A number (1 to 512) indicating that a replacement should be made for only the replacement should be made for only the nth occurrence of the patternnth occurrence of the pattern

– g: Make changes globally on all g: Make changes globally on all occurrences in the pattern space. Normally occurrences in the pattern space. Normally only the first occurrence is replaces.only the first occurrence is replaces.

1616

SubstituteSubstitute

sed ‘s/Fruit/Fruit_list/’ fruit_pricessed ‘s/Fruit/Fruit_list/’ fruit_pricessed ‘s/a/A/g’ fruit_pricessed ‘s/a/A/g’ fruit_pricesTry the previous command without gTry the previous command without g

If you like to change the original file, you must do If you like to change the original file, you must do copy and redirect to update original filecopy and redirect to update original file cp fruit_prices fruit_prices.oldcp fruit_prices fruit_prices.oldsed ‘s/Fruit/Fruit_list/’ sed ‘s/Fruit/Fruit_list/’ fruit_prices.old>fruit_pricesfruit_prices.old>fruit_prices

1717

SubstituteSubstitute

Reuse the matched string with ‘&’Reuse the matched string with ‘&’ Sed ‘s/[0-9]\.[0-9][0-9]*/\$&/’ filenameSed ‘s/[0-9]\.[0-9][0-9]*/\$&/’ filename

1818

sed – the Stream Editor.1sed – the Stream Editor.1

sed is an editor for editing data on the fly as part of sed is an editor for editing data on the fly as part of a pipelinea pipeline

Usage:Usage:sed -e 'command' -e 'command' -e 'command' ...sed -e 'command' -e 'command' -e 'command' ...

– Reads stdin and applies the commands to Reads stdin and applies the commands to eacheach line in the line in the order they are on the command lineorder they are on the command line

– Prints each line of stdin to stdout Prints each line of stdin to stdout afterafter the commands the commands have been appliedhave been applied

– Most commands are s/ . . . / . . . / commandsMost commands are s/ . . . / . . . / commands Can use regexps in the 1Can use regexps in the 1stst part part Can use parentheses, back references in the 1Can use parentheses, back references in the 1stst part part Can use & and \1 \2 \3 . . . in the second partCan use & and \1 \2 \3 . . . in the second part Can append ‘g’ to the s/ . . . / . . . / command to change Can append ‘g’ to the s/ . . . / . . . / command to change allall

occurrences on a lineoccurrences on a line

1919

sed – the Stream Editor.2sed – the Stream Editor.2

Examples:Examples:– Print out all usernames from /etc/passwdPrint out all usernames from /etc/passwd

sed -e 's/^\([^:]*\):.*$/\1/' </etc/passwdsed -e 's/^\([^:]*\):.*$/\1/' </etc/passwd

– Print out ONLY the hidden files in the working directoryPrint out ONLY the hidden files in the working directory# delete lines that do NOT begin with a period# delete lines that do NOT begin with a periodls -a | sed -e '/^[^.]/d‘ls -a | sed -e '/^[^.]/d‘

OROR# print ONLY lines that DO begin with a period# print ONLY lines that DO begin with a period# NOTE: -n option suppresses printing unless # NOTE: -n option suppresses printing unless # indicated via a p command# indicated via a p commandls -a | sed -n -e '/^\./p'ls -a | sed -n -e '/^\./p'

– Print out file names followed by the time of modificationPrint out file names followed by the time of modification# NOTE: -r option enables extended regexps WITHOUT# NOTE: -r option enables extended regexps WITHOUT# the need to escape parentheses# the need to escape parenthesesls -l | sed -r -e 's/^([^ ]+ +){5}//' \ls -l | sed -r -e 's/^([^ ]+ +){5}//' \ -e 's/^(.*) (.*)/\2 -> \1/'-e 's/^(.*) (.*)/\2 -> \1/'

2020

AppendAppend

a \ text a \ text Appends text to the Appends text to the lineline followingfollowing the one the one

matchedmatched

sed ‘/^F/a\ #here is a list of the fruit’ sed ‘/^F/a\ #here is a list of the fruit’ fruit_pricesfruit_prices

2121

InsertInsert

i \ text i \ text Inserts text Inserts text prior to the lineprior to the line that is matched that is matched

sed ‘/^Pi/i\ Orange:sed ‘/^Pi/i\ Orange: $1.99’ $1.99’ fruit_prices.old>fruit_pricesfruit_prices.old>fruit_prices

2222

ScriptsScripts

When? When? Series of sed commands Series of sed commands SyntaxSyntax

sed –f scriptname datafilenamesed –f scriptname datafilename

2323

ScriptsScripts

Cat scriptTestCat scriptTests/^A/a/gs/^A/a/g

s/^P/p/s/^P/p/

sed –f scriptTest fruit_pricessed –f scriptTest fruit_prices sed –e ‘s/^A/a/g’ –e ‘s/^P/p/’ fruit_pricessed –e ‘s/^A/a/g’ –e ‘s/^P/p/’ fruit_prices

2424

Using sed in a PipelineUsing sed in a Pipeline

idid uid=500(yxp) gid=100(fac)uid=500(yxp) gid=100(fac)

id | sed ‘s/(.*$//’id | sed ‘s/(.*$//’ uid=500uid=500

2525

Using sed in a PipelineUsing sed in a Pipeline

cat fruit_prices | sed ‘s/A/a/g’cat fruit_prices | sed ‘s/A/a/g’

ls –l | sed –n pls –l | sed –n p

echo Hello | sed pecho Hello | sed p