Download - Sed Introduction
Colloquium - sedv1.0.1
A. Magee
March 26, 2010
1 / 22
Colloquium - sed, v1.0.1
A. Magee
Outline
1 IntroductionWhat does sed offer?When should I use sed?
2 Learning by exampleSample FileProbing for infoEighty-sixing specific linesThe almighty regular expression
3 Sed’s limitations
2 / 22
Colloquium - sed, v1.0.1
A. Magee
Outline
1 IntroductionWhat does sed offer?When should I use sed?
2 Learning by exampleSample FileProbing for infoEighty-sixing specific linesThe almighty regular expression
3 Sed’s limitations
2 / 22
Colloquium - sed, v1.0.1
A. Magee
Outline
1 IntroductionWhat does sed offer?When should I use sed?
2 Learning by exampleSample FileProbing for infoEighty-sixing specific linesThe almighty regular expression
3 Sed’s limitations
2 / 22
Colloquium - sed, v1.0.1
A. Magee
Introduction What?
What does sed offer?
sed is a stream editor.
It operates on a file or stream of characters where a newline characterterminates a line.
It can operate on specific lines that you describe.
It can make changes immediately without a conventional GUI window.
It can make programatic text manipulation quick and painless.
3 / 22
Colloquium - sed, v1.0.1
A. Magee
Introduction What?
What does sed offer?
sed is a stream editor.
It operates on a file or stream of characters where a newline characterterminates a line.
It can operate on specific lines that you describe.
It can make changes immediately without a conventional GUI window.
It can make programatic text manipulation quick and painless.
3 / 22
Colloquium - sed, v1.0.1
A. Magee
Introduction What?
What does sed offer?
sed is a stream editor.
It operates on a file or stream of characters where a newline characterterminates a line.
It can operate on specific lines that you describe.
It can make changes immediately without a conventional GUI window.
It can make programatic text manipulation quick and painless.
3 / 22
Colloquium - sed, v1.0.1
A. Magee
Introduction What?
What does sed offer?
sed is a stream editor.
It operates on a file or stream of characters where a newline characterterminates a line.
It can operate on specific lines that you describe.
It can make changes immediately without a conventional GUI window.
It can make programatic text manipulation quick and painless.
3 / 22
Colloquium - sed, v1.0.1
A. Magee
Introduction When?
When should I use sed?
For repetitive file edits or probes.
For editing a file at precisely defined places.
When you are too lazy (or smart) to open a WYSIWYG editor.
4 / 22
Colloquium - sed, v1.0.1
A. Magee
Introduction When?
When should I use sed?
For repetitive file edits or probes.
For editing a file at precisely defined places.
When you are too lazy (or smart) to open a WYSIWYG editor.
4 / 22
Colloquium - sed, v1.0.1
A. Magee
Introduction When?
When should I use sed?
For repetitive file edits or probes.
For editing a file at precisely defined places.
When you are too lazy (or smart) to open a WYSIWYG editor.
4 / 22
Colloquium - sed, v1.0.1
A. Magee
Examples Sample File
A sample file
Here’s a short file that we can play with, let’s call it sample.txt.
This is some text.
1234 Main St. #9Cambridge MA, 12345gold 3 roadsilver 3 streetbronze 1 avenuediamond 5 lane
5 / 22
Colloquium - sed, v1.0.1
A. Magee
Examples Printing
Example 1
> sed -n ’4p’ sample.txtCambridge MA, 12345
-n: Suppress default behavior of echoing each line as it is processed.
4: The line number to process (1addr).
p: Print the lines addressed.
6 / 22
Colloquium - sed, v1.0.1
A. Magee
Examples Printing
Example 2 & 3
> sed -n ’3,4p’ sample.txt1234 Main St. #9Cambridge MA, 12345
3,4: The line number range to process (1addr,2addr).
> sed -n ’$p’ sample.txtdiamond 5 lane
$: Special character means last line.
7 / 22
Colloquium - sed, v1.0.1
A. Magee
Examples Printing
Example 4 & 5
> sed -n "$p" sample.txt
returns nothing: Where is the error, bash or sed?
> num=5> sed -n "${num}p" sample.txtgold 3 road
${num}: bash interrupts this first because of the double quotes.Appendix:Bash notes
8 / 22
Colloquium - sed, v1.0.1
A. Magee
Examples Printing
Example 6
> sed -n ’1∼3p’ sample.txtThis is some text.Cambridge MA, 12345bronze 1 avenue
x∼y: Print every y’th line starting at x.
9 / 22
Colloquium - sed, v1.0.1
A. Magee
Examples Deleting
Example 7
> sed -i .bak ’2d’ sample.txt
-i: Edit the file in place, an extension can be provided to make abackup (.bak).
Notice we dropped the -n.
d: Delete the addressed lines.
To restore the file: mv sample.txt{.bak,} Appendix:Bash notes
10 / 22
Colloquium - sed, v1.0.1
A. Magee
Examples Deleting
Example 7 - cont.
> cat sample.txtThis is some text.1234 Main St. #9Cambridge MA, 12345gold 3 roadsilver 3 streetbronze 1 avenuediamond 5 lane> cat sample.txt.bakThis is some text.
1234 Main St. #9Cambridge MA, 12345gold 3 roadsilver 3 streetbronze 1 avenuediamond 5 lane
11 / 22
Colloquium - sed, v1.0.1
A. Magee
Examples Regex
Example 8
> sed ’s/ˆ$//g’ sample.txt
s: Search and replace command.
s/pattern/replacement/options
ˆ: Special character that means beginning of line.
$: Special character that means end of line.
Remember earlier when it meant end of file?Why is it different now?
12 / 22
Colloquium - sed, v1.0.1
A. Magee
Examples Regex
Example 8 - cont.
> sed ’s/ˆ$//g’ sample.txt
What does ˆ$ mean?
What happens when there is no replacement argument?
g: global option - Applies the command to every occurrence.
Other options:
# - Only replaces the #th occurrence.p - Prints the replacements made.
13 / 22
Colloquium - sed, v1.0.1
A. Magee
Examples Regex
Example 8 - cont.
> sed ’s/ˆ$//g’ sample.txt
What does ˆ$ mean?
What happens when there is no replacement argument?
g: global option - Applies the command to every occurrence.
Other options:
# - Only replaces the #th occurrence.p - Prints the replacements made.
13 / 22
Colloquium - sed, v1.0.1
A. Magee
Examples Regex
Example 8 - cont.
> sed ’s/ˆ$//g’ sample.txt
What does ˆ$ mean?
What happens when there is no replacement argument?
g: global option - Applies the command to every occurrence.
Other options:
# - Only replaces the #th occurrence.p - Prints the replacements made.
13 / 22
Colloquium - sed, v1.0.1
A. Magee
Examples Regex
Example 9
> sed -E ’s/[0-9]+/###/g’ sample.txtThis is some text.
### Main St. ####Cambridge MA, ###gold ### roadsilver ### streetbronze ### avenuediamond ### lane
-E: Use extended regex (very common, suggest alias). Appendix:Bash notes
[0-9]+: Match one or more digits.
###: Replacement is three octothorpes.
14 / 22
Colloquium - sed, v1.0.1
A. Magee
Examples Regex
Example 10
> sed -E ’s/[0-9]/#/g’ sample.txtThis is some text.
#### Main St. ##Cambridge MA, #####gold # roadsilver # streetbronze # avenuediamond # lane
[0-9]: Match one digit.
#: Replacement is a single octothorpe.
Notice the length of the hidden number is the same as the actualnumber.
15 / 22
Colloquium - sed, v1.0.1
A. Magee
Examples Regex
Example 11
> sed -n -E ’s/[0-9]/#/gp’ sample.txt#### Main St. ##Cambridge MA, #####gold # roadsilver # streetbronze # avenuediamond # lane
From earlier, the -n suppresses the default printing behavior.
The p option prints lines that were matched with replacements.
Effect is that non-matched lines are not printed.
16 / 22
Colloquium - sed, v1.0.1
A. Magee
Examples Regex
Example 12
> sed -e ’s/ˆ$//’ -e ’s/ /,/g’ sample.txtThis,is,some,text.
1234,Main,St.,#9Cambridge,MA,,12345gold,3,roadsilver,3,streetbronze,1,avenuediamond,5,lane
s/ˆ$//: We’ve seen this before.
-e: Allows more than one command (repeat for each command).Slightly more efficient.
s/ /,/g: What does this do?
17 / 22
Colloquium - sed, v1.0.1
A. Magee
Sed’s limitations
What basic sed can’t do
Non-greedy matching (easily). Recommend ssed (super-sed).
Multiple line matches, or any operation that needs the newlinecharacter.
Arithmetics, don’t even try. Use awk instead.
18 / 22
Colloquium - sed, v1.0.1
A. Magee
Appendix
4 AppendixQuick notes on bashComplex examples
19 / 22
Colloquium - sed, v1.0.1
A. Magee
Appendix Quick notes on bash
Bashiness
See the bash presentation for much more info.From the bash reference manual:
The braces ${...} are required when a parameter is followed by acharacter that is not to be interpreted as part of its name.
Brace expansion is a mechanism by which arbitrary strings may begenerated. ’sample.txt{.bak,}’ expands literally to ’sample.txt.baksample.txt’ because the second argument is null
Aliases are used to keep shortcuts to commands:alias sed=’sed -E’
20 / 22
Colloquium - sed, v1.0.1
A. Magee
Appendix Complex examples
Some harder stuff
> sed -E -i ’s/[[:space:]]+/,/g’ somefile> sed -i ’s/.$//’ somefile> sed -E -i ’/ˆ#/ s/[0-9]+//’ somefile> sed -i ’\:s:[0-9][0-9]*::g’ somefile
21 / 22
Colloquium - sed, v1.0.1
A. Magee
Appendix Complex examples
Harder stuff explained
1 Replace all groups of whitespace with a comma.
2 Convert Windows line endings to Unix line endings. The .$ matches\r\n and replaces the \r with null.
3 /.../ gives addresses matching a pattern, like awk and perl. Deletesthe first number of any line that starts with a #.
4 Starting with \ redefines the argument separator; a colon in this case.
5 Notice that [0-9]+ requires -E, whereas [0-9][0-9]* does not. Theymatch the same expressions.
22 / 22
Colloquium - sed, v1.0.1
A. Magee