Basic Text Processing,Redirection and Pipes

222

Lecture Overview

Basic text processing commands head, tail, wc

Redirection and pipes

Getting to know vi

Basic vi commands

Advanced vi commands

333

Text Processing Commands

UNIX is strongly text oriented

It contains many utilities to search, modify or view text files

This provides a simple mechanism for storing data and for passing it between applications and utilities

444

Viewing Portions of a File –head and tail

The head command shows only the first few lines of a file

With no options, head shows first 10 lines Command line options for head:

-N – specify number of lines to show (N is a number, not the letter N)

-cN – show first N bytes, not lines

head [options] [files]

555

Viewing Portions of a File –head and tail

The tail command is similar to head, but shows the last lines of the file

In addition, tail provides these options: +N – show lines starting with line N -f – output appended data as the file grows. With

this option, the number of lines printed keeps growing as the file grows

666

head and tail – Examples

Given a file called 'my_phones.txt':ADAMS, Andrew 7583BARRETT, Bruce 6466BAYES, Ryan 6585BECK, Bill 6346BENNETT, Peter 7456GRAHAM, Linda 6141HARMER, Peter 7484MAKORTOFF, Peter 7328MEASDAY, David 6494NAKAMURA, Satoshi 6453REEVE, Shirley 7391ROSNER, David 6830

777

head and tail – Examples

head -3 my_phones.txt

ADAMS, Andrew 7583BARRETT, Bruce 6466BAYES, Ryan 6585

tail -3 my_phones.txt

NAKAMURA, Satoshi 6453REEVE, Shirley 7391ROSNER, David 6830

head –c30 my_phones.txt

ADAMS, Andrew 7583BARRETT, Br

888

head and tail – Examples

tail +9 my_phones.txt

MEASDAY, David 6494NAKAMURA, Satoshi 6453REEVE, Shirley 7391ROSNER, David 6830

tail –c50 my_phones.txt

toshi 6453REEVE, Shirley 7391ROSNER, David 6830

999

Counting Words – wc

The wc command counts the number of bytes, words and lines in a file

Command line options for wc: -c – print the number of bytes -w – print the number of words -l – print the number of lines

With no options, all three are printed

wc [options] [files]

101010

Lecture Overview

Basic text processing commands head, tail, wc

Redirection and pipes

Getting to know vi

Basic vi commands

Advanced vi commands

111111

Input and Output Redirection

Normally, when you run a command, it gets input from the keyboard (stdin),and sends output to the screen (stdout) Errors are sent to a third stream (stderr), which

by default also points to the screen

Input and output (as well as errors) can be read from or written to a file, by using the redirection meta-characters: '<' and '>'

121212

Output Redirection

We use the '>' operator to redirect the output of a command

For example, to create a file containing a listing of '.c' files in the current directory:ls *.c > c_files.txtcat !$

boxes.cparse_tools.cshape.ctools.c

131313

Input Redirection

We use the '<' operator to redirect the input of a command

For example, to see only the last two lines in the file 'c_files.txt':tail -2 < c_files.txt

shape.ctools.c

141414

Input / Output Redirection

Input and output redirection can be combined in a single command

For example:cat -n < my_phones.txt > numbered.txthead -4 !$

head -4 numbered.txt 1 ADAMS, Andrew 7583 2 BARRETT, Bruce 6466 3 BAYES, Ryan 6585 4 BECK, Bill 6346

151515

Appending Output

When redirecting output to a file, the specified file is either created (if it does not exist), or overwritten (if it exists)

The '>>' operator tells the shell to append the output to the file, without overwriting it

To redirect stderr in addition to stdout, we use the operator '>&'

161616

Appending Output – Example

echo first line > lines.txtecho second line >> lines.txtecho third line >> lines.txtcat lines.txt

first linesecond linethird line

echo fourth line > lines.txtcat lines.txt

fourth line

171717

Forced Redirection

Output redirection fails if we try to: Overwrite an existing file Append to a non-existent file

This behavior can be overridden by using the '!' modifier

When '!' is added to a redirection operator, the redirection is forced to succeed

181818

Forced Redirection – Examples

The following will create the file 'lines.txt' if it does not exist,or overwrite it if it does:

This will also create 'lines.txt' if it does not exist, but append to it if it does

echo first line >! lines.txt

echo first line >>! lines.txt

191919

Pipes

The following commands count how many '.c' files are in the current directory

This is not very efficient, and also leaves behind a temporary file as a side effect

What we really want is to link the output of ls directly to the input of wc

ls *.c > c_files.txtwc -l < c_files.txt

202020

Pipes

A pipe takes the output of one command, and passes it as input to another

Program 1(ls *.c)

Program 1(ls *.c)

Program 2(wc –l)

Program 2(wc –l)

stdout(or temp file)

stdin (or temp file)

Pipe

212121

Pipes

The symbol for a pipe is a vertical bar – '|'

The format of a command line which uses a pipe is:

Now the two commands from the previous example can be simply combined:

command1 [args] | command2 [args]

ls *.c | wc -l

222222

Pipes

The use of pipes is actually a simple and easy form of inter-process communication

Pipes can also be chained, so complex utilities can be created by combining simple building blocks

When data passes through several pipes, these are sometimes referred to as filters

232323

Pipes – Examples

Print lines 6 and 7 of 'my_phones.txt':

Another way to get the same results:

To find the most recently changed file:

head -7 my_phones.txt | tail -2

GRAHAM, Linda 6141HARMER, Peter 7484

tail +6 my_phones.txt | head -2

ls -t | head -1

242424

Lecture Overview

Basic text processing commands head, tail, wc

Redirection and pipes

Getting to know vi

Basic vi commands

Advanced vi commands

252525

Editing Files With vi

vi is the standard text editor in UNIX

Unlike most editors or word processors that you have encountered, it is modal – the same input produces different results in different modes Insert mode is for inserting text Command mode is for everything else

262626

Editing Files With vi

vi has hundreds of commands, whose names require memorization

It does not have menus or a GUI

All of these characteristics make vi: Very uncomfortable for beginners, but – Very powerful for experts

272727

Starting With vi

To edit a file, type:

You are now in command mode, so you cannot insert any text!

To enter insert mode, type: 'i' Now you can type in text as with any other

editor or word processor

vi [options] [file]

282828

Basic vi Commands

To switch back to command mode, press Escape (this has no effect if you are already in command mode)

In command mode, you can move around using the arrow keys, or these keys: 'h', 'j', 'k', 'l'

To delete a character, type 'x'

292929

Terminating a vi Session

To exit vi, you need to switch to a third mode, called last line mode, by typing ':'

Then, type one of the following: w – to save your changes q – to quit an unmodified file wq – to save your changes and quit q! – to quit without saving your changes

303030

Learning vi

The best way to learn vi is: Start using it with a few basic commands Use available help sources to learn more features as

you need them

Getting help: man vi Typing ':help' within vi UNIX books, the internet, etc.

313131

Lecture Overview

Basic text processing commands head, tail, wc

Redirection and pipes

Getting to know vi

Basic vi commands

Advanced vi commands

323232

vi Command Types

And now for a more detailed look

vi commands can be divided into groups: Commands for entering insert mode Commands for moving the cursor Commands for deleting or changing text Commands for searching or replacing text Miscellaneous commands

333333

Ways to Enter Insert Mode

Command Description

i Insert before cursor

I Insert at start of line

a Append after cursor

A Append to end of line

o Open a line below current line

O Open a line above current line

343434

Moving the Cursor

Command Description

h, j, k, l Left, down, up, right

Arrow keys Left, down, up, right

w Forward to start of next word

b Backward to start of previous word

Ctrl-F Forward one screen

Ctrl-B Backward one screen

353535

Moving the Cursor

Command Description

Ctrl-U Forward half a screen

Ctrl-D Backward half a screen

Ctrl-E Shift text down one line in window (cursor stays on same line)

Ctrl-Y Shift text up one line in window (cursor stays on same line)

% Move to the "mate" of this parenthesis or bracket

363636

Moving the Cursor

Command Description

$ Move to end of current line

0 Move to start of current line

^ Move to first non-whitespace character in current line

nG, :n Go to line n

gg Go to start of file

G Go to end of file

373737

Deleting Text

Command Description

x Delete character under cursor

X Delete character before cursor

dd Delete entire line

dw Delete current word

D Delete until end of line

383838

Changing Text

Command Description

cc Change entire line

cw Change current word

C Change until end of line

r Replace character under cursor

R Start overwriting until ESC

s Change one character and insert

393939

Copy and Paste

Command Description

yy Yank (copy) current line

yw Yank current word

p Put after (or below) cursor

P (capital) Put before (or above) cursor

]p Put, with indent adjusted

404040

Lecture Overview

Basic text processing commands head, tail, wc

Redirection and pipes

Getting to know vi

Basic vi commands

Advanced vi commands

414141

Defining Range Using Motion

With some commands, it is possible to add a motion modifier which defines the range that they effect

The general format is:

where motion is any of the available motion commands (e.g. d$)

command[motion]

424242

Defining Range Using Motion

In the command dw, d is the command (delete), and w is the motion (word) dd (and similarly, cc) is a special case – operates

on whole lines

Examples:

c$ Change until end of line

dG Delete until end of file

y% Yank block (from current to pair bracket)

434343

Repeating Commands

Most vi commands can be repeated, using the following format:

repeat defines how many times the given command should be executed

repeat and motion can be combined in a single command (e.g. 3dh)

[repeat]command

444444

Combining Repetition and Motion – Examples

Note: In 4yw, 4 is the repetition;in y4w, it is part of the motion

5dd Delete 5 lines, starting from current

3x Delete 3 characters, starting from current

2w Move forward 2 words

3dw Delete 3 words, starting from current

4yw Yank 4 words, starting from current

y4w Same as above

454545

Searching for Patterns

pattern can also be a regular expression

Command Description

/pattern Search forward

?pattern Search backward

n Repeat previous search

N Repeat previous search in reverse direction

464646

Replacing Text

The following command can be used to search and replace a pattern

This searches from the current position The g flag defines whether to replace only the

first occurrence, or all matches pattern can also be a regular expression

:s/search_string/replace_string/[g]

474747

Additional Commands

Command Description

. Repeat previous command

u Undo last command

U Undo all changes on current line

Ctrl-R Redo last undo

~ Switch case and advance cursor

J Merge current and next lines

484848

Additional Commands

Command Description

>> Indent line one tab-stop

<< Un-indent line one tab-stop

'' Go back to position before last move

ma Mark current position as a

'a Jump to position marked as a

494949

Commands in Last Line Mode

Command Description

:r file Insert file in cursor position

:w file Write current file with new name

:f Display details of current file

:!command Run a shell command

:r !command Run command, and insert result

:set var Set a vi environment variable

505050

Last Line Mode – Examples

:!pwd Show current directory

:!ls List contents of current directory

:r !date Insert current date in cursor position

:set nu Display line numbers

:set nonu Do not display line numbers

:set ai Auto-indent new lines according to indentation of previous ones

:set noai Do not auto-indent new lines

515151

The vi Startup File – .exrc

vi provides various customization options

Whenever vi is started, it first reads a file named .exrc from your home directory

This file may contain any last line mode command, such as: set – to set vi environment variables map – to define new behavior for keys

525252

A Sample .exrc File

set autoindentset numberset tabstop=4set shiftwidth=4set expandtabset showmatch

map ^\ :r ~/template/header.txt^Mmap ^_ :r !date +\%D^Mmap #6 :r ~/template/header.txt^M/Date^M:r !date +"\%b \%d, \%Y"^MkJjjj0map #9 :w!%^M:!chmod u+x %^M:!%^M