what is awk awk is a programming language used for manipulating data and generating reports the data...

What is Awk

Awk is a programming language used for manipulating data and generating reports

The data may come from standard input, one or more files, or as output from a process

Awk scans a file (or input) line by line, from the first to the last line, searching for lines that match a specified pattern and performing selected actions (enclosed in curly braces) on those lines.

If there is a pattern with no specific action, all lines that match the pattern are displayed;

If there is an action with no pattern, all input lines specified by the action are executed upon.

Which Awk

The command is awk if using the old version, nawk if using the new version, and gawk is using the gnu version

Awk’s format

An awk program consists of: the awk command the program instructions enclosed in quotes (or a file) ,

and the name of the input file

If an input file is not specified, input comes from standard input (stdin), the keyboard

Awk instructions consists of patterns, actions, or a combination of patterns and actions

A pattern is a statement consisting of an expression of some type

Awk’s format (continue.)

Actions consist of one or more statements separated by semicolons or new lines and enclosed in curly braces

Patterns cannot be enclosed in curly braces, and consist of regular expressions enclosed in forward slashes or expressions consisting of one or more of the many operators provided by awk

awk commands can be typed at the command line or in awk script files

The input lines can come from files, pipes, or standard input


Format:nawk 'pattern' filename

nawk '{action}' filename

nawk 'pattern {action}' filename

Input from Files

Example 1:

$ cat employeesChen Cho 5/19/63 203-344-1234 $76Tom Billy 4/12/45 913-972-4536 $102Larry White 11/2/54 908-657-2389 $54Bill Clinton 1/14/60 654-576-4114 $201Steve Ann 9/15/71 202-545-8899 $58

$ nawk '/Tom/' employeesTom Billy 4/12/45 913-972-4536 $102$

Input from Files (continue.)

Example 2:

$ cat employeesChen Cho 5/19/63 203-344-1234 $76Tom Billy 4/12/45 913-972-4536 $102Larry White 11/2/54 908-657-2389 $54Bill Clinton 1/14/60 654-576-4114 $201Steve Ann 9/15/71 202-545-8899 $58

$ nawk '{print $1}' employeesChenTomLarryBillSteve


Example 3:

$ cat employeesChen Cho 5/19/63 203-344-1234 $76Tom Billy 4/12/45 913-972-4536 $102Larry White 11/2/54 908-657-2389 $54Bill Clinton 1/14/60 654-576-4114 $201Steve Ann 9/15/71 202-545-8899 $58$ nawk '/Steve/{print $1, $2}' employeesSteve Ann

The print function

The default action is to print the lines that are matched to the screen

The print function can also be explicitly used in the action part of awk as {print}

The print function accepts arguments as variables, computed values, or string constants

String must be enclosed in double quotes Commas are used to separate the arguments: if

commas are not provided, the arguments are concatenated together

The print function (continue.)

The comma evaluates to the value of the output field separator (OFS), which is by default a space

The output of the print function can be redirected or piped to another program, and another program can be piped to awk for printing

The print function (continue.)

Example:

$ dateFri Feb 9 07:49:28 EST 2001 $ date | nawk '{ print "Month: " $2 "\nYear: ", $6}'Month: FebYear: 2001

Escape sequences

Escape sequences are represented by a backslash and a letter or number

Escape sequence

Meaning

\b Backspace

\f Form feed

\n Newline

\r Carriage return

\t Tab

\047 Octal value 47, a single quote

\c c represents any other character, e.g., \"

Escape sequences (continue.)

Example:

$ cat employeesChen Cho 5/19/63 203-344-1234 $76Tom Billy 4/12/45 913-972-4536 $102Larry White 11/2/54 908-657-2389 $54Bill Clinton 1/14/60 654-576-4114 $201Steve Ann 9/15/71 202-545-8899 $58 $ nawk '/Ann/{print "\t\tHave a nice day, " $1, $2 "\!"}' employees Have a nice day, Steve Ann!

The printf Function

The printf function can be used for formatting fancy output

The printf function returns a formatted string to standard output, like the printf statement in C.

Unlike the print function, printf does not provide a newline. The escape, \n, must be provided if a newline is desired

When an argument is printed, the place where the output is printed is called the field, and when the width of the field is the number of characters contained in that field

The printf Function (continue.)

Example 1:

$ echo "UNIX" | nawk ' {printf "|%-15s|\n", $1}'|UNIX |$ echo "UNIX" | nawk '{printf "|%15s|\n", $1}'| UNIX|$


Example 2:$ cat employeesChen Cho 5/19/63 203-344-1234 $76Tom Billy 4/12/45 913-972-4536 $102Larry White 11/2/54 908-657-2389 $54Bill Clinton 1/14/60 654-576-4114 $201Steve Ann 9/15/71 202-545-8899 $58$ nawk '{printf "The name is: %-15s ID is %8d\n", $1, $3}' employeesThe name is: Chen ID is 5The name is: Tom ID is 4The name is: Larry ID is 11The name is: Bill ID is 1The name is: Steve ID is 9$


Conversion Character

Definition

c Character

s String

d Decimal number

ld Long decimal number

u Unsigned decimal number

lu Long unsigned decimal number


Conversion Character Definition

x Hexadecimal number

lx Long hexadecimal number

o Octal number

lo Long octal number

e Floating point number in scientific notation (e-notation)

f Floating point number

g Floating point number using either e or f conversion, whichever takes the least space


Modifier Character Definition

- Left-justification modifier

# Integers in octal form are displayed with a leading 0; integers in hexadecimal form are displayed with a leading 0x.

+ For conversions using d,e,f, and g, integers are displayed with a numeric sign + or -

0 The displayed value is padded with zeros instead of white space


Printf Format Specifier What it Does

Given x=‘A’,y=15,z=2.3, and $1 = Bob Smith:

%c Prints a single ASCII character.printf("The character is %c\n",x) prints: The character is A

%d Prints a decimal numberprintf("The boy is %d years old\n", y) prints: The boy is 15 years old

%e Prints the e notation of a numberprintf("z is %f\n", z) prints: z is 2.3e+01

%f Prints a floating point numberprintf("z is %e\n", z) prints: z is 4.600000


Printf Format Specifier

What it Does

Given x=‘A’,y=15,z=2.3, and $1 = Bob Smith:

%o Prints the octal value of a numberprintf("y is %o\n",y) prints: y is 16

%s Prints a string of charactersprintf("The name of the culprit is %s\n", $1) prints: The name of the culprit is Bob Smith

%x Prints the hex value of a numberprintf("y is %x\n", y) prints: y is F

awk commands from within a file (continue.)

If awk commands are placed in a file, the -f option is used with the name of the awk file, followed by the name of the input file to be processed

A record is read into awk's buffer and each of the commands in the awk file are tested and executed for that record

If an action is not controlled by a pattern, the default behavior is to print the entire record

If a pattern does not have an action associated with it, the default is to print the record where the pattern matches an input line



Example:Chen Cho 5/19/63 203-344-1234 $76Tom Billy 4/12/45 913-972-4536 $102Larry White 11/2/54 908-657-2389 $54Bill Clinton 1/14/60 654-576-4114 $201Steve Ann 9/15/71 202-545-8899 $58$ cat awkfile/^Steve/{print "Hello Steve!"}{print $1, $2, $3}$ nawk -f awkfile employeesChen Cho 5/19/63Tom Billy 4/12/45Larry White 11/2/54Bill Clinton 1/14/60Hello Steve!Steve Ann 9/15/71

Records

By default, each line is called a record and is terminated with a newline

The Record Separator

By default, the output and input record separator (line separator) is a carriage return, stored in the built-in awk variables ORS and RS, respectively

The ORS and RS values can be changed, but only in a limited fashion

The $0 Variable

An entire record is referenced as $0 by awk When $0 is changed by substitution or

assignment, the value of NF, the number of fields, may be changed

The newline value is stored in awk's built-in variable RS, a carriage return by default

The $0 Variable (continue.)

Example:

$ cat employeesChen Cho 5/19/63 203-344-1234 $76Tom Billy 4/12/45 913-972-4536 $102Larry White 11/2/54 908-657-2389 $54Bill Clinton 1/14/60 654-576-4114 $201Steve Ann 9/15/71 202-545-8899 $58$ nawk '{print $0}' employeesChen Cho 5/19/63 203-344-1234 $76Tom Billy 4/12/45 913-972-4536 $102Larry White 11/2/54 908-657-2389 $54Bill Clinton 1/14/60 654-576-4114 $201Steve Ann 9/15/71 202-545-8899 $58

The NR Variable

The number of each record is stored in awk's built-in variable, NR

After a record has been processed, the value of NR is incremented by one

The NR Variable (continue.)

Example:

$ cat employeesChen Cho 5/19/63 203-344-1234 $76Tom Billy 4/12/45 913-972-4536 $102Larry White 11/2/54 908-657-2389 $54Bill Clinton 1/14/60 654-576-4114 $201Steve Ann 9/15/71 202-545-8899 $58$ nawk '{print NR, $0}' employees1 Chen Cho 5/19/63 203-344-1234 $762 Tom Billy 4/12/45 913-972-4536 $1023 Larry White 11/2/54 908-657-2389 $544 Bill Clinton 1/14/60 654-576-4114 $2015 Steve Ann 9/15/71 202-545-8899 $58

Fields

Each record consists of words called fields which, by default, are separated by white space, that is, blank spaces or tabs. Each of these words is called a field, an awk keeps track of the number of fields in its built-in variable, NF

The value of NF can vary from line to line, and the limit is implementation-dependent, typically 100 fields per line

Fields (continue.)

Example 1:$1 $2 $3 $4 $5Chen Cho 5/19/63 203-344-1234 $76Tom Billy 4/12/45 913-972-4536 $102Larry White 11/2/54 908-657-2389 $54Bill Clinton 1/14/60 654-576-4114 $201Steve Ann 9/15/71 202-545-8899 $58$ nawk '{print NR, $1, $2, $5}' employees1 Chen Cho $762 Tom Billy $1023 Larry White $544 Bill Clinton $2015 Steve Ann $58

Fields (continue.)

Example 2:

nawk '{print $0, NF}' employeesChen Cho 5/19/63 203-344-1234 $76 5Tom Billy 4/12/45 913-972-4536 $102 5Larry White 11/2/54 908-657-2389 $54 5Bill Clinton 1/14/60 654-576-4114 $201 5Steve Ann 9/15/71 202-545-8899 $58 5

The Input Field Separator

• awk's built-in variable, FS, holds the value of the input field separator.

• When the default value of FS is used, awk separates fields by spaces and/or tabs, stripping leading blanks and tabs

• The FS can be changed by assigning new value to it, either:– in a BEGIN statement, or– at the command line

The Input Field Separator (continue.)

• To change the value of FS at the command line, the –F option is used, followed by the character representing the new separator

The Input Field Separator (continue.)

• Example:

$ cat employeesChen Cho:5/19/63:203-344-1234:$76Tom Billy:4/12/45:913-972-4536:$102Larry White:11/2/54:908-657-2389:$54Bill Clinton:1/14/60:654-576-4114:$201Steve Ann:9/15/71:202-545-8899:$58$ nawk -F: '/Tom Billy/{print $1, $2}' employeesTom Billy 4/12/45

Using More than One Field Separator

• You may specify more than one input separator• If more than one character is used for the field

separator, FS, then the string is a regular expression and is enclosed in square brackets

• Example

$ nawk -F'[ :\t]' '{print $1, $2, $3}‘ employeesChen Cho 5/19/63Tom Billy 4/12/45Larry White 11/2/54Bill Clinton 1/14/60Steve Ann 9/15/71

The Output Field Separator

• The default output field separator is a single space and is stored in awk's internal variable, OFS

• The OFS will not be evaluated unless the comma separates the fields

• Example:

$ cat employeesChen Cho:5/19/63:203-344-1234:$76Tom Billy:4/12/45:913-972-4536:$102Larry White:11/2/54:908-657-2389:$54Bill Clinton:1/14/60:654-576-4114:$201Steve Ann:9/15/71:202-545-8899:$58$ nawk -F: '/Tom Billy/{print $1 $2 $3 $4}' employeesTom Billy4/12/45913-972-4536$102

Patterns

• A pattern consists of– a regular expression,– an expression resulting in a true or false condition, or – a combination of these

• When reading a pattern expression, there is an implied if statement

Patterns (continue.)

• Example:

$ cat employeesChen Cho 5/19/63 203-344-1234 $76Tom Billy 4/12/45 913-972-4536 $102Larry White 11/2/54 908-657-2389 $54Bill Clinton 1/14/60 654-576-4114 $201Steve Ann 9/15/71 202-545-8899 $58$ nawk '/Tom/' employeesTom Billy 4/12/45 913-972-4536 $102 $ nawk '$4 < 40' employeesChen Cho 5/19/63 203-344-1234 $76Steve Ann 9/15/71 202-545-8899 $58

Actions

• Actions are statements enclosed within curly braces and separated by semicolons

• Actions can be simple statements or complex groups of statements

• Statements are separated – by semicolons, or– by a newline if placed on their own line

Regular Expressions

• A regular expression to awk is a pattern that consists of characters enclosed in forward slashes

• Example 1:

• Example 2:

$ nawk '/Steve/' employeesSteve Ann 9/15/71 202-545-8899 $58

$ nawk '/Steve/{print $1, $2}' employeesSteve Ann

Regular Expression Meta characters

^ Matches at the beginning of string

$ Matches at the end of string

. Matches for a single character

* Matches zero or more of preceding character

+ Matches for one or more of preceding character

? Matches for zero or one of preceding character

[ABC] Matches for any one character in the set of characters, i.e., A,B, or C

Regular Expression Meta characters (continue.)

[^ABC] Matches characters not in the set of characters, i.e.,A,B or C

[A-Z] Matches for any character in the range from A to Z

A|B Matches either A or B

(AB)+ Matches one or more sets of AB

\* Matches for a literal asterisk

& Used in the replacement to represent what was found in the search string

Regular Expressions (continue.)

• Example 3:

• Example 4:

$ nawk '/^Steve/' employeesSteve Ann 9/15/71 202-545-8899 $58

$ nawk '/^[A-Z][a-z]+ /' employeesChen Cho 5/19/63 203-344-1234 $76Tom Billy 4/12/45 913-972-4536 $102Larry White 11/2/54 908-657-2389 $54Bill Clinton 1/14/60 654-576-4114 $201Steve Ann 9/15/71 202-545-8899 $58

The Match Operator

• The match operator, the tilde (~), is used to match an expression within a record or a field

• Example 1:

$ cat employeesChen Cho 5/19/63 203-344-1234 $76Tom Billy 4/12/45 913-972-4536 $102Larry White 11/2/54 908-657-2389 $54Bill Clinton 1/14/60 654-576-4114 $201Steve Ann 9/15/71 202-545-8899 $58$ nawk '$1 ~ /[Bb]ill/' employeesBill Clinton 1/14/60 654-576-4114 $201

The Match Operator (continue.)

• Example 2:

$ nawk '$1 !~ /lee$/' employeesChen Cho 5/19/63 203-344-1234 $76Tom Billy 4/12/45 913-972-4536 $102Larry White 11/2/54 908-657-2389 $54Bill Clinton 1/14/60 654-576-4114 $201Steve Ann 9/15/71 202-545-8899 $58

awk Commands in a Script File

• When you have multiple awk pattern/action statements, it is often easier to put the statements in a script

• The script file is a file containing awk comments and statements

• If statements and actions are on the same line, they are separated by semicolons

• Comments are preceded by a pound (#) sign

awk Commands in a Script File (continue.)

• Example:$ cat employeesChen Cho:5/19/63:203-344-1234:$76Tom Billy:4/12/45:913-972-4536:$102Larry White:11/2/54:908-657-2389:$54Bill Clinton:1/14/60:654-576-4114:$201Steve Ann:9/15/71:202-545-8899:$58$ cat info# My first awk script by Abdelshakour Abuzneid# Script name: info; Date: February 09, 2001/Tom/{print "Tom's birthday is "$3}/Bill/{print NR, $0}/^Steve/{print "Hi Steve. " $1 " has a salary of " $4 "."}#End of info script

awk Commands in a Script File(continue.)

• Example (continue.):

• To view info script, click here

$ nawk -F: -f info employeesTom's birthday is 913-972-45362 Tom Billy:4/12/45:913-972-4536:$1024 Bill Clinton:1/14/60:654-576-4114:$201Hi Steve. Steve Ann has a salary of $58.

Comparison Expressions

• Comparison expressions match lines where if the condition is true, the action is performed

• The value of the expression evaluates true, and 0 if false• Comparison expressions match lines where if the

condition is true, the action is performed• The value of the expression evaluates true, and 0 if false

Relational Operators

Operator

Meaning Example

< Less than x < y

<= Less than or equal to x <= y

== Equal to x == y

!= Not equal to x != y

>= Greater than or equal to x >= y

> Greater than x > y

~ Matched by regular expression x ~ / y/

!~ Not matched by regular expression

x !~ /y/

Relational Operators (continue.)

• Example:

$ cat employeesChen Cho 5/19/63 203-344-1234 76Tom Billy 4/12/45 913-972-4536 102Larry White 11/2/54 908-657-2389 54Bill Clinton 1/14/60 654-576-4114 201Steve Ann 9/15/71 202-545-8899 58$ nawk '$5 == 201' employeesBill Clinton 1/14/60 654-576-4114 201$ nawk '$5 > 100' employeesTom Billy 4/12/45 913-972-4536 102 $ nawk '$2 ~ /Ann/ ' employeesSteve Ann 9/15/71 202-545-8899 58

Relational Operators (continue.)

• Example (continue):

$ nawk '$2 !~ /Ann/ ' employeesChen Cho 5/19/63 203-344-1234 76Tom Billy 4/12/45 913-972-4536 102Larry White 11/2/54 908-657-2389 54Bill Clinton 1/14/60 654-576-4114 201

Conditional Expressions

• A conditional expression uses two symbols, the

question mark and the colon, to evaluate expression

• Format:

conditional expression1 ? expression2 : expression3

Conditional Expressions (continue.)

• Example:

$ nawk '{max=($1 > $2) ? $1 : $2; print max}' employeesChoTomWhiteClintonSteve

Computation

• awk performs all arithmetic in floating point

Operator Meaning Example

+ Add x + y

- Substract x – y

* Multiply x * y

/ Divide x / y

% Modulus x % y

^ Exponentiation

x ^ y

Computation (continue.)

• Example:

$ nawk '$3 * $4 > 500' filename

Compound Patterns

• Compounds patterns are expressions that combine patterns with logical operators

Operator Meaning Example

&& Logical AND a && b

|| Logical OR a || b

! NOT !a

Compound Patterns (continue.)

• Example :

$ nawk '$2 > 5 && $2 <= 15' employees$$ nawk '$5 == 1000 || $3 > 50' employeesSteve Ann 9/15/71 202-545-8899 58$

Range Patterns

• Range patterns match from the first occurrence of one pattern to the first occurrence of the second pattern, then match for the next occurrence of the second pattern, etc

• If the first pattern is matched and the second pattern is not found, awk will display all lines to the end of the file

• Example :

$ nawk '/Tom/,/Steve/' employeesTom Billy 4/12/45 913-972-4536 102Larry White 11/2/54 908-657-2389 54Bill Clinton 1/14/60 654-576-4114 201Steve Ann 9/15/71 202-545-8899 58$

Numeric and String Constants

• Numeric constants can be represented as– Integer like 243– Floating point numbers like 3.14, or– Numbers using scientific notation like .723E-1 or

3.4• Strings, such as Hello are enclosed in double quotes

Initialization and Type Coercion

• A variable can be – a string– a number, or– both

• When it is set, it becomes the type of the expression on the right-hand side of the equal sign

• Initialized variables have the value zero or the value " ", depending on the context in which they are used

User-Defined Variables

• User-defined variables consist of letters, digits, and underscores, and cannot begin with a digit

• Variables in awk are not declared• If the variable is not initialized, awk initializes string

variables to null and numeric variables to zero• Variables are assigned values with awk's assignment

operators• Example :$ nawk '$1 ~ /Tom/ {wage = $5 * 40; print wage}' employees4080

Increment and Decrement Operators

•The expression x++ is equivalent to x=x+1•The expression x– is equivalent to x=x-1•You can use the increment and decrement operators either

preceding operator, as in ++x, or after the operator, as x++{x = 1;y = x++; print x, y}name=“Nancy” name is stringx++ x is a number; x is

initialized to zero and incremented by 1

number=35 number is a number

User-Defined Variables at the Command line

•A variable can be assigned a value at the command line and passed into awk script

•Example:

$ nawk –F: -f awkscript month=4 year=1999 filename

The –v Option (nawk)

•The –v option provided by nawk allows command line arguments to be processed within a BEGIN statement

•For each argument passed at the command line, there must be a –v option preceding it

Built-in Variables

•Built-in variables have uppercase names. They can be used in expressions and can be resetVariable Name

Variable Contents

ARGC Number of command line argument

ARGV Array of command line arguments

FILENAME Name of current input file

FNR Record number in current file

FS The input field separator, by default a space

Built-in Variables (continue.)

Variable Name Variable Contents

NF Number of fields in current record

NR Number of record so far

OFMT Output format for numbers

OFS Output field separator

ORS Output record separator

RLENGTH Length of string matched by match function

RS Input record separator

RSTART Offset of string matched by match function

SUBSEP Subscript separator

Built-in Variables (continue.)

• Example:

$ nawk -F: '$1 == "Steve Ann"{print NR, $1, $2, $NF}' employees25 Steve Ann 9/15/71 $58

BEGIN Patterns

•The BEGIN pattern is followed by an action block that is executed before awk processes any lines from the input file

•The BEGIN action is often used to change the value of the built-in variables, OFS, RS, FS, and so forth, to assign initial values to user-defined variables, and to print headers or titles as part of the output

BEGIN Patterns (continue.)

•Example 1 :$ nawk 'BEGIN{FS=":"; OFS="\t"; ORS="\n\n"}{print $1,$2,$3}' employees2Chen Cho 5/19/63 203-344-1234

Tom Billy 4/12/45 913-972-4536

Larry White 11/2/54 908-657-2389

Bill Clinton 1/14/60 654-576-4114

Steve Ann 9/15/71 202-545-8899

$

BEGIN Patterns (continue.)

•Example 2 :

$ nawk 'BEGIN{print "Make Year"}'Make Year

END Patterns

•END patterns do not match any input lines, but executes any actions that are associated with the END pattern. END patterns are handled after all lines of input have been processed

•Examples:

$ nawk 'END{print "The number of records is " NR }' employeesThe number of records is 5

$ nawk '/Steve/{count++}END{print "Steve was found " count " times."}' employeesSteve was found 1 times.

$

Output Redirection

•When redirecting output from within awk to a UNIX file, the shell redirection operators are used

•The filename must be enclosed in double quotes•Once the file is opened, it remains opened until explicitly closed

or the awk program terminates•Example:

$ nawk '$5 >= 70 {print $1, $2 > "passing_file" }' employees$ cat passing_fileChen ChoTom BillyBill Clinton

The getline Function

•Reads input from– The standard input,– a pipe, or– a file other than from the current file being processed

•It gets the next line of input and sets the NF, NR and the FNR built-in variables

•The getline function returns – 1 if a record is found– 0 if EOF (end of file)– -1 if there is an error

The getline Function (continue.)

•Examples :$ nawk 'BEGIN{ "date" | getline d; print d}' employees2Fri Feb 9 09:39:53 EST 2001$ nawk 'BEGIN{ "date" | getline d; split( d, mon); print mon[2]}' employeesFeb $ nawk 'BEGIN{while("ls" | getline) print}UNIXvarfilevarfile2varfile3varfile4varfile5varfile6

The getline Function (continue.)

•Examples :$ nawk 'BEGIN{ print "What is your name?" ;\> getline name < "/dev/tty"}\> $1 ~ name {print "Found " name " on line ", NR "."}\> END{print "See ya, " name "."}' employeesWhat is your name?abdulSee ya, abdul.$

Pipes

•If you open a pipe in an awk program, you must close it before opening another one

•The command on the right-hand side of the pipe symbol is enclosed in double quotes

Pipes (continue.)

•Example:$ cat namesjhon smithalice chebatony tramdan savageeliza goldborg$ nawk '{print $1, $2 | "sort -r +1 -2 +0 -1 "}' namestony tramjhon smithdan savageeliza goldborgalice cheba$

Closing Files and Pipes

•The pipe remains opened until awk exits

•Statements in the END block will also be affected by

the pipe. The first line in the END block closes the pipe•Example:( In script){ print $1, $2, $3 | “ sort –r +1 –2 +0 –1”}END{Close(“sort –r +1 –2 +0 –1”)<rest of statement> }

The System Function

•The built-in system function takes a UNIX (operating system command) command as its argument, executes the command, and returns the exit status to the awk program

•The UNIX command must be enclosed in double quotes

•Example:( In script){System ( “cat “ $1 )System ( “clear” )}

If Statement

•Format:If (expression) {

statement; statement; …}

If/else Statement

•Format:{If (expression) {


else {statement; statement; …}

}

If/else Statement

•Example:

$ nawk '{if($6 > 50) print $1 "Too high"; \> else print "Range is OK"}' namesRange is OKRange is OKRange is OKRange is OKRange is OK$

If/else else if Statement

•Format:{ If (expression) {


else if (expression) {statement; statement; …}

else if (expression) {statement; statement; …}

else {statement; statement; …}

}

Loops

• Loops are used to iterate through the field within a record and to loop through the elements of an array in the END block

While Loop

•The first step in using a while loop is to set a variable to an initial value

•The do/while loop is similar to the while loop, except that the expression is not tested until the body of the loop is executed at least once

While Loop (continue.)

•Example:$ nawk '{ i = 1; while (i <= NF ) { print NF, $i; i++}}' names2 jhon2 smith2 alice2 cheba2 tony2 tram2 dan2 savage2 eliza2 goldborg$

for Loop

•for loop requires three expressions within the parentheses: the initialization expression, the test expression and the expression to update the variables within the test expression

•The first statement within the parentheses of the for loop can perform only one initialization

for Loop (continue.)

•Example:$ nawk '{ i = 1; while (i <= NF ) { print NF, $i; i++}}' names2 jhon2 smith2 alice2 cheba2 tony2 tram2 dan2 savage2 eliza2 goldborg$

break and continue Statement

•The break statement lets you break out of a loop if a certain condition is true

•The continue statement causes the loop to skip any statement that follow if a certain condition is true, and returns control to the top of the loop, starting at the next iteration

•Example:

(In Script){ if ($1 Peter} {next}}

else {print}}

next Statement

•The next statement gets the next line of input from the input file, restarting execution at the top of the awk script

•Example:(In Script)1 {for ( x = 3; x <= NF; x++)

if ( $x < 0) {print “Bottomed out!”; break}# breaks out of the loop

}2 {for ( x = 3; x <= NF; x++ )

if ( $x == 0 ) { print “Get next item”; continue}# starts next iteration of the for loop

}

exit Statement

•The exit statement is used to terminate the awk program. It stops processing records, but does not skip over an END statement

•If the exit statement is given a value between 0 and 255 as an argument (exit 1), this value can be printed at the command line to indicate success or failure by typing:

exit Statement (continue.)

•Example:(In Script)

{exit (1)}(The Command Line)% echo $status (csh)

1$ echo $? (sh/ksh)

1

Arrays

•Arrays in awk are called associative arrays because

the subscripts can be either –number, or–string

•The keys and values are stored internally is a table where a hashing algorithm is applied to the value of the key in question•An array is created by using it, and awk can infer whether or not is used to store numbers or strings

Arrays (continue.)

•Array elements are initialized with–numeric value, and–string value null

•You do not have to declare the size of an aw array•awk arrays are used to collect information from records and may be used for accumulating totals, counting words, tracking the number of times a pattern occurred

Arrays (continue.)

•Example:$ cat employeesChen Cho 5/19/63 203-344-1234 76Tom Billy 4/12/45 913-972-4536 102Larry White 11/2/54 908-657-2389 54Bill Clinton 1/14/60 654-576-4114 201Steve Ann 9/15/71 202-545-8899 58$ nawk '{name[x++]=$2};END{for(i=0; i<NR;i++)\> print i, name[i]}' employees0 Cho1 Billy2 White3 Clinton4 Ann

Arrays (continue.)

•Example:

$ nawk '{id[NR]=$3};END{for(x = 1; x<= NR; x++)\> print id[x]}' employees5/19/634/12/4511/2/541/14/609/15/71$

The special for Loop

•The special for loop is used to read through an associative array when strings are used as subscripts or the subscripts are not consecutive numbers

•When strings are used as subscripts or the subscripts are not consecutive numbers

The special for Loop (continue.)

•Example:$ cat dbTom JonesMary AdamsSally ChangBilly BlackTom Savage$ nawk '/^Tom/{name[NR]=$1};\> END{for( i=1; i <= NR; i++ )print name[i]}' dbTom

Tom$

The special for Loop (continue.)

•Example:

$ nawk '/^Tom/{name[NR]=$1};\> END{for(i in name){print name[i]}}' dbTomTom$

Using Strings as Array Subscripts

•A subscript may consist of a variable containing a string or literal string

•If the string is a literal, it must be enclosed in double quotes

•Example$ cat dbTom JonesMary AdamsSally ChangBilly BlackTom Savage$ nawk -f awkscript dbThere are 2 Tom's in the file and 1 Mary's in the file.

Using Field Values as Array Subscripts (continue.)

•Any expression can be used as a subscript in an array. Therefore, fields can be used

•Example 1: $ cat dbTom JonesMary AdamsSally ChangBilly BlackTom Savage$ nawk '{count[$2]++}END{for(name in count)print name, count[name] }' dbChang 1Black 1Jones 1Savage 1Adams 1$

Using Field Values as Array Subscripts (continue.)

•Example 2: $ cat dbTom JonesMary AdamsSally ChangBilly BlackTom Savage$ nawk '{count[$2]++}END{for(name in count)print name, count[name] }' dbChang 1Black 1Jones 1Savage 1Adams 1$

Arrays and the split Function

•awk’s built-in split function allows you to split string into words and store them in an array

•You can define the field separator or use the value currently stored in FS

•Format:

split(string, array, field separator)split(string, array)

The delete Function

•The delete function removes an array elements

Multidimensional Arrays: Nawk

•Multidimensional array is done by concatenating the indices into a string separated by the value of a special built-in variable, SUBSEP

•The SUBSEP variable contains the value “\034”, an unprintable character

•The expression matrix[2,8] is really the array matrix[2 SUBSEP 8] which evaluates to matrix[“2\0348”]

ARGV

•Command line arguments are available to nawk with the built-in array called ARGV

•These arguments include the command nawk, but not any of the options passed to nawk

•The index of the ARGV array starts at zero

ARGC

•ARGC is a built-in variable that contains the number of command line arguments

•Example:

$ cat myscript#This script is called myscriptBEGIN{ for ( i = 0; i< ARGC; i++){ printf("argv[%d] is %s\n", i, ARGV[i]) } printf("The number of arguments, ARGC=%d\n", ARGC)}

ARGC (continue.)

•Example :

$ cat ARGVS#This script is called argvsBEGIN{ for ( i=0; i < ARGC; i++ ) { printf("argv[%d] is %s\n", i, ARGV[i]) }printf("The number of arguments, ARGC=%d\n", ARGC)}$$ nawk -f ARGVS datafileargv[0] is nawkargv[1] is datafileThe number of arguments, ARGC=2$

ARGC (continue.)

•Example :

$ nawk -f ARGVS datafile "Peter Pan" 12argv[0] is nawkargv[1] is datafileargv[2] is Peter Panargv[3] is 12The number of arguments, ARGC=4$

ARGC (continue.)

•Example :$ cat arging#This script is called arginBEGIN{FS=":"; name=ARGV[2] print "ARGV[2] is "ARGV[2]}$1 ~ name { print $0 }$ nawk -f arging employees2 "Chen Cho"ARGV[2] is Chen ChoChen Cho:5/19/63:203-344-1234:$76nawk: can't open file Chen Cho input record number 5, file Chen Cho source line number 1

ARGC (continue.)

•Example :$ cat arging1#This script is called arginBEGIN{FS=":"; name=ARGV[2] print "ARGV[2] is "ARGV[2]delete ARGV[2]}$1 ~ name { print $0 }$ nawk -f arging1 employees2 "Chen Cho"ARGV[2] is Chen ChoChen Cho:5/19/63:203-344-1234:$76

The sub and gsub Functions

•The sub function matches the regular expression for the largest and leftmost substring in the record, and replaces that substring with the substitution string

•If a target string is specified, the regular expression is matched for the largest and leftmost substring in the target string, and the substring is replaced with the substitution string

•If a target string is not specified, the entire record is used

The sub and gsub Functions (continue.)

•Format:sub (regular expression, substitution string);sub (regular string, substitution string,target string)

•Format:gsub (regular expression, substitution string)sub (regular string, substitution string,target string)

The index Function

•The index function returns the first position where a substring is found in a string

•Offset starts at position 1

The length Function

•The length function returns the number of characters in a string

•Without an argument, the length function returns the number of characters in a record

•Format:length (string)length

The length Function

•Example

$ nawk '{ print length("hello") }' employees55555$

The substr Function

•The substr function returns the substring of a string starting at a position where the first position is one

•If the length of the substring is given, that part of the string is returned

•If the specified length exceeds the actual string, the string is returned

•Format:substr (string, starting position)substr (string, starting position,

length of string

The match Function

•The match function returns the index where the regular expression is found in the string, or zero if not found

•The match function sets the built-in variable RSTART to the starting position of the substring within the string, and RLENGTH to the number of characters to the end of the substring

•Format:match (string, regular expression)

The match Function (continue.)

•Example :

$ nawk 'END{start=match("Good ole USA", /[A-Z]+$/);\> print RSTART, RLENGTH}' employees10 3

The split Function

•The split function splits a string into an array using whatever field separator is designated as the third parameter

•If the third parameter is not provided, awk will use the current value of FS

•Format:split (string, array, field separator)split (string, array)

The sprintf Function

•The sprintf function returns an expression in a specified format. It allows you to apply the format specifications of the printf function

•Format:Variable=sprintf(“string with format

specifiers” expr1, expr2, …, exprn)

The sprintf Function (continue.)

•Example :

$ awk '{line = sprintf ( "%-15s %6.2f ", $1 , $3 );\> print line}' employeesChen 0.00Tom 0.00Larry 0.00Bill 0.00Steve 0.00

Built-in Arithmetic Functions

Name Value returned

atan2(x,y) Arctangent of y/x in the range

cos(x) Cosine of x, with x in radians

exp(x) Exponential function of x, e

int(x) Integer part of x; truncated toward 0 when x>0

log(x) Natural (base e) logarithm of x

rand() Random number r, where 0<r<1

Built-in Arithmetic Functions (continue.)

Name Value returned

sin() Sine of x, with x radians

sqrt(x) Square root of x

srand(x) X is a new seed for rand()

Integer Function

•The int function truncates any digits to the right of the decimal point to create a whole number. There is no rounding off

The rand Function

•The rand function generates a pseudorandom floating point number greater than or equal to zero and less than one

•Example

$ nawk '{print rand()}' employees0.5138710.1757260.3086340.5345320.94763$ nawk '{print rand()}' employees0.5138710.1757260.3086340.5345320.94763

The srand Function

•The srand function without an argument uses the time of day to generate the seed for the rand function

•srand(x) uses x as the seed. Normally, x should vary during the run of the program

•Example $ nawk 'BEGIN{srand()};{print rand()}' employees0.5487530.3922540.9724720.8214970.153722$

The srand Function (continue.)

•Example $ nawk 'BEGIN{srand()};{print rand()}' employees0.5487530.3922540.9724720.8214970.153722$ nawk 'BEGIN{srand()};{print rand()}' employees0.005371260.3127840.237220.1320230.17304

User-Defined Functions (nawk)

•A user-defined function can be placed anywhere in the script that a pattern action rule can

•Formatfunction name (parameter, parameter, parameter, …){statements return expression(the return statement and expression are optional)}

User-Defined Functions (nawk) (continue.)

•Variables are passed by value and are local to the function where they are used•Arrays are passed by address or by references, so

array elements can be directly changed within the function•Any variable used within the function that has not been passed in the parameter list is considered a global variable that is, it is visible to the entire awk program, and if changed in the function, is changed throughout the program


•The only way to provide local variables within a function is to include them in the parameter list•If there is not a formal parameter provided in the function call, the parameter is initially set to null•The return statement returns control and possibly a value to the caller•Example

$ cat grades44 55 66 22 77 99100 22 77 99 33 6655 66 100 99 88 45$


•Example $ cat sorter#Script is called sorter# It sorts numbers in ascending orderfunction sort ( scores, num_elements, temp, i, j ) { # temp, i, and j will be local and private, #with an initila value of null. for( i = 2; i <= num_elements ; i++ )

{ for ( j = i; scores [j-1] > scores[j]; --j)

{ temp = scores[j] scores[j] = scores[j-1] scores[j-1] = temp } }}


•Example

•To view sorter script, click here

{for ( i = 1; i <= NF; i++) grades[i]=$isort(grades, NF)for (j = 1; j <= NF; j++ ) printf( "%d ", grades[j] )printf("\n")}$ nawk -f sorter grades22 44 55 66 77 9922 33 66 77 99 10045 55 66 88 99 100

The substr function

•In the following example, the fields are of fixed width, but are not separated by a field separator. The substr function is used to create field

Empty Fields

•If the data is stored in fixed-width fields, it is possible that some of the fields are empty. In the following example, the substr function is used to preserve the fields, whether or not they contain data

•Example$ cat filexxx xxxxxx abc xxxxxx a bbbxxx xx

Empty Fields (continue.)

$ cat awkfix#Preserving empty fields. Field width is fixed.{f[1]=substr($0,1,3)f[2]=substr($0,5,3)f[3]=substr($0,9,3)line=sprintf("%-4s%-4s%-4s\n", f[1], f[2], f[3])print line}$ nawk -f awkfix filexxx xxxxxx abc xxxxxx a bbbxxx xx

Empty Fields (continue.)

• To view awkfix script, click here

References UNIX SHELLS BY EXAMPLE BY ELLIE

QUIGLEY UNIX FOR PROGRAMMERS AND

USERS BY G. GLASS AND K ABLES UNIX SHELL PROGRAMMING BY S.

KOCHAN AND P. WOOD

what is awk awk is a programming language used for manipulating data and generating reports the data...

Documents