language programming - wikimedia...a wk language programming a user's guide for gnu a wk edition 1.0...

AWK Language Programming

A User's Guide for GNU AWKEdition 1.0

January 1996

Arnold D. Robbins

Based on The GAWK Manual,by Robbins, Close, Rubin, and Stallman

\To boldly go where no man has gone before" is a Registered Trademark ofParamount Pictures Corporation.

Copyright c 1989, 1991 - 1996 Free Software Foundation, Inc.

This is Edition 1.0 of AWK Language Programming,for the 3.0 (or later) version of the GNU implementation of AWK.

Published by the Free Software Foundation59 Temple Place | Suite 330Boston, MA 02111-1307 USAPhone: +1-617-542-5942Fax (including Japan): +1-617-542-2652Printed copies are available for $25 each.ISBN 1-882114-26-4

Permission is granted to make and distribute verbatim copies of this manualprovided the copyright notice and this permission notice are preserved on allcopies.

Permission is granted to copy and distribute modi�ed versions of this manualunder the conditions for verbatim copying, provided that the entire resultingderived work is distributed under the terms of a permission notice identicalto this one.

Permission is granted to copy and distribute translations of this manual intoanother language, under the above conditions for modi�ed versions, exceptthat this permission notice may be stated in a translation approved by theFoundation.

Cover art by Etienne Suvasa.

To Miriam, for making me complete.

To Chana, for the joy you bring us.

To Rivka, for the exponential increase.

Preface 1

Preface

This book teaches you about the awk language and how you can use ite�ectively. You should already be familiar with basic system commands,such as cat and ls,1 and basic shell facilities, such as Input/Output (I/O)redirection and pipes.

Implementations of the awk language are available for many di�erentcomputing environments. This book, while describing the awk language ingeneral, also describes a particular implementation of awk called gawk (whichstands for \GNU Awk"). gawk runs on a broad range of Unix systems,ranging from 80386 PC-based computers, up through large scale systems,such as Crays. gawk has also been ported to MS-DOS and OS/2 PC's, Atariand Amiga micro-computers, and VMS.

History of awk and gawk

The name awk comes from the initials of its designers: Alfred V. Aho,Peter J. Weinberger, and Brian W. Kernighan. The original version of awkwas written in 1977 at AT&T Bell Laboratories. In 1985 a new versionmade the programming language more powerful, introducing user-de�nedfunctions, multiple input streams, and computed regular expressions. Thisnew version became generally available with Unix System V Release 3.1. Theversion in System V Release 4 added some new features and also cleaned upthe behavior in some of the \dark corners" of the language. The speci�cationfor awk in the POSIX Command Language and Utilities standard furtherclari�ed the language based on feedback from both the gawk designers, andthe original Bell Labs awk designers.

The GNU implementation, gawk, was written in 1986 by Paul Rubin andJay Fenlason, with advice from Richard Stallman. John Woods contributedparts of the code as well. In 1988 and 1989, David Trueman, with help fromArnold Robbins, thoroughly reworked gawk for compatibility with the newerawk. Current development focuses on bug �xes, performance improvements,standards compliance, and occasionally, new features.

The GNU Project and This Book

The Free Software Foundation (FSF) is a non-pro�t organization ded-icated to the production and distribution of freely distributable software.

1 These commands are available on POSIX compliant systems, as well ason traditional Unix based systems. If you are using some other operatingsystem, you still need to be familiar with the ideas of I/O redirection andpipes

2 AWK Language Programming

It was founded by Richard M. Stallman, the author of the original Emacseditor. GNU Emacs is the most widely used version of Emacs today.

The GNU project is an on-going e�ort on the part of the Free SoftwareFoundation to create a complete, freely distributable, POSIX compliant com-puting environment. (GNU stands for \GNU's not Unix".) The FSF usesthe \GNU General Public License" (or GPL) to ensure that source codefor their software is always available to the end user. A copy of the GPLis included for your reference (see [GNU GENERAL PUBLIC LICENSE],page 311). The GPL applies to the C language source code for gawk.

As of this writing (1995), the only major component of the GNU environ-ment still uncompleted is the operating system kernel, and work proceedsapace on that. A shell, an editor (Emacs), highly portable optimizing C,C++, and Objective-C compilers, a symbolic debugger, and dozens of largeand small utilities (such as gawk), have all been completed and are freelyavailable.

Until the GNU operating system is released, the FSF recommends theuse of Linux, a freely distributable, Unix-like operating system for 80386and other systems. There are many books on Linux. One freely availableone is Linux Installation and Getting Started, by Matt Welsh. Many Linuxdistributions are available, often in computer stores or bundled on CD-ROMwith books about Linux. Also, the FSF provides a Linux distribution (\De-bian"); contact them for more information. See Section B.1.1 [Getting thegawk Distribution], page 279, for the FSF's contact information. (There aretwo other freely available, Unix-like operating systems for 80386 and othersystems, NetBSD and FreeBSD. Both are based on the 4.4-Lite BerkeleySoftware Distribution, and both use recent versions of gawk for their ver-sions of awk.)

This book you are reading now is actually free. The information in it isfreely available to anyone, the machine readable source code for the bookcomes with gawk, and anyone may take this book to a copying machine andmake as many copies of it as they like. (Take a moment to check the copyingpermissions on the Copyright page.)

If you paid money for this book, what you actually paid for was the book'snice printing and binding, and the publisher's associated costs to produce it.We have made an e�ort to keep these costs reasonable; most people wouldprefer a bound book to over 300 pages of photo-copied text that would thenhave to be held in a loose-leaf binder (not to mention the time and laborinvolved in doing the copying). The same is true of producing this bookfrom the machine readable source; the retail price is only slightly more thanthe cost per page of printing it on a laser printer.

This book itself has gone through several previous, preliminary editions.I started working on a preliminary draft of The GAWK Manual, by DianeClose, Paul Rubin, and Richard Stallman in the fall of 1988. It was around

Preface 3

90 pages long, and barely described the original, \old" version of awk. Af-ter substantial revision, the �rst version of the The GAWK Manual to bereleased was Edition 0.11 Beta in October of 1989. The manual then un-derwent more substantial revision for Edition 0.13 of December 1991. DavidTrueman, Pat Rankin, and Michal Jaegermann contributed sections of themanual for Edition 0.13. That edition was published by the FSF as a boundbook early in 1992. Since then there have been several minor revisions,notably Edition 0.14 of November 1992 that was published by the FSF inJanuary of 1993, and Edition 0.16 of August 1993.

Edition 1.0 of AWK Language Programming represents a signi�cant re-working of The GAWK Manual, with much additional material. The FSFand I agree that I am now the primary author. I also felt that it needed amore descriptive title.

AWK Language Programming will undoubtedly continue to evolve. Anelectronic version comes with the gawk distribution from the FSF. If you�nd an error in this book, please report it! See Section B.7 [Reporting Prob-lems and Bugs], page 292, for information on submitting problem reportselectronically, or write to me in care of the FSF.

Acknowledgements

I would like to acknowledge Richard M. Stallman, for his vision of abetter world, and for his courage in founding the FSF and starting the GNUproject.

The initial draft of The GAWK Manual had the following acknowledge-ments:

Many people need to be thanked for their assistance in produc-ing this manual. Jay Fenlason contributed many ideas and sam-ple programs. Richard Mlynarik and Robert Chassell gave helpfulcomments on drafts of this manual. The paper A SupplementalDocument for awk by John W. Pierce of the Chemistry Depart-ment at UC San Diego, pinpointed several issues relevant both toawk implementation and to this manual, that would otherwise haveescaped us.

The following people provided many helpful comments on Edition 0.13 ofThe GAWK Manual: Rick Adams, Michael Brennan, Rich Burridge, DianeClose, Christopher (\Topher") Eliot, Michael Lijewski, Pat Rankin, MiriamRobbins, and Michal Jaegermann.

The following people provided many helpful comments for Edition 1.0 ofAWK Language Programming : Karl Berry, Michael Brennan, Darrel Han-kerson, Michal Jaegermann, Michael Lijewski, and Miriam Robbins. Pat


Rankin, Michal Jaegermann, Darrel Hankerson and Scott Dei�k updatedtheir respective sections for Edition 1.0.

Robert J. Chassell provided much valuable advice on the use of Texinfo.He also deserves special thanks for convincing me not to title this book HowTo Gawk Politely. Karl Berry helped signi�cantly with the TEX part ofTexinfo.

David Trueman deserves special credit; he has done a yeoman job ofevolving gawk so that it performs well, and without bugs. Although he is nolonger involved with gawk, working with him on this project was a signi�cantpleasure.

Scott Dei�k, Darrel Hankerson, Kai Uwe Rommel, Pat Rankin, andMichal Jaegermann (in no particular order) are long time members of thegawk \crack portability team." Without their hard work and help, gawkwould not be nearly the �ne program it is today. It has been and continuesto be a pleasure working with this team of �ne people.

Je�rey Friedl provided invaluable help in tracking down a number of lastminute problems with regular expressions in gawk 3.0.

David and I would like to thank Brian Kernighan of Bell Labs for in-valuable assistance during the testing and debugging of gawk, and for helpin clarifying numerous points about the language. We could not have donenearly as good a job on either gawk or its documentation without his help.

I would like to thank Marshall and Elaine Hartholz of Seattle, and Dr.Bert and Rita Schreiber of Detroit for large amounts of quiet vacation timein their homes, which allowed me to make signi�cant progress on this bookand on gawk itself. Phil Hughes of SSC contributed in a very important wayby loaning me his laptop Linux system, not once, but twice, allowing me todo a lot of work while away from home.

Finally, I must thank my wonderful wife, Miriam, for her patience throughthe many versions of this project, for her proof-reading, and for sharing mewith the computer. I would like to thank my parents for their love, and forthe grace with which they raised and educated me. I also must acknowledgemy gratitude to G-d, for the many opportunities He has sent my way, aswell as for the gifts He has given me with which to take advantage of thoseopportunities.

Arnold RobbinsAtlanta, GeorgiaJanuary, 1996

Chapter 1: Introduction 5

1 Introduction

If you are like many computer users, you would frequently like to makechanges in various text �les wherever certain patterns appear, or extract datafrom parts of certain lines while discarding the rest. To write a program todo this in a language such as C or Pascal is a time-consuming inconveniencethat may take many lines of code. The job may be easier with awk.

The awk utility interprets a special-purpose programming language thatmakes it possible to handle simple data-reformatting jobs with just a fewlines of code.

The GNU implementation of awk is called gawk; it is fully upward com-patible with the System V Release 4 version of awk. gawk is also upwardcompatible with the POSIX speci�cation of the awk language. This meansthat all properly written awk programs should work with gawk. Thus, weusually don't distinguish between gawk and other awk implementations.

Using awk you can:

� manage small, personal databases

� generate reports

� validate data

� produce indexes, and perform other document preparation tasks

� even experiment with algorithms that can be adapted later to othercomputer languages

1.1 Using This Book

The term awk refers to a particular program, and to the language youuse to tell this program what to do. When we need to be careful, we call theprogram \the awk utility" and the language \the awk language." The termgawk refers to a version of awk developed as part the GNU project. Thepurpose of this book is to explain both the awk language and how to run theawk utility.

The main purpose of the book is to explain the features of awk, as de�nedin the POSIX standard. It does so in the context of one particular imple-mentation, gawk. While doing so, it will also attempt to describe importantdi�erences between gawk and other awk implementations. Finally, any gawkfeatures that are not in the POSIX standard for awk will be noted.

This book has the di�cult task of being both tutorial and reference. Ifyou are a novice, feel free to skip over details that seem too complex. Youshould also ignore the many cross references; they are for the expert user,and for the on-line Info version of the document.


The term awk program refers to a program written by you in the awkprogramming language.

See Chapter 2 [Getting Started with awk], page 9, for the bare essentialsyou need to know to start using awk.

Some useful \one-liners" are included to give you a feel for the awk lan-guage (see Chapter 3 [Useful One Line Programs], page 21).

Many sample awk programs have been provided for you (see Chapter 15[A Library of awk Functions], page 169; also see Chapter 16 [Practical awkPrograms], page 203).

The entire awk language is summarized for quick reference in Appendix A[gawk Summary], page 257. Look there if you just need to refresh yourmemory about a particular feature.

If you �nd terms that you aren't familiar with, try looking them up inthe glossary (see Appendix D [Glossary], page 303).

Most of the time complete awk programs are used as examples, but insome of the more advanced sections, only the part of the awk program thatillustrates the concept being described is shown.

While this book is aimed principally at people who have not been ex-posed to awk, there is a lot of information here that even the awk expertshould �nd useful. In particular, the description of POSIX awk, and theexample programs in Chapter 15 [A Library of awk Functions], page 169,and Chapter 16 [Practical awk Programs], page 203, should be of interest.

Dark Corners

Until the POSIX standard (and The Gawk Manual), many features ofawk were either poorly documented, or not documented at all. Descriptionsof such features (often called \dark corners") are noted in this book with\(d.c.)". They also appear in the index under the heading \dark corner."

1.2 Typographical Conventions

This book is written using Texinfo, the GNU documentation formattinglanguage. A single Texinfo source �le is used to produce both the printedand on-line versions of the documentation. Because of this, the typographicalconventions are slightly di�erent than in other books you may have read.

Examples you would type at the command line are preceded by the com-mon shell primary and secondary prompts, `$' and `>'. Output from thecommand is preceded by the glyph \a ". This typically represents the com-mand's standard output. Error messages, and other output on the com-mand's standard error, are preceded by the glyph \ error ". For example:

Chapter 1: Introduction 7

$ echo hi on stdout

a hi on stdout$ echo hello on stderr 1>&2

error hello on stderr

In the text, command names appear in this font, while code segmentsappear in the same font and quoted, `like this'. Some things will be em-phasized like this, and if a point needs to be made strongly, it will be donelike this. The �rst occurrence of a new term is usually its de�nition, andappears in the same font as the previous occurrence of \de�nition" in thissentence. File names are indicated like this: `/path/to/ourfile'.

Characters that you type at the keyboard look like this. In particular,there are special characters called \control characters." These are charactersthat you type by holding down both the CONTROL key and another key, atthe same time. For example, a Control-d is typed by �rst pressing andholding the CONTROL key, next pressing the d key, and �nally releasing bothkeys.

1.3 Data Files for the Examples

Many of the examples in this book take their input from two sample data�les. The �rst, called `BBS-list', represents a list of computer bulletin boardsystems together with information about those systems. The second data�le, called `inventory-shipped', contains information about shipments ona monthly basis. In both �les, each line is considered to be one record.

In the �le `BBS-list', each record contains the name of a computer bul-letin board, its phone number, the board's baud rate(s), and a code for thenumber of hours it is operational. An `A' in the last column means the boardoperates 24 hours a day. A `B' in the last column means the board operatesevening and weekend hours, only. A `C' means the board operates only onweekends.

aardvark 555-5553 1200/300 B

alpo-net 555-3412 2400/1200/300 A

barfly 555-7685 1200/300 A

bites 555-1675 2400/1200/300 A

camelot 555-0542 300 C

core 555-2912 1200/300 C

fooey 555-1234 2400/1200/300 B

foot 555-6699 1200/300 B

macfoo 555-6480 1200/300 A

sdace 555-3430 2400/1200/300 A

sabafoo 555-2127 1200/300 C


The second data �le, called `inventory-shipped', represents informa-tion about shipments during the year. Each record contains the monthof the year, the number of green crates shipped, the number of red boxesshipped, the number of orange bags shipped, and the number of blue pack-ages shipped, respectively. There are 16 entries, covering the 12 months ofone year and four months of the next year.

Jan 13 25 15 115

Feb 15 32 24 226

Mar 15 24 34 228

Apr 31 52 63 420

May 16 34 29 208

Jun 31 42 75 492

Jul 24 34 67 436

Aug 15 34 47 316

Sep 13 55 37 277

Oct 29 54 68 525

Nov 20 87 82 577

Dec 17 35 61 401

Jan 21 36 64 620

Feb 26 58 80 652

Mar 24 75 70 495

Apr 21 70 74 514

Chapter 2: Getting Started with awk 9

2 Getting Started with awk

The basic function of awk is to search �les for lines (or other units of text)that contain certain patterns. When a line matches one of the patterns, awkperforms speci�ed actions on that line. awk keeps processing input lines inthis way until the end of the input �les are reached.

Programs in awk are di�erent from programs in most other languages,because awk programs are data-driven; that is, you describe the data youwish to work with, and then what to do when you �nd it. Most otherlanguages are procedural; you have to describe, in great detail, every stepthe program is to take. When working with procedural languages, it isusually much harder to clearly describe the data your program will process.For this reason, awk programs are often refreshingly easy to both write andread.

When you run awk, you specify an awk program that tells awk what todo. The program consists of a series of rules. (It may also contain functionde�nitions, an advanced feature which we will ignore for now. See Chapter 13[User-de�ned Functions], page 153.) Each rule speci�es one pattern to searchfor, and one action to perform when that pattern is found.

Syntactically, a rule consists of a pattern followed by an action. Theaction is enclosed in curly braces to separate it from the pattern. Rules areusually separated by newlines. Therefore, an awk program looks like this:

pattern { action }pattern { action }: : :

2.1 A Rose By Any Other Name

The awk language has evolved over the years. Full details are provided inChapter 17 [The Evolution of the awk Language], page 251. The languagedescribed in this book is often referred to as \new awk."

Because of this, many systems have multiple versions of awk. Some sys-tems have an awk utility that implements the original version of the awklanguage, and a nawk utility for the new version. Others have an oawk forthe \old awk" language, and plain awk for the new one. Still others onlyhave one version, usually the new one.1

All in all, this makes it di�cult for you to know which version of awk youshould run when writing your programs. The best advice we can give hereis to check your local documentation. Look for awk, oawk, and nawk, as wellas for gawk. Chances are, you will have some version of new awk on your

1 Often, these systems use gawk for their awk implementation!


system, and that is what you should use when running your programs. (Ofcourse, if you're reading this book, chances are good that you have gawk!)

Throughout this book, whenever we refer to a language feature thatshould be available in any complete implementation of POSIX awk, we sim-ply use the term awk. When referring to a feature that is speci�c to theGNU implementation, we use the term gawk.

2.2 How to Run awk Programs

There are several ways to run an awk program. If the program is short,it is easiest to include it in the command that runs awk, like this:

awk 'program' input-�le1 input-�le2 : : :

where program consists of a series of patterns and actions, as describedearlier. (The reason for the single quotes is described below, in Section 2.2.1[One-shot Throw-away awk Programs], page 10.)

When the program is long, it is usually more convenient to put it in a�le and run it with a command like this:

awk -f program-�le input-�le1 input-�le2 : : :

2.2.1 One-shot Throw-away awk Programs

Once you are familiar with awk, you will often type in simple programsthe moment you want to use them. Then you can write the program as the�rst argument of the awk command, like this:

awk 'program' input-�le1 input-�le2 : : :

where program consists of a series of patterns and actions, as describedearlier.

This command format instructs the shell, or command interpreter, tostart awk and use the program to process records in the input �le(s). Thereare single quotes around program so that the shell doesn't interpret any awkcharacters as special shell characters. They also cause the shell to treat allof program as a single argument for awk and allow program to be more thanone line long.

This format is also useful for running short or medium-sized awk programsfrom shell scripts, because it avoids the need for a separate �le for the awkprogram. A self-contained shell script is more reliable since there are noother �les to misplace.

Chapter 3 [Useful One Line Programs], page 21, presents several short,self-contained programs.


As an interesting side point, the command

awk '/foo/' �les : : :

is essentially the same as

egrep foo �les : : :

2.2.2 Running awk without Input Files

You can also run awk without any input �les. If you type the commandline:

awk 'program'

then awk applies the program to the standard input, which usually meanswhatever you type on the terminal. This continues until you indicate end-of-�le by typing Control-d. (On other operating systems, the end-of-�lecharacter may be di�erent. For example, on OS/2 and MS-DOS, it isControl-z.)

For example, the following program prints a friendly piece of advice (fromDouglas Adams' The Hitchhiker's Guide to the Galaxy), to keep you fromworrying about the complexities of computer programming (`BEGIN' is afeature we haven't discussed yet).

$ awk "BEGIN { print \"Don't Panic!\" }"

a Don't Panic!

This program does not read any input. The `\' before each of the innerdouble quotes is necessary because of the shell's quoting rules, in particularbecause it mixes both single quotes and double quotes.

This next simple awk program emulates the cat utility; it copies what-ever you type at the keyboard to its standard output. (Why this works isexplained shortly.)

$ awk '{ print }'

Now is the time for all good men

a Now is the time for all good mento come to the aid of their country.

a to come to the aid of their country.Four score and seven years ago, ...

a Four score and seven years ago, ...What, me worry?

a What, me worry?Control-d


2.2.3 Running Long Programs

Sometimes your awk programs can be very long. In this case it is moreconvenient to put the program into a separate �le. To tell awk to use that�le for its program, you type:

awk -f source-�le input-�le1 input-�le2 : : :

The `-f' instructs the awk utility to get the awk program from the �lesource-�le. Any �le name can be used for source-�le. For example, you couldput the program:

BEGIN { print "Don't Panic!" }

into the �le àdvice'. Then this command:

awk -f advice

does the same thing as this one:

awk "BEGIN { print \"Don't Panic!\" }"

which was explained earlier (see Section 2.2.2 [Running awk without InputFiles], page 11). Note that you don't usually need single quotes aroundthe �le name that you specify with `-f', because most �le names don'tcontain any of the shell's special characters. Notice that in àdvice', theawk program did not have single quotes around it. The quotes are onlyneeded for programs that are provided on the awk command line.

If you want to identify your awk program �les clearly as such, you canadd the extension `.awk' to the �le name. This doesn't a�ect the executionof the awk program, but it does make \housekeeping" easier.

2.2.4 Executable awk Programs

Once you have learned awk, you may want to write self-contained awkscripts, using the `#!' script mechanism. You can do this on many Unixsystems1 (and someday on the GNU system).

For example, you could update the �le àdvice' to look like this:

#! /bin/awk -f


After making this �le executable (with the chmod utility), you can simplytype àdvice' at the shell, and the system will arrange to run awk2 as if youhad typed àwk -f advice'.

1 The `#!' mechanism works on Linux systems, Unix systems derived fromBerkeley Unix, System V Release 4, and some System V Release 3systems.

2 The line beginning with `#!' lists the full �le name of an interpreter tobe run, and an optional initial command line argument to pass to that


$ advice

a Don't Panic!

Self-contained awk scripts are useful when you want to write a program whichusers can invoke without their having to know that the program is writtenin awk.

Some older systems do not support the `#!' mechanism. You can get asimilar e�ect using a regular shell script. It would look something like this:

: The colon ensures execution by the standard shell.

awk 'program' "$@"

Using this technique, it is vital to enclose the program in single quotesto protect it from interpretation by the shell. If you omit the quotes, only ashell wizard can predict the results.

The "$@" causes the shell to forward all the command line arguments tothe awk program, without interpretation. The �rst line, which starts witha colon, is used so that this shell script will work even if invoked by a userwho uses the C shell. (Not all older systems obey this convention, but manydo.)

2.2.5 Comments in awk Programs

A comment is some text that is included in a program for the sake ofhuman readers; it is not really part of the program. Comments can explainwhat the program does, and how it works. Nearly all programming lan-guages have provisions for comments, because programs are typically hardto understand without their extra help.

In the awk language, a comment starts with the sharp sign character, `#',and continues to the end of the line. The `#' does not have to be the �rstcharacter on the line. The awk language ignores the rest of a line followinga sharp sign. For example, we could have put the following into `advice':

# This program prints a nice friendly message. It helps

# keep novice users from being afraid of the computer.


You can put comment lines into keyboard-composed throw-away awk pro-grams also, but this usually isn't very useful; the purpose of a comment isto help you or another person understand the program at a later time.

interpreter. The operating system then runs the interpreter with thegiven argument and the full argument list of the executed program. The�rst argument in the list is the full �le name of the awk program. Therest of the argument list will either be options to awk, or data �les, orboth.


2.3 A Very Simple Example

The following command runs a simple awk program that searches the in-put �le `BBS-list' for the string of characters: `foo'. (A string of charactersis usually called a string. The term string is perhaps based on similar usagein English, such as \a string of pearls," or, \a string of cars in a train.")

awk '/foo/ { print $0 }' BBS-list

When lines containing `foo' are found, they are printed, because `print $0'means print the current line. (Just `print' by itself means the same thing,so we could have written that instead.)

You will notice that slashes, `/', surround the string `foo' in the awkprogram. The slashes indicate that `foo' is a pattern to search for. Thistype of pattern is called a regular expression, and is covered in more detaillater (see Chapter 4 [Regular Expressions], page 23). The pattern is allowedto match parts of words. There are single-quotes around the awk programso that the shell won't interpret any of it as special shell characters.

Here is what this program prints:

$ awk '/foo/ { print $0 }' BBS-list

a fooey 555-1234 2400/1200/300 B

a foot 555-6699 1200/300 B

a macfoo 555-6480 1200/300 A

a sabafoo 555-2127 1200/300 C

In an awk rule, either the pattern or the action can be omitted, but notboth. If the pattern is omitted, then the action is performed for every inputline. If the action is omitted, the default action is to print all lines thatmatch the pattern.

Thus, we could leave out the action (the print statement and the curlybraces) in the above example, and the result would be the same: all linesmatching the pattern `foo' would be printed. By comparison, omitting theprint statement but retaining the curly braces makes an empty action thatdoes nothing; then no lines would be printed.

2.4 An Example with Two Rules

The awk utility reads the input �les one line at a time. For each line, awktries the patterns of each of the rules. If several patterns match then severalactions are run, in the order in which they appear in the awk program. If nopatterns match, then no actions are run.

After processing all the rules (perhaps none) that match the line, awkreads the next line (however, see Section 9.7 [The next Statement], page 111,and also see Section 9.8 [The nextfile Statement], page 112). This contin-ues until the end of the �le is reached.


For example, the awk program:

/12/ { print $0 }

/21/ { print $0 }

contains two rules. The �rst rule has the string `12' as the pattern and`print $0' as the action. The second rule has the string `21' as the patternand also has `print $0' as the action. Each rule's action is enclosed in itsown pair of braces.

This awk program prints every line that contains the string `12' or thestring `21'. If a line contains both strings, it is printed twice, once by eachrule.

This is what happens if we run this program on our two sample data �les,`BBS-list' and `inventory-shipped', as shown here:

$ awk '/12/ { print $0 }

> /21/ { print $0 }' BBS-list inventory-shipped

a aardvark 555-5553 1200/300 B

a alpo-net 555-3412 2400/1200/300 A

a barfly 555-7685 1200/300 A

a bites 555-1675 2400/1200/300 A

a core 555-2912 1200/300 C

a fooey 555-1234 2400/1200/300 B

a foot 555-6699 1200/300 B

a macfoo 555-6480 1200/300 A

a sdace 555-3430 2400/1200/300 Aa sabafoo 555-2127 1200/300 C

a sabafoo 555-2127 1200/300 C

a Jan 21 36 64 620

a Apr 21 70 74 514

Note how the line in `BBS-list' beginning with `sabafoo' was printed twice,once for each rule.

2.5 AMore Complex Example

Here is an example to give you an idea of what typical awk programsdo. This example shows how awk can be used to summarize, select, andrearrange the output of another utility. It uses features that haven't beencovered yet, so don't worry if you don't understand all the details.

ls -lg | awk '$6 == "Nov" { sum += $5 }

END { print sum }'

This command prints the total number of bytes in all the �les in thecurrent directory that were last modi�ed in November (of any year). (Inthe C shell you would need to type a semicolon and then a backslash at the


end of the �rst line; in a POSIX-compliant shell, such as the Bourne shell orBash, the GNU Bourne-Again shell, you can type the example as shown.)

The `ls -lg' part of this example is a system command that gives you alisting of the �les in a directory, including �le size and the date the �le waslast modi�ed. Its output looks like this:

-rw-r--r-- 1 arnold user 1933 Nov 7 13:05 Makefile

-rw-r--r-- 1 arnold user 10809 Nov 7 13:03 gawk.h

-rw-r--r-- 1 arnold user 983 Apr 13 12:14 gawk.tab.h

-rw-r--r-- 1 arnold user 31869 Jun 15 12:20 gawk.y

-rw-r--r-- 1 arnold user 22414 Nov 7 13:03 gawk1.c


-rw-r--r-- 1 arnold user 27511 Dec 9 13:07 gawk3.c


The �rst �eld contains read-write permissions, the second �eld contains thenumber of links to the �le, and the third �eld identi�es the owner of the�le. The fourth �eld identi�es the group of the �le. The �fth �eld containsthe size of the �le in bytes. The sixth, seventh and eighth �elds contain themonth, day, and time, respectively, that the �le was last modi�ed. Finally,the ninth �eld contains the name of the �le.

The `$6 == "Nov"' in our awk program is an expression that tests whetherthe sixth �eld of the output from `ls -lg' matches the string `Nov'. Eachtime a line has the string `Nov' for its sixth �eld, the action `sum += $5' isperformed. This adds the �fth �eld (the �le size) to the variable sum. As aresult, when awk has �nished reading all the input lines, sum is the sum ofthe sizes of �les whose lines matched the pattern. (This works because awkvariables are automatically initialized to zero.)

After the last line of output from ls has been processed, the END rule isexecuted, and the value of sum is printed. In this example, the value of sumwould be 80600.

These more advanced awk techniques are covered in later sections (seeSection 8.2 [Overview of Actions], page 102). Before you can move on tomore advanced awk programming, you have to know how awk interpretsyour input and displays your output. By manipulating �elds and usingprint statements, you can produce some very useful and impressive lookingreports.

2.6 awk Statements Versus Lines

Most often, each line in an awk program is a separate statement or sepa-rate rule, like this:

awk '/12/ { print $0 }

/21/ { print $0 }' BBS-list inventory-shipped


However, gawk will ignore newlines after any of the following:

, { ? : || && do else

A newline at any other point is considered the end of the statement. (Split-ting lines after `?' and `:' is a minor gawk extension. The `?' and `:' referredto here is the three operand conditional expression described in Section 7.12[Conditional Expressions], page 93.)

If you would like to split a single statement into two lines at a point wherea newline would terminate it, you can continue it by ending the �rst linewith a backslash character, `\'. The backslash must be the �nal characteron the line to be recognized as a continuation character. This is allowedabsolutely anywhere in the statement, even in the middle of a string orregular expression. For example:

awk '/This regular expression is too long, so continue it\

on the next line/ { print $1 }'

We have generally not used backslash continuation in the sample programsin this book. Since in gawk there is no limit on the length of a line, itis never strictly necessary; it just makes programs more readable. For thissame reason, as well as for clarity, we have kept most statements short in thesample programs presented throughout the book. Backslash continuation ismost useful when your awk program is in a separate source �le, insteadof typed in on the command line. You should also note that many awkimplementations are more particular about where you may use backslashcontinuation. For example, they may not allow you to split a string constantusing backslash continuation. Thus, for maximal portability of your awkprograms, it is best not to split your lines in the middle of a regular expressionor a string.

Caution: backslash continuation does not work as described above withthe C shell. Continuation with backslash works for awk programs in �les,and also for one-shot programs provided you are using a POSIX-compliantshell, such as the Bourne shell or Bash, the GNU Bourne-Again shell. Butthe C shell (csh) behaves di�erently! There, you must use two backslashesin a row, followed by a newline. Note also that when using the C shell,every newline in your awk program must be escaped with a backslash. Toillustrate:

% awk 'BEGIN { \

? print \\

? "hello, world" \

? }'

a hello, world

Here, the `%' and `?' are the C shell's primary and secondary prompts, anal-ogous to the standard shell's `$' and `>'.


awk is a line-oriented language. Each rule's action has to begin on thesame line as the pattern. To have the pattern and action on separate lines,you must use backslash continuation|there is no other way.

When awk statements within one rule are short, you might want to putmore than one of them on a line. You do this by separating the statementswith a semicolon, `;'.

This also applies to the rules themselves. Thus, the previous programcould have been written:

/12/ { print $0 } ; /21/ { print $0 }

Note: the requirement that rules on the same line must be separated with asemicolon was not in the original awk language; it was added for consistencywith the treatment of statements within an action.

2.7 Other Features of awk

The awk language provides a number of prede�ned, or built-in variables,which your programs can use to get information from awk. There are othervariables your program can set to control how awk processes your data.

In addition, awk provides a number of built-in functions for doing commoncomputational and string related operations.

As we develop our presentation of the awk language, we introduce mostof the variables and many of the functions. They are de�ned systemati-cally in Chapter 10 [Built-in Variables], page 115, and Chapter 12 [Built-inFunctions], page 135.

2.8 When to Use awk

You might wonder how awk might be useful for you. Using utility pro-grams, advanced patterns, �eld separators, arithmetic statements, and otherselection criteria, you can produce much more complex output. The awk lan-guage is very useful for producing reports from large amounts of raw data,such as summarizing information from the output of other utility programslike ls. (See Section 2.5 [A More Complex Example], page 15.)

Programs written with awk are usually much smaller than they would bein other languages. This makes awk programs easy to compose and use. Of-ten, awk programs can be quickly composed at your terminal, used once, andthrown away. Since awk programs are interpreted, you can avoid the (usu-ally lengthy) compilation part of the typical edit-compile-test-debug cycle ofsoftware development.

Complex programs have been written in awk, including a complete retar-getable assembler for eight-bit microprocessors (see Appendix D [Glossary],


page 303, for more information) and a microcode assembler for a specialpurpose Prolog computer. However, awk's capabilities are strained by tasksof such complexity.

If you �nd yourself writing awk scripts of more than, say, a few hundredlines, you might consider using a di�erent programming language. EmacsLisp is a good choice if you need sophisticated string or pattern matching ca-pabilities. The shell is also good at string and pattern matching; in addition,it allows powerful use of the system utilities. More conventional languages,such as C, C++, and Lisp, o�er better facilities for system programming andfor managing the complexity of large programs. Programs in these languagesmay require more lines of source code than the equivalent awk programs, butthey are easier to maintain and usually run more e�ciently.

Chapter 3: Useful One Line Programs 21

3 Useful One Line Programs

Many useful awk programs are short, just a line or two. Here is a collectionof useful, short programs to get you started. Some of these programs containconstructs that haven't been covered yet. The description of the programwill give you a good idea of what is going on, but please read the rest of thebook to become an awk expert!

Most of the examples use a data �le named `data'. This is just a place-holder; if you were to use these programs yourself, you would substitute yourown �le names for `data'.

awk '{ if (length($0) > max) max = length($0) }

END { print max }' data

This program prints the length of the longest input line.

awk 'length($0) > 80' data

This program prints every line that is longer than 80 characters.The sole rule has a relational expression as its pattern, and hasno action (so the default action, printing the record, is used).

expand data | awk '{ if (x < length()) x = length() }

END { print "maximum line length is " x }'

This program prints the length of the longest line in `data'. Theinput is processed by the expand program to change tabs intospaces, so the widths compared are actually the right-margincolumns.

awk 'NF > 0' data

This program prints every line that has at least one �eld. Thisis an easy way to delete blank lines from a �le (or rather, tocreate a new �le similar to the old �le but from which the blanklines have been deleted).

awk 'BEGIN { for (i = 1; i


awk 'NR % 2' data

This program prints the even numbered lines in the data �le. Ifyou were to use the expression `NR % 2 == 1' instead, it wouldprint the odd number lines.

Chapter 4: Regular Expressions 23

4 Regular Expressions

A regular expression, or regexp, is a way of describing a set of strings. Be-cause regular expressions are such a fundamental part of awk programming,their format and use deserve a separate chapter.

A regular expression enclosed in slashes (`/') is an awk pattern thatmatches every input record whose text belongs to that set.

The simplest regular expression is a sequence of letters, numbers, or both.Such a regexp matches any string that contains that sequence. Thus, theregexp `foo' matches any string containing `foo'. Therefore, the pattern/foo/ matches any input record containing the three characters `foo', any-where in the record. Other kinds of regexps let you specify more complicatedclasses of strings.

Initially, the examples will be simple. As we explain more about howregular expressions work, we will present more complicated examples.

4.1 How to Use Regular Expressions

A regular expression can be used as a pattern by enclosing it in slashes.Then the regular expression is tested against the entire text of each record.(Normally, it only needs to match some part of the text in order to succeed.)For example, this prints the second �eld of each record that contains thethree characters `foo' anywhere in it:

$ awk '/foo/ { print $2 }' BBS-list

a 555-1234

a 555-6699

a 555-6480

a 555-2127

Regular expressions can also be used in matching expressions. Theseexpressions allow you to specify the string to match against; it need not bethe entire current input record. The two operators, `~' and `!~', performregular expression comparisons. Expressions using these operators can beused as patterns or in if, while, for, and do statements.

exp ~ /regexp/This is true if the expression exp (taken as a string) is matchedby regexp. The following example matches, or selects, all inputrecords with the upper-case letter `J' somewhere in the �rst �eld:

$ awk '$1 ~ /J/' inventory-shipped

a Jan 13 25 15 115

a Jun 31 42 75 492

a Jul 24 34 67 436

a Jan 21 36 64 620


So does this:

awk '{ if ($1 ~ /J/) print }' inventory-shipped

exp !~ /regexp/This is true if the expression exp (taken as a character string)is not matched by regexp. The following example matches, orselects, all input records whose �rst �eld does not contain theupper-case letter `J':

$ awk '$1 !~ /J/' inventory-shipped

a Feb 15 32 24 226

a Mar 15 24 34 228

a Apr 31 52 63 420

a May 16 34 29 208: : :

When a regexp is written enclosed in slashes, like /foo/, we call it aregexp constant, much like 5.27 is a numeric constant, and "foo" is a stringconstant.

4.2 Escape Sequences

Some characters cannot be included literally in string constants ("foo")or regexp constants (/foo/). You represent them instead with escape se-quences, which are character sequences beginning with a backslash (`\').

One use of an escape sequence is to include a double-quote character in astring constant. Since a plain double-quote would end the string, you mustuse `\"' to represent an actual double-quote character as a part of the string.For example:

$ awk 'BEGIN { print "He said \"hi!\" to her." }'

a He said "hi!" to her.

The backslash character itself is another character that cannot be in-cluded normally; you write `\\' to put one backslash in the string or regexp.Thus, the string whose contents are the two characters `"' and `\' must bewritten "\"\\".

Another use of backslash is to represent unprintable characters such astab or newline. While there is nothing to stop you from entering mostunprintable characters directly in a string constant or regexp constant, theymay look ugly.

Here is a table of all the escape sequences used in awk, and what theyrepresent. Unless noted otherwise, all of these escape sequences apply toboth string constants and regexp constants.


\\ A literal backslash, `\'.

\a The \alert" character, Control-g, ASCII code 7 (BEL).

\b Backspace, Control-h, ASCII code 8 (BS).

\f Formfeed, Control-l, ASCII code 12 (FF).

\n Newline, Control-j, ASCII code 10 (LF).

\r Carriage return, Control-m, ASCII code 13 (CR).

\t Horizontal tab, Control-i, ASCII code 9 (HT).

\v Vertical tab, Control-k, ASCII code 11 (VT).

\nnn The octal value nnn, where nnn are one to three digits between`0' and `7'. For example, the code for the ASCII ESC (escape)character is `\033'.

\xhh: : : The hexadecimal value hh, where hh are hexadecimal digits (`0'through `9' and either `A' through `F' or `a' through `f'). Likethe same construct in ANSI C, the escape sequence continuesuntil the �rst non-hexadecimal digit is seen. However, usingmore than two hexadecimal digits produces unde�ned results.(The `\x' escape sequence is not allowed in POSIX awk.)

\/ A literal slash (necessary for regexp constants only). You usethis when you wish to write a regexp constant that containsa slash. Since the regexp is delimited by slashes, you need toescape the slash that is part of the pattern, in order to tell awkto keep processing the rest of the regexp.

\" A literal double-quote (necessary for string constants only). Youuse this when you wish to write a string constant that containsa double-quote. Since the string is delimited by double-quotes,you need to escape the quote that is part of the string, in orderto tell awk to keep processing the rest of the string.

In gawk, there are additional two character sequences that begin withbackslash that have special meaning in regexps. See Section 4.4 [AdditionalRegexp Operators Only in gawk], page 31.

In a string constant, what happens if you place a backslash before some-thing that is not one of the characters listed above? POSIX awk purposelyleaves this case unde�ned. There are two choices.

� Strip the backslash out. This is what Unix awk and gawk both do. Forexample, "a\qc" is the same as "aqc".

� Leave the backslash alone. Some other awk implementations do this. Insuch implementations, "a\qc" is the same as if you had typed "a\\qc".


In a regexp, a backslash before any character that is not in the abovetable, and not listed in Section 4.4 [Additional Regexp Operators Only ingawk], page 31, means that the next character should be taken literally, evenif it would normally be a regexp operator. E.g., /a\+b/ matches the threecharacters `a+b'.

For complete portability, do not use a backslash before any character notlisted in the table above.

Another interesting question arises. Suppose you use an octal or hexadec-imal escape to represent a regexp metacharacter (see Section 4.3 [RegularExpression Operators], page 26). Does awk treat the character as literalcharacter, or as a regexp operator?

It turns out that historically, such characters were taken literally (d.c.).However, the POSIX standard indicates that they should be treated as realmetacharacters, and this is what gawk does. However, in compatibility mode(see Section 14.1 [Command Line Options], page 161), gawk treats the char-acters represented by octal and hexadecimal escape sequences literally whenused in regexp constants. Thus, /a\52b/ is equivalent to /a\*b/.

To summarize:

1. The escape sequences in the table above are always processed �rst, forboth string constants and regexp constants. This happens very early,as soon as awk reads your program.

2. gawk processes both regexp constants and dynamic regexps (see Sec-tion 4.7 [Using Dynamic Regexps], page 35), for the special opera-tors listed in Section 4.4 [Additional Regexp Operators Only in gawk],page 31.

3. A backslash before any other character means to treat that characterliterally.

4.3 Regular Expression Operators

You can combine regular expressions with the following characters, calledregular expression operators, or metacharacters, to increase the power andversatility of regular expressions.

The escape sequences described above in Section 4.2 [Escape Sequences],page 24, are valid inside a regexp. They are introduced by a `\'. They arerecognized and converted into the corresponding real characters as the very�rst step in processing regexps.

Here is a table of metacharacters. All characters that are not escapesequences and that are not listed in the table stand for themselves.


\ This is used to suppress the special meaning of a character whenmatching. For example:

\$

matches the character `$'.

^ This matches the beginning of a string. For example:

^@chapter

matches the `@chapter' at the beginning of a string, and canbe used to identify chapter beginnings in Texinfo source �les.The `^' is known as an anchor, since it anchors the pattern tomatching only at the beginning of the string.

It is important to realize that `^' does not match the beginningof a line embedded in a string. In this example the condition isnot true:

if ("line1\nLINE 2" ~ /^L/) : : :

$ This is similar to `^', but it matches only at the end of a string.For example:

p$

matches a record that ends with a `p'. The `$' is also an anchor,and also does not match the end of a line embedded in a string.In this example the condition is not true:

if ("line1\nLINE 2" ~ /1$/) : : :

. The period, or dot, matches any single character, including thenewline character. For example:

.P

matches any single character followed by a `P' in a string. Usingconcatenation we can make a regular expression like Ù.A', whichmatches any three-character sequence that begins with Ù' andends with À'.

In strict POSIX mode (see Section 14.1 [Command Line Op-tions], page 161), `.' does not match the nul character, whichis a character with all bits equal to zero. Otherwise, nul is justanother character. Other versions of awk may not be able tomatch the nul character.

[: : :] This is called a character list. It matches any one of the char-acters that are enclosed in the square brackets. For example:

[MVX]

matches any one of the characters `M', `V', or `X' in a string.

Ranges of characters are indicated by using a hyphen betweenthe beginning and ending characters, and enclosing the wholething in brackets. For example:


[0-9]

matches any digit. Multiple ranges are allowed. E.g., the list[A-Za-z0-9] is a common way to express the idea of \all al-phanumeric characters."

To include one of the characters `\', `]', `-' or `^' in a characterlist, put a `\' in front of it. For example:

[d\]]

matches either `d', or `]'.

This treatment of `\' in character lists is compatible with otherawk implementations, and is also mandated by POSIX. The reg-ular expressions in awk are a superset of the POSIX speci�ca-tion for Extended Regular Expressions (EREs). POSIX EREsare based on the regular expressions accepted by the traditionalegrep utility.

Character classes are a new feature introduced in the POSIXstandard. A character class is a special notation for describinglists of characters that have a speci�c attribute, but where theactual characters themselves can vary from country to countryand/or from character set to character set. For example, thenotion of what is an alphabetic character di�ers in the USA andin France.

A character class is only valid in a regexp inside the brackets ofa character list. Character classes consist of `[:', a keyword de-noting the class, and `:]'. Here are the character classes de�nedby the POSIX standard.

[:alnum:]

Alphanumeric characters.

[:alpha:]

Alphabetic characters.

[:blank:]

Space and tab characters.

[:cntrl:]

Control characters.

[:digit:]

Numeric characters.

[:graph:]

Characters that are printable and are also visible.(A space is printable, but not visible, while an `a' isboth.)


[:lower:]

Lower-case alphabetic characters.

[:print:]

Printable characters (characters that are not controlcharacters.)

[:punct:]

Punctuation characters (characters that are not let-ter, digits, control characters, or space characters).

[:space:]

Space characters (such as space, tab, and formfeed,to name a few).

[:upper:]

Upper-case alphabetic characters.

[:xdigit:]

Characters that are hexadecimal digits.

For example, before the POSIX standard, to match alphanu-meric characters, you had to write /[A-Za-z0-9]/. If your char-acter set had other alphabetic characters in it, this would notmatch them. With the POSIX character classes, you can write/[[:alnum:]]/, and this will match all the alphabetic and nu-meric characters in your character set.

Two additional special sequences can appear in character lists.These apply to non-ASCII character sets, which can have singlesymbols (called collating elements) that are represented withmore than one character, as well as several characters that areequivalent for collating, or sorting, purposes. (E.g., in French, aplain \e" and a grave-accented \�e" are equivalent.)

Collating SymbolsA collating symbol is a multi-character collating el-ement enclosed in `[.' and `.]'. For example, if `ch'is a collating element, then [[.ch.]] is a regexpthat matches this collating element, while [ch] is aregexp that matches either `c' or `h'.

Equivalence ClassesAn equivalence class is a list of equivalent charactersenclosed in `[=' and `=]'. Thus, [[=e�e=]] is regexpthat matches either `e' or `�e'.

These features are very valuable in non-English speaking locales.

Caution: The library functions that gawk uses for regular expres-sion matching currently only recognize POSIX character classes;they do not recognize collating symbols or equivalence classes.


[^ : : :] This is a complemented character list. The �rst character afterthe `[' must be a `^'. It matches any characters except those inthe square brackets, or newline. For example:

[^0-9]

matches any character that is not a digit.

| This is the alternation operator, and it is used to specify alter-natives. For example:

^P|[0-9]

matches any string that matches either `^P' or `[0-9]'. Thismeans it matches any string that starts with `P' or contains adigit.

The alternation applies to the largest possible regexps on eitherside. In other words, `|' has the lowest precedence of all theregular expression operators.

(: : :) Parentheses are used for grouping in regular expressions asin arithmetic. They can be used to concatenate regular ex-pressions containing the alternation operator, `|'. For exam-ple, `@(samp|code)\{[^}]+\}' matches both `@code{foo}' and`@samp{bar}'. (These are Texinfo formatting control sequences.)

* This symbol means that the preceding regular expression is tobe repeated as many times as necessary to �nd a match. Forexample:

ph*

applies the `*' symbol to the preceding `h' and looks for matchesof one `p' followed by any number of `h's. This will also matchjust `p' if no `h's are present.

The `*' repeats the smallest possible preceding expression. (Useparentheses if you wish to repeat a larger expression.) It �ndsas many repetitions as possible. For example:

awk '/$c[ad][ad]*r x$/ { print }' sample

prints every record in `sample' containing a string of the form`(car x)', `(cdr x)', `(cadr x)', and so on. Notice the escapingof the parentheses by preceding them with backslashes.

+ This symbol is similar to `*', but the preceding expression mustbe matched at least once. This means that:

wh+y

would match `why' and `whhy' but not `wy', whereas `wh*y' wouldmatch all three of these strings. This is a simpler way of writingthe last `*' example:

awk '/$c[ad]+r x$/ { print }' sample


? This symbol is similar to `*', but the preceding expression canbe matched either once or not at all. For example:

fe?d

will match `fed' and `fd', but nothing else.

{n}{n,}{n,m} One or two numbers inside braces denote an interval expres-

sion. If there is one number in the braces, the preceding regexpis repeated n times. If there are two numbers separated by acomma, the preceding regexp is repeated n to m times. If thereis one number followed by a comma, then the preceding regexpis repeated at least n times.

wh{3}y matches `whhhy' but not `why' or `whhhhy'.

wh{3,5}y matches `whhhy' or `whhhhy' or `whhhhhy', only.

wh{2,}y matches `whhy' or `whhhy', and so on.

Interval expressions were not traditionally available in awk. Aspart of the POSIX standard they were added, to make awk andegrep consistent with each other.

However, since old programs may use `{' and `}' in regexp con-stants, by default gawk does not match interval expressions inregexps. If either `--posix' or `--re-interval' are speci�ed(see Section 14.1 [Command Line Options], page 161), then in-terval expressions are allowed in regexps.

In regular expressions, the `*', `+', and `?' operators, as well as the braces`{' and `}', have the highest precedence, followed by concatenation, and�nally by `|'. As in arithmetic, parentheses can change how operators aregrouped.

If gawk is in compatibility mode (see Section 14.1 [Command Line Op-tions], page 161), character classes and interval expressions are not availablein regular expressions.

The next section discusses the GNU-speci�c regexp operators, and pro-vides more detail concerning how command line options a�ect the way gawkinterprets the characters in regular expressions.

4.4 Additional Regexp Operators Only in gawk

GNU software that deals with regular expressions provides a number ofadditional regexp operators. These operators are described in this section,and are speci�c to gawk; they are not available in other awk implementations.


Most of the additional operators are for dealing with word matching.For our purposes, a word is a sequence of one or more letters, digits, orunderscores (`_').

\w This operator matches any word-constituent character, i.e. anyletter, digit, or underscore. Think of it as a short-hand for[[:alnum:]_].

\W This operator matches any character that is not word-constituent. Think of it as a short-hand for [^[:alnum:]_].

\< This operator matches the empty string at the beginning ofa word. For example, /\ This operator matches the empty string at the end of a word.For example, /stow\>/ matches `stow', but not `stowaway'.

\y This operator matches the empty string at either the begin-ning or the end of a word (the word boundary). For exam-ple, `\yballs?\y' matches either `ball' or `balls' as a separateword.

\B This operator matches the empty string within a word. In otherwords, `\B' matches the empty string that occurs between twoword-constituent characters. For example, /\Brat\B/ matches`crate', but it does not match `dirty rat'. `\B' is essentiallythe opposite of `\y'.

There are two other operators that work on bu�ers. In Emacs, a bu�er is,naturally, an Emacs bu�er. For other programs, the regexp library routinesthat gawk uses consider the entire string to be matched as the bu�er.

For awk, since `^' and `$' always work in terms of the beginning and endof strings, these operators don't add any new capabilities. They are providedfor compatibility with other GNU software.

\` This operator matches the empty string at the beginning of thebu�er.

\' This operator matches the empty string at the end of the bu�er.

In other GNU software, the word boundary operator is `\b'. However,that conicts with the awk language's de�nition of `\b' as backspace, so gawkuses a di�erent letter.

An alternative method would have been to require two backslashes in theGNU operators, but this was deemed to be too confusing, and the currentmethod of using `\y' for the GNU `\b' appears to be the lesser of two evils.

The various command line options (see Section 14.1 [Command Line Op-tions], page 161) control how gawk interprets characters in regexps.


No optionsIn the default case, gawk provide all the facilities of POSIX reg-exps and the GNU regexp operators described above. However,interval expressions are not supported.

--posix Only POSIX regexps are supported, the GNU operators are notspecial (e.g., `\w' matches a literal `w'). Interval expressions areallowed.

--traditional

Traditional Unix awk regexps are matched. The GNU operatorsare not special, interval expressions are not available, and nei-ther are the POSIX character classes ([[:alnum:]] and so on).Characters described by octal and hexadecimal escape sequencesare treated literally, even if they represent regexp metacharac-ters.

--re-interval

Allow interval expressions in regexps, even if `--traditional'has been provided.

4.5 Case-sensitivity in Matching

Case is normally signi�cant in regular expressions, both when matchingordinary characters (i.e. not metacharacters), and inside character sets. Thusa `w' in a regular expression matches only a lower-case `w' and not an upper-case `W'.

The simplest way to do a case-independent match is to use a characterlist: `[Ww]'. However, this can be cumbersome if you need to use it often;and it can make the regular expressions harder to read. There are twoalternatives that you might prefer.

One way to do a case-insensitive match at a particular point in the pro-gram is to convert the data to a single case, using the tolower or toupperbuilt-in string functions (which we haven't discussed yet; see Section 12.3[Built-in Functions for String Manipulation], page 137). For example:

tolower($1) ~ /foo/ { : : : }

converts the �rst �eld to lower-case before matching against it. This willwork in any POSIX-compliant implementation of awk.

Another method, speci�c to gawk, is to set the variable IGNORECASE toa non-zero value (see Chapter 10 [Built-in Variables], page 115). WhenIGNORECASE is not zero, all regexp and string operations ignore case. Chang-ing the value of IGNORECASE dynamically controls the case sensitivity of yourprogram as it runs. Case is signi�cant by default because IGNORECASE (likemost variables) is initialized to zero.


x = "aB"

if (x ~ /ab/) : : : # this test will fail

IGNORECASE = 1

if (x ~ /ab/) : : : # now it will succeed

In general, you cannot use IGNORECASE to make certain rules case-insensitive and other rules case-sensitive, because there is no way to setIGNORECASE just for the pattern of a particular rule. To do this, you mustuse character lists or tolower. However, one thing you can do only withIGNORECASE is turn case-sensitivity on or o� dynamically for all the rules atonce.

IGNORECASE can be set on the command line, or in a BEGIN rule (seeSection 14.2 [Other Command Line Arguments], page 165; also see Sec-tion 8.1.5.1 [Startup and Cleanup Actions], page 100). Setting IGNORECASEfrom the command line is a way to make a program case-insensitive withouthaving to edit it.

Prior to version 3.0 of gawk, the value of IGNORECASE only a�ected regexpoperations. It did not a�ect string comparison with `==', `!=', and so on.Beginning with version 3.0, both regexp and string comparison operationsare a�ected by IGNORECASE.

Beginning with version 3.0 of gawk, the equivalences between upper-caseand lower-case characters are based on the ISO-8859-1 (ISO Latin-1) char-acter set. This character set is a superset of the traditional 128 ASCIIcharacters, that also provides a number of characters suitable for use withEuropean languages.

The value of IGNORECASE has no e�ect if gawk is in compatibility mode(see Section 14.1 [Command Line Options], page 161). Case is always sig-ni�cant in compatibility mode.

4.6 HowMuch Text Matches?

Consider the following example:

echo aaaabcd | awk '{ sub(/a+/, ""); print }'

This example uses the sub function (which we haven't discussed yet, seeSection 12.3 [Built-in Functions for String Manipulation], page 137) to makea change to the input record. Here, the regexp /a+/ indicates \one or moreà' characters," and the replacement text is `'.

The input contains four à' characters. What will the output be? In otherwords, how many is \one or more"|will awk match two, three, or all fourà' characters?


The answer is, awk (and POSIX) regular expressions always match theleftmost, longest sequence of input characters that can match. Thus, in thisexample, all four `a' characters are replaced with `'.

$ echo aaaabcd | awk '{ sub(/a+/, ""); print }'

a bcd

For simple match/no-match tests, this is not so important. But when do-ing regexp-based �eld and record splitting, and text matching and substitu-tions with the match, sub, gsub, and gensub functions, it is very important.Understanding this principle is also important for regexp-based record and�eld splitting (see Section 5.1 [How Input is Split into Records], page 37,and also see Section 5.5 [Specifying How Fields are Separated], page 44).

4.7 Using Dynamic Regexps

The right hand side of a `~' or `!~' operator need not be a regexp constant(i.e. a string of characters between slashes). It may be any expression. Theexpression is evaluated, and converted if necessary to a string; the contentsof the string are used as the regexp. A regexp that is computed in this wayis called a dynamic regexp. For example:

BEGIN { identifier_regexp = "[A-Za-z_][A-Za-z_0-9]+" }

$0 ~ identifier_regexp { print }

sets identifier_regexp to a regexp that describes awk variable names, andtests if the input record matches this regexp.

Caution: When using the `~' and `!~' operators, there is a di�erencebetween a regexp constant enclosed in slashes, and a string constant enclosedin double quotes. If you are going to use a string constant, you have tounderstand that the string is in essence scanned twice; the �rst time whenawk reads your program, and the second time when it goes to match thestring on the left-hand side of the operator with the pattern on the right.This is true of any string valued expression (such as identifier_regexpabove), not just string constants.

What di�erence does it make if the string is scanned twice? The answerhas to do with escape sequences, and particularly with backslashes. To geta backslash into a regular expression inside a string, you have to type twobackslashes.

For example, /\*/ is a regexp constant for a literal `*'. Only one backslashis needed. To do the same thing with a string, you would have to type"\\*". The �rst backslash escapes the second one, so that the string actuallycontains the two characters `\' and `*'.

Given that you can use both regexp and string constants to describe reg-ular expressions, which should you use? The answer is \regexp constants,"for several reasons.


1. String constants are more complicated to write, and more di�cult toread. Using regexp constants makes your programs less error-prone.Not understanding the di�erence between the two kinds of constants isa common source of errors.

2. It is also more e�cient to use regexp constants: awk can note that youhave supplied a regexp and store it internally in a form that makespattern matching more e�cient. When using a string constant, awkmust �rst convert the string into this internal form, and then performthe pattern matching.

3. Using regexp constants is better style; it shows clearly that you intenda regexp match.

Chapter 5: Reading Input Files 37

5 Reading Input Files

In the typical awk program, all input is read either from the standardinput (by default the keyboard, but often a pipe from another command) orfrom �les whose names you specify on the awk command line. If you specifyinput �les, awk reads them in order, reading all the data from one beforegoing on to the next. The name of the current input �le can be found in thebuilt-in variable FILENAME (see Chapter 10 [Built-in Variables], page 115).

The input is read in units called records, and processed by the rules ofyour program one record at a time. By default, each record is one line. Eachrecord is automatically split into chunks called �elds. This makes it moreconvenient for programs to work on the parts of a record.

On rare occasions you will need to use the getline command. Thegetline command is valuable, both because it can do explicit input from anynumber of �les, and because the �les used with it do not have to be namedon the awk command line (see Section 5.8 [Explicit Input with getline],page 54).

5.1 How Input is Split into Records

The awk utility divides the input for your awk program into records and�elds. Records are separated by a character called the record separator. Bydefault, the record separator is the newline character. This is why recordsare, by default, single lines. You can use a di�erent character for the recordseparator by assigning the character to the built-in variable RS.

You can change the value of RS in the awk program, like any other variable,with the assignment operator, `=' (see Section 7.7 [Assignment Expressions],page 84). The new record-separator character should be enclosed in quo-tation marks, which indicate a string constant. Often the right time to dothis is at the beginning of execution, before any input has been processed,so that the very �rst record will be read with the proper separator. To dothis, use the special BEGIN pattern (see Section 8.1.5 [The BEGIN and ENDSpecial Patterns], page 100). For example:

awk 'BEGIN { RS = "/" } ; { print $0 }' BBS-list

changes the value of RS to "/", before reading any input. This is a stringwhose �rst character is a slash; as a result, records are separated by slashes.Then the input �le is read, and the second rule in the awk program (theaction with no pattern) prints each record. Since each print statementadds a newline at the end of its output, the e�ect of this awk program is tocopy the input with each slash changed to a newline. Here are the results ofrunning the program on `BBS-list':


$ awk 'BEGIN { RS = "/" } ; { print $0 }' BBS-list

a aardvark 555-5553 1200

a 300 B

a alpo-net 555-3412 2400

a 1200

a 300 A

a barfly 555-7685 1200

a 300 A

a bites 555-1675 2400

a 1200a 300 A

a camelot 555-0542 300 C

a core 555-2912 1200

a 300 C

a fooey 555-1234 2400

a 1200

a 300 B

a foot 555-6699 1200

a 300 Ba macfoo 555-6480 1200

a 300 A

a sdace 555-3430 2400

a 1200

a 300 A

a sabafoo 555-2127 1200

a 300 C

a

Note that the entry for the `camelot' BBS is not split. In the original data�le (see Section 1.3 [Data Files for the Examples], page 7), the line looks likethis:

camelot 555-0542 300 C

It only has one baud rate; there are no slashes in the record.

Another way to change the record separator is on the command line,using the variable-assignment feature (see Section 14.2 [Other CommandLine Arguments], page 165).

awk '{ print $0 }' RS="/" BBS-list

This sets RS to `/' before processing `BBS-list'.

Using an unusual character such as `/' for the record separator producescorrect behavior in the vast majority of cases. However, the following (ex-treme) pipeline prints a surprising `1'. There is one �eld, consisting of a


newline. The value of the built-in variable NF is the number of �elds in thecurrent record.

$ echo | awk 'BEGIN { RS = "a" } ; { print NF }'

a 1

Reaching the end of an input �le terminates the current input record, evenif the last character in the �le is not the character in RS (d.c.).

The empty string, "" (a string of no characters), has a special meaningas the value of RS: it means that records are separated by one or more blanklines, and nothing else. See Section 5.7 [Multiple-Line Records], page 51, formore details.

If you change the value of RS in the middle of an awk run, the new value isused to delimit subsequent records, but the record currently being processed(and records already processed) are not a�ected.

After the end of the record has been determined, gawk sets the variableRT to the text in the input that matched RS.

The value of RS is in fact not limited to a one-character string. It canbe any regular expression (see Chapter 4 [Regular Expressions], page 23).In general, each record ends at the next string that matches the regularexpression; the next record starts at the end of the matching string. Thisgeneral rule is actually at work in the usual case, where RS contains just anewline: a record ends at the beginning of the next matching string (thenext newline in the input) and the following record starts just after the endof this string (at the �rst character of the following line). The newline, sinceit matches RS, is not part of either record.

When RS is a single character, RT will contain the same single character.However, when RS is a regular expression, then RT becomes more useful; itcontains the actual input text that matched the regular expression.

The following example illustrates both of these features. It sets RS equalto a regular expression that matches either a newline, or a series of one ormore upper-case letters with optional leading and/or trailing white space(see Chapter 4 [Regular Expressions], page 23).

$ echo record 1 AAAA record 2 BBBB record 3 |

> gawk 'BEGIN { RS = "\n|( *[[:upper:]]+ *)" }

> { print "Record =", $0, "and RT =", RT }'

a Record = record 1 and RT = AAAA

a Record = record 2 and RT = BBBB

a Record = record 3 and RT =

a

The �nal line of output has an extra blank line. This is because the value ofRT is a newline, and then the print statement supplies its own terminatingnewline.


See Section 16.2.8 [A Simple Stream Editor], page 240, for a more usefulexample of RS as a regexp and RT.

The use of RS as a regular expression and the RT variable are gawk exten-sions; they are not available in compatibility mode (see Section 14.1 [Com-mand Line Options], page 161). In compatibility mode, only the �rst char-acter of the value of RS is used to determine the end of the record.

The awk utility keeps track of the number of records that have been readso far from the current input �le. This value is stored in a built-in variablecalled FNR. It is reset to zero when a new �le is started. Another built-invariable, NR, is the total number of input records read so far from all data�les. It starts at zero but is never automatically reset to zero.

5.2 Examining Fields

When awk reads an input record, the record is automatically separatedor parsed by the interpreter into chunks called �elds. By default, �elds areseparated by whitespace, like words in a line. Whitespace in awk means anystring of one or more spaces and/or tabs; other characters such as newline,formfeed, and so on, that are considered whitespace by other languages arenot considered whitespace by awk.

The purpose of �elds is to make it more convenient for you to refer tothese pieces of the record. You don't have to use them|you can operate onthe whole record if you wish|but �elds are what make simple awk programsso powerful.

To refer to a �eld in an awk program, you use a dollar-sign, `$', followedby the number of the �eld you want. Thus, $1 refers to the �rst �eld, $2 tothe second, and so on. For example, suppose the following is a line of input:

This seems like a pretty nice example.

Here the �rst �eld, or $1, is `This'; the second �eld, or $2, is `seems'; andso on. Note that the last �eld, $7, is èxample.'. Because there is no spacebetween the è' and the `.', the period is considered part of the seventh �eld.

NF is a built-in variable whose value is the number of �elds in the currentrecord. awk updates the value of NF automatically, each time a record isread.

No matter how many �elds there are, the last �eld in a record can berepresented by $NF. So, in the example above, $NF would be the same as$7, which is èxample.'. Why this works is explained below (see Section 5.3[Non-constant Field Numbers], page 41). If you try to reference a �eldbeyond the last one, such as $8 when the record has only seven �elds, youget the empty string.


$0, which looks like a reference to the \zeroth" �eld, is a special case: itrepresents the whole input record. $0 is used when you are not interested in�elds.

Here are some more examples:

$ awk '$1 ~ /foo/ { print $0 }' BBS-list

a fooey 555-1234 2400/1200/300 B

a foot 555-6699 1200/300 B

a macfoo 555-6480 1200/300 A

a sabafoo 555-2127 1200/300 C

This example prints each record in the �le `BBS-list' whose �rst �eld con-tains the string `foo'. The operator `~' is called a matching operator (seeSection 4.1 [How to Use Regular Expressions], page 23); it tests whether astring (here, the �eld $1) matches a given regular expression.

By contrast, the following example looks for `foo' in the entire recordand prints the �rst �eld and the last �eld for each input record containing amatch.

$ awk '/foo/ { print $1, $NF }' BBS-list

a fooey B

a foot B

a macfoo A

a sabafoo C

5.3 Non-constant Field Numbers

The number of a �eld does not need to be a constant. Any expression inthe awk language can be used after a `$' to refer to a �eld. The value of theexpression speci�es the �eld number. If the value is a string, rather than anumber, it is converted to a number. Consider this example:

awk '{ print $NR }'

Recall that NR is the number of records read so far: one in the �rst record,two in the second, etc. So this example prints the �rst �eld of the �rstrecord, the second �eld of the second record, and so on. For the twentiethrecord, �eld number 20 is printed; most likely, the record has fewer than 20�elds, so this prints a blank line.

Here is another example of using expressions as �eld numbers:

awk '{ print $(2*2) }' BBS-list

awk must evaluate the expression `(2*2)' and use its value as the numberof the �eld to print. The `*' sign represents multiplication, so the expression`2*2' evaluates to four. The parentheses are used so that the multiplicationis done before the `$' operation; they are necessary whenever there is abinary operator in the �eld-number expression. This example, then, prints


the hours of operation (the fourth �eld) for every line of the �le `BBS-list'.(All of the awk operators are listed, in order of decreasing precedence, inSection 7.14 [Operator Precedence (How Operators Nest)], page 95.)

If the �eld number you compute is zero, you get the entire record. Thus,$(2-2) has the same value as $0. Negative �eld numbers are not allowed;trying to reference one will usually terminate your running awk program.(The POSIX standard does not de�ne what happens when you reference anegative �eld number. gawk will notice this and terminate your program.Other awk implementations may behave di�erently.)

As mentioned in Section 5.2 [Examining Fields], page 40, the numberof �elds in the current record is stored in the built-in variable NF (also seeChapter 10 [Built-in Variables], page 115). The expression $NF is not aspecial feature: it is the direct consequence of evaluating NF and using itsvalue as a �eld number.

5.4 Changing the Contents of a Field

You can change the contents of a �eld as seen by awk within an awkprogram; this changes what awk perceives as the current input record. (Theactual input is untouched; awk never modi�es the input �le.)

Consider this example and its output:

$ awk '{ $3 = $2 - 10; print $2, $3 }' inventory-shipped

a 13 3

a 15 5

a 15 5: : :

The `-' sign represents subtraction, so this program reassigns �eld three, $3,to be the value of �eld two minus ten, `$2 - 10'. (See Section 7.5 [ArithmeticOperators], page 82.) Then �eld two, and the new value for �eld three, areprinted.

In order for this to work, the text in �eld $2 must make sense as anumber; the string of characters must be converted to a number in orderfor the computer to do arithmetic on it. The number resulting from thesubtraction is converted back to a string of characters which then becomes�eld three. See Section 7.4 [Conversion of Strings and Numbers], page 81.

When you change the value of a �eld (as perceived by awk), the text ofthe input record is recalculated to contain the new �eld where the old onewas. Therefore, $0 changes to reect the altered �eld. Thus, this programprints a copy of the input �le, with 10 subtracted from the second �eld ofeach line.


$ awk '{ $2 = $2 - 10; print $0 }' inventory-shipped

a Jan 3 25 15 115

a Feb 5 32 24 226

a Mar 5 24 34 228: : :

You can also assign contents to �elds that are out of range. For example:

$ awk '{ $6 = ($5 + $4 + $3 + $2)

> print $6 }' inventory-shipped

a 168

a 297

a 301: : :

We've just created $6, whose value is the sum of �elds $2, $3, $4, and$5. The `+' sign represents addition. For the �le `inventory-shipped', $6represents the total number of parcels shipped for a particular month.

Creating a new �eld changes awk's internal copy of the current inputrecord|the value of $0. Thus, if you do `print $0' after adding a �eld, therecord printed includes the new �eld, with the appropriate number of �eldseparators between it and the previously existing �elds.

This recomputation a�ects and is a�ected by NF (the number of �elds;see Section 5.2 [Examining Fields], page 40), and by a feature that has notbeen discussed yet, the output �eld separator, OFS, which is used to separatethe �elds (see Section 6.3 [Output Separators], page 63). For example, thevalue of NF is set to the number of the highest �eld you create.

Note, however, that merely referencing an out-of-range �eld does notchange the value of either $0 or NF. Referencing an out-of-range �eld onlyproduces an empty string. For example:

if ($(NF+1) != "")

print "can't happen"

else

print "everything is normal"

should print `everything is normal', because NF+1 is certain to be out ofrange. (See Section 9.1 [The if-else Statement], page 105, for more infor-mation about awk's if-else statements. See Section 7.10 [Variable Typingand Comparison Expressions], page 88, for more information about the `!='operator.)

It is important to note that making an assignment to an existing �eldwill change the value of $0, but will not change the value of NF, even whenyou assign the empty string to a �eld. For example:


$ echo a b c d | awk '{ OFS = ":"; $2 = ""

> print $0; print NF }'

a a::c:d

a 4

The �eld is still there; it just has an empty value. You can tell because thereare two colons in a row.

This example shows what happens if you create a new �eld.

$ echo a b c d | awk '{ OFS = ":"; $2 = ""; $6 = "new"

> print $0; print NF }'

a a::c:d::new

language programming - wikimedia...a wk language programming a user's guide for gnu a wk edition 1.0...

Documents