1

Regexes vs Regular Expressions; andRecursive Descent Parser

Ras Bodik, Thibaud Hottelier, James IdeUC Berkeley

CS164: Introduction to Programming Languages and Compilers Fall 2010

Expressiveness of recognizersWhat does it mean to "tell strings apart"?

Or "test a string" or "recognize a language", where language = a (potentially infinite) set of strings

It is to accept only a string with that has some property

such as can be written as ('1'*k)*m, k>1, m>1or contains only balanced parentheses: ((())()(()))

Why can't a reg expression test for ('1'*k)*m, k>1,m>1 ?

Recall reg expression: char . | *We can use sugar to add e+, by rewriting e+ to e.e*We can also add e++, which means 2+ of e: e++ --> e.e.e*

2

… continuedSo it seems we can test for ('1'*k)*m, k>1,m>1, right?

(1++)++ rewrite 1++ using e++ --> e.e+(11+)++ rewrite (11+)++ using e++ --> e.e+(11+)(11+)+

Now why isn't (11+)(11+)+ the same as (11+)\1+ ?

How do we show these test for different property?

3

A refresherRegexes and regular expressions both support operators in this grammar

R ::= char | R R | R* | R ‘|’ R

Regexes suppot more operators, such as backreferences \1, \2,Capturing groups

but let’s ignore this for now.

4

Regexes vs RERegexes implemented with backtracking

This regex requires exponential time to discover that it does not match the input string X==============.

X(.+)+X

REs implemented by translation to NFA, which is then translated to DFA.

Corresponding regular expression requires only linear time, after converted to DFA.

5

MatchAllOn the problem of detecting whether a pattern (regex or RE) matches the entire string, both regex and RE interpretation of a patter agree

– After all, to match the whole string, it is sufficient to find any number of times that a Kleene star matches

6

Let’s now focus on when regex and RE differ

7

Example from Jeff Friedl’s book Imagine you want to parse a config file:

filesToCompile=a.cpp b.cppThe regex for this command line format:

[a-zA-Z]+=.*Now let’s allow an optional \n-separated 2nd

line:filesToCompile=a.cpp b.cpp \<\n> d.cpp e.h

We extend the original regex: [a-zA-Z]+=.*(\\\n.*)?

This regex does not match our two-line input. Why?

What compiler textbooks don’t teach youThe textbook string matching problem is

simple:Does a regex r match the entire string s?– a clean statement and suitable for theoretical

study– here is where regexes and FSMs are equivalent

The matching problem in the Real World:Given a string s and a regex r, find a substring

in s matching r.Do you see the language design issue here?

– There may be many such substrings. – We need to decide which substring to find.

It is easy to agree where the substring should start:– the matched substring should be the leftmost

match

Two schools of regexesThey differ in where it should end:Declarative approach: longest of all matches

– conceptually, enumerate all matches and return longest

Operational approach: define behavior of *, | operatorse* match e as many times as possible while

allowing the remainder of the regex t o match

e|e select leftmost choice while allowing remainder to match

[a-zA-Z]+ = .* ( \\ \n .* )?

filesToCompile=a.cpp b.cpp \<\n> d.cpp e.h

These are important differencesWe saw a non-contrived regex can behave

differently– personal story: I spent 3 hours debugging a

similar regex– despite reading the manual carefully

The (greedy) operational semantics of * – does not guarantee longest match (in case you

need it)– forces the programmer to reason about

backtrackingIt seems that backtracking is nice to reason

about– because it’s local: no need to consider the

entire regex– cognitive load is actually higher, as it breaks

composition

11

Where in history of re did things go wrong?

It’s tempting to blame perl– but the greedy regex semantics seems older– there are other reasons why backtracking is used

Hypothesis 1:creators of re libs knew not that NFA can – can be the target language for compiling regexes– find all matches simultaneously (no backtracking)– be implemented efficiently (convert NFA to DFA)

Hypothesis 2: their hands were tied– Ken Thompson’s algorithm for re-to-NFA was

patentedWith backtracking came the greedy semantics

– longest match would be expensive (must try all matches)

– so semantics was defined greedily, and non-compositionally

Concepts• Syntax tree-directed translation (re to NFA)• recognizers: tell strings apart• NFA, DFA, regular expressions = equally

powerful• but \1 (backreference) makes regexes

more pwrful• Syntax sugar: e+ to e.e*• Compositionality: be weary of greedy

semantics• Metacharacters: characters with special

meaning13

Summary of DFA, NFA, RegexpWhat you need to understand and remember

– what is DFA, NFA, regular expression– the three have equal expressive power– what is the “expressive power”– you can convert

• RE NFA DFA• NFA RE • and hence also DFA RE, because DFA is a special

case of NFA– NFAs are easier to use, more costly to execute

• NFA emulation O(S2)-times slower than DFA• conversion NFADFA incurs exponential cost in space

Some of these concepts will be covered in the section 14

Recursive descent parser

15

16

Recursive Descent ParserPoor man’s backtracking parser

does not do full backtracking you must be a bit carefulbut quite fast, despite backtracking, and simple to implement

many successful languages implemented with r.d. parser

– in many situations, this parser is all you will needwhen could you use an even simpler parser?

– when the grammar is not (heavily) recursive • ex: parse a formatted email message for answers to a

quiz– you could use the “spaghetti code” parser from

last lecture– but this simplification may not be worth it– because r.d. parser makes the grammar clear

maintainable

17

A Recursive Descent Parser (1)

write a function for each terminal, production, non-terminal

– return true iff input matches that terminal, production, n/t

– advance next

Terminals:bool term(TOKEN tok) { return in[next++] == tok; }

nth production of non-terminal S:bool Sn() { … }

non-terminal S: bool S() { … }

18

A Recursive Descent Parser (2)

For production E T + E bool E1() { return T() && term(PLUS) && E(); }For production E T bool E2() { return T(); }For all productions of E (with backtracking)

bool E() { int save = next; return (next = save, E1()) || (next =

save, E2()); }

Ras Bodik, CS 164, Spring 2007

19

A Recursive Descent Parser (4)Functions for non-terminal Tbool T1() { return term(OPEN) && E() &&

term(CLOSE); }bool T2() { return term(INT) && term(TIMES) &&

T(); }bool T3() { return term(INT); }

bool T() { int save = next; return (next = save, T1()) || (next = save, T2()) || (next = save, T3()); }

20

Recursive Descent Parsing. Notes.To start the parser

– Initialize next to point to first token– Invoke E()

Notice how this simulates our backtracking parser

– but r.d. parser does not perform full backtracking

– this is important to remember (see example in a HW)

LL and LR parsing algorithms are more efficient

– see a compiler textbook if interested

Ras Bodik, CS 164, Spring 2007

21

First problem with Recursive-Descent Parsing

Parsing: – given a string of tokens t1 t2 ... tn, find its parse

treeRecursive-descent parsing, backtracking parsing

– Try all the productions (almost) exhaustively– At a given moment the fringe of the parse tree is:

t1 t2 … tk A …– ie, parser will eventually derive a string starting

with terminals– parser compares this prefix with the remainder of

the input– if mismatch, parser backtracks

• but there are grammars such that – parser will NEVER derive a string starting with a

terminal

Eliminating left recursion

23

When Recursive Descent Does Not Always WorkConsider a production S S a:

– In the process of parsing S we try the above rule

– What goes wrong?A left-recursive grammar has a non-terminal S

S + S for some + : derives in one or more steps

Recursive descent may not work in such cases

– It may go into an loopYou say “may”?

– is there a left-recursive. grammar that r.d. parsers can handle?

24

Elimination of Left Recursion• Consider the left-recursive grammar S S |

• S generates all strings starting with a and followed by a number of

• Can rewrite using right-recursion S S’ S’ S’ |

25

Elimination of Left-Recursion. Example• Consider the grammar

S 1 | S 0 ( = 1 and = 0 )

can be rewritten as S 1 S’

S’ 0 S’ |

26

Oops, didn’t we break anything in the process?Consider the grammar for additions:

E E + id idAfter left-recursion elimination:

E id E’ E’ + id E’

Draw the parse tree for id+id+idyour figure comes here

Ras Bodik, CS 164, Spring 2007

27

More Elimination of Left-RecursionIn general S S 1 | … | S n | 1 | … | m

All strings derived from S start with one of 1,…,m and continue with several instances of 1,…,n Rewrite as

S 1 S’ | … | m S’ S’ 1 S’ | … | n S’ |

28

General Left Recursion• The grammar

S A | A S is also left-recursive because

S + S

• This left-recursion can also be eliminated• See [ALSU], Section 4.3 for general

algorithm

29

A comment on removing left recursion• Not a big deal in practice

– ie, you won’t have to convert from left recursion too often

• Just define a right-recursive grammar from the start– works for many cases

• Example: list of arguments – btw, lists are common in programming

language grammars– Left recursive: LIST id LIST , id– Right recursive: LIST id id , LIST – Just opt for the second alternative!

Left Factoring

Ras Bodik, CS 164, Spring 2007

31

Are all grammars equally efficient for r.d.p.?• Consider this grammar:

E T + E T – E TT id * T id / T id

• Parse this stringid * id

• Do you see the inefficiency?– the parser will repeat this derivation three

times (try it)T id * T id * id

32

Left Factoring• reduces backtracking in r.d. parser

• beforeE T + E T – E TT id * T id / T id

• afterE T E’E’ + E – E T id T’T’ * T / T

Limited Backtracking

34

Order of productions may matter in r.d. parser• Consider this grammar

E T + E T – E TT F F * T F / T ---- here we are

trying T F firstF id n ( E )

• Now try to parseid * id

• Why does the r.d. parser return “syntax error”?– it never backtracks and tries T F * T– it only tries T F and succeeds

• Lesson: put longer productions first

35

Summary of Recursive Descent• Simple and general parsing strategy

– Left recursion must be eliminated first– Left factoring not essential but helps reduce

backtracking– Ambiguity must be removed– Order of productions compensates for limited

backtracking

• Do you have to do all these by hand?– first two can be done automatically– third needs intelligence– last could perhaps be automated, too