5-parserconstruction

8/4/2019 5-parserconstruction

1/58

COS 320

Compilers

David Walker


2/58

last time

context free grammars (Appel 3.1)

terminals, non-terminals, rules

derivations & parse trees

ambiguous grammars

recursive descent parsers (Appel 3.2)

parse LL(k) grammars

easy to write as ML programs

algorithms for automatic construction from a CFG


4/58

Constructing RD Parsers

To construct an RD parser, we need to knowwhat rule to apply when

we have seen a non terminal X

we see the next terminal a in input

We apply rule X ::= s when

a is the first symbol that can be generated by string s,OR

s reduces to the empty string (is nullable) and a is thefirst symbol in any string that can followX


5/58

Computing Nullable Sets

Non-terminal X is Nullable only if thefollowing constraints are satisfied(computed using iterative analysis)

base case:

if (X := ) then X is Nullable

inductive case:

if (X := ABC...) and A, B, C, ... are all Nullable thenX is Nullable


6/58

Computing First Sets

First(X) is computed iteratively

base case:

if T is a terminal symbol then First (T) = {T}

inductive case:

if X is a non-terminal and (X:= ABC...) then

First (X) = First (X) U First (ABC...)

where First(ABC...) = F1 U F2 U F3 U ... and

F1 = First (A)

F2 = First (B), if A is Nullable

F3 = First (C), if A is Nullable & B is Nullable

...


7/58

Computing Follow Sets

Follow(X) is computed iteratively

base case:

initially, we assume nothing in particular follows X

(Follow (X) is initially { })

inductive case:

if (Y := s1 X s2) for any strings s1, s2 then

Follow (X) = First (s2) U Follow (X)

if (Y := s1 X s2) for any strings s1, s2 then

Follow (X) = Follow(Y) U Follow (X), if s2 is Nullable


8/58

building a predictive parser

Z ::= X Y ZZ ::= d

Y ::= cY ::=

X ::= aX ::= b Y e

nullable first follow

Z

YX


9/58


Z ::= X Y ZZ ::= d

Y ::= cY ::=

X ::= aX ::= b Y e


Z no

Y yesX no

base case


10/58


Z ::= X Y ZZ ::= d

Y ::= cY ::=

X ::= aX ::= b Y e


Z no

Y yesX no

after one round of induction, we realize we have reached a fixed point


11/58


Z ::= X Y ZZ ::= d

Y ::= cY ::=

X ::= aX ::= b Y e


Z no d

Y yes cX no a,b

base case


12/58


Z ::= X Y ZZ ::= d

Y ::= cY ::=

X ::= aX ::= b Y e


Z no d,a,b

Y yes cX no a,b

after one round of induction, no fixed point


13/58


Z ::= X Y ZZ ::= d

Y ::= cY ::=

X ::= aX ::= b Y e


Z no d,a,b

Y yes cX no a,b

after two rounds of induction, no more changes ==> fixed point


14/58


Z ::= X Y ZZ ::= d

Y ::= cY ::=

X ::= aX ::= b Y e


Z no d,a,b { }

Y yes c { }X no a,b { }

base case


15/58


Z ::= X Y ZZ ::= d

Y ::= cY ::=

X ::= aX ::= b Y e


Z no d,a,b { }

Y yes c e,d,a,bX no a,b c,d,a,b

after one round of induction, no fixed point


16/58


Z ::= X Y ZZ ::= d

Y ::= cY ::=

X ::= aX ::= b Y e


Z no d,a,b { }

Y yes c e,d,a,bX no a,b c,d,a,b

after two rounds of induction, fixed point

(but notice, computing Follow(X) before Follow (Y) would have required 3rd round)


17/58

Z ::= X Y ZZ ::= d

Y ::= cY ::=

X ::= aX ::= b Y e


Z no d,a,b { }

Y yes c e,d,a,b

X no a,b c,d,a,b

Grammar: Computed Sets:

Build parsing table where row X, col Ttells parser which clause to execute infunction X with next-token T:

a b c d e

Z

Y

X

if T First(s) thenenter (X ::= s) in row X, col T

if s is Nullable and T Follow(X)enter (X ::= s) in row X, col T


18/58

Z ::= X Y ZZ ::= d

Y ::= cY ::=

X ::= aX ::= b Y e


Z no d,a,b { }

Y yes c e,d,a,b

X no a,b c,e,d,a,b



a b c d e

Z Z ::= XYZ Z ::= XYZ

Y

X




19/58

Z ::= X Y ZZ ::= d

Y ::= cY ::=

X ::= aX ::= b Y e


Z no d,a,b { }

Y yes c e,d,a,b

X no a,b c,e,d,a,b



a b c d e

Z Z ::= XYZ Z ::= XYZ Z ::= d

Y

X




20/58

Z ::= X Y ZZ ::= d

Y ::= cY ::=

X ::= aX ::= b Y e


Z no d,a,b { }

Y yes c e,d,a,b

X no a,b c,e,d,a,b



a b c d e

Z Z ::= XYZ Z ::= XYZ Z ::= d

Y Y ::= c

X




21/58

Z ::= X Y ZZ ::= d

Y ::= cY ::=

X ::= aX ::= b Y e


Z no d,a,b { }

Y yes c e,d,a,b

X no a,b c,e,d,a,b



a b c d e

Z Z ::= XYZ Z ::= XYZ Z ::= d

Y Y ::= Y ::= Y ::= c Y ::= Y ::=

X




22/58

Z ::= X Y ZZ ::= d

Y ::= cY ::=

X ::= aX ::= b Y e


Z no d,a,b { }

Y yes c e,d,a,b

X no a,b c,e,d,a,b



a b c d e

Z Z ::= XYZ Z ::= XYZ Z ::= d

Y Y ::= Y ::= Y ::= c Y ::= Y ::=

X X ::= a X ::= b Y e




23/58

Z ::= X Y ZZ ::= d

Y ::= cY ::=

X ::= aX ::= b Y e


Z no d,a,b { }

Y yes c e,d,a,b

X no a,b c,e,d,a,b


What are the blanks?

a b c d e

Z Z ::= XYZ Z ::= XYZ Z ::= d

Y Y ::= Y ::= Y ::= c Y ::= Y ::=

X X ::= a X ::= b Y e


24/58

Z ::= X Y ZZ ::= d

Y ::= cY ::=

X ::= aX ::= b Y e


Z no d,a,b { }

Y yes c e,d,a,b

X no a,b c,e,d,a,b


What are the blanks? --> syntax errors

a b c d e

Z Z ::= XYZ Z ::= XYZ Z ::= d

Y Y ::= Y ::= Y ::= c Y ::= Y ::=

X X ::= a X ::= b Y e


25/58

Z ::= X Y ZZ ::= d

Y ::= cY ::=

X ::= aX ::= b Y e


Z no d,a,b { }

Y yes c e,d,a,b

X no a,b c,e,d,a,b


Is it possible to put 2 grammar rules in the same box?

a b c d e

Z Z ::= XYZ Z ::= XYZ Z ::= d

Y Y ::= Y ::= Y ::= c Y ::= Y ::=

X X ::= a X ::= b Y e


26/58

Z ::= X Y ZZ ::= dZ ::= d e

Y ::= cY ::=

X ::= aX ::= b Y e


Z no d,a,b { }

Y yes c e,d,a,b

X no a,b c,e,d,a,b


Is it possible to put 2 grammar rules in the same box?

a b c d e

Z Z ::= XYZ Z ::= XYZ Z ::= d

Z ::= d e

Y Y ::= Y ::= Y ::= c Y ::= Y ::=

X X ::= a X ::= b Y e


27/58

predictive parsing tables

if a predictive parsing table constructed this waycontains no duplicate entries, the grammar iscalled LL(1)

Left-to-right parse, Left-most derivation, 1 symbollookahead

if not, of the grammar is not LL(1)

in LL(k) parsing table, columns include every k-

length sequence of terminals:

aa ab ba bb ac ca ...


28/58

another trick

Previously, we saw that grammars withleft-recursion were problematic, but couldbe transformed into LL(1) in some cases

the example non-LL(1) grammar we justsaw:

how do we fix it?

Z ::= X Y ZZ ::= dZ ::= d e

Y ::= cY ::=

X ::= aX ::= b Y e


29/58

another trick

Previously, we saw that grammars withleft-recursion were problematic, but couldbe transformed into LL(1) in some cases

the example non-LL(1) grammar we justsaw:

solution here is left-factoring:

Z ::= X Y ZZ ::= dZ ::= d e

Y ::= cY ::=

X ::= aX ::= b Y e

Z ::= X Y ZZ ::= d W Y ::= c

Y ::=X ::= aX ::= b Y e

W ::=W ::= e


30/58

summary of RD parsing

CFGs are good at specifying programming language structure

parsing general CFGs is expensive so we define parsers forsimpler classes of CFG LL(k), LR(k)

we can build a recursive descent parser for LL(k) grammars by: computing nullable, first and follow sets

constructing a parse table from the sets

checking for duplicate entries, which indicates failure

creating an ML program from the parse table

if parser construction fails we can rewrite the grammar (left factoring, eliminating left recursion) and try

again

try to build a parser using some other method


31/58

summary of RD parsing

CFGs are good at specifying programming language structure

parsing general CFGs is expensive so we define parsers forsimpler classes of CFG LL(k), LR(k)

we can build a recursive descent parser for LL(k) grammars by: computing nullable, first and follow sets

constructing a parse table from the sets

checking for duplicate entries, which indicates failure

creating an ML program from the parse table

if parser construction fails we can rewrite the grammar (left factoring, eliminating left recursion) and try

again

try to build a parser using some other method...such as using a bottom-up parsing technique


32/58

Bottom-up (Shift-Reduce) Parsing


33/58

shift-reduce parsing

shift-reduce parsing aka: bottom-up parsing

aka: LR(k) Left-to-right parse, Rightmost

derivation, k-token lookahead more powerful than LL(k) parsers

LALR variant:

the basis for parsers for most modernprogramming languages

implemented in tools such as ML-Yacc


34/58

shift-reduce parsing example

A ::= S EOF L ::= L ; SL ::= S

Grammar:

S ::= ( L )S ::= id = num

Parsing Table


35/58

State of parse so far:

( id = num ; id = num ) EOF


A ::= S EOF L ::= L ; SL ::= S

Grammar:

S ::= ( L )S ::= id = num

Input from lexer:

yet to read

Parsing Table


36/58




A ::= S EOF L ::= L ; SL ::= S

Grammar:

S ::= ( L )S ::= id = num

Input from lexer:

(

yet to read

Parsing Table

SHIFT


37/58




A ::= S EOF L ::= L ; SL ::= S

Grammar:

S ::= ( L )S ::= id = num

Input from lexer:

( id

yet to read

Parsing Table

SHIFT


38/58




A ::= S EOF L ::= L ; SL ::= S

Grammar:

S ::= ( L )S ::= id = num

Input from lexer:

( id =

yet to read

Parsing Table

SHIFT


39/58




A ::= S EOF L ::= L ; SL ::= S

Grammar:

S ::= ( L )S ::= id = num

Input from lexer:

( id = num

yet to read

Parsing Table

SHIFT


40/58




A ::= S EOF L ::= L ; SL ::= S

Grammar:

S ::= ( L )S ::= id = num

Input from lexer:

( S

yet to read

Parsing Table

REDUCE

S ::= id = num


41/58




A ::= S EOF L ::= L ; SL ::= S

Grammar:

S ::= ( L )S ::= id = num

Input from lexer:

( L

yet to read

Parsing Table

REDUCE

L ::= S


42/58




A ::= S EOF L ::= L ; SL ::= S

Grammar:

S ::= ( L )S ::= id = num

Input from lexer:

( L ;

yet to read

Parsing Table

SHIFT


43/58




A ::= S EOF L ::= L ; SL ::= S

Grammar:

S ::= ( L )S ::= id = num

Input from lexer:

( L ; id = num

yet to read

Parsing Table

SHIFTSHIFTSHIFT


44/58




A ::= S EOF L ::= L ; SL ::= S

Grammar:

S ::= ( L )S ::= id = num

Input from lexer:

( L ; S

yet to read

Parsing Table

REDUCES ::= id = num


45/58




A ::= S EOF L ::= L ; SL ::= S

Grammar:

S ::= ( L )S ::= id = num

Input from lexer:

( L

yet to read

Parsing Table

REDUCES ::= L ; S


46/58




A ::= S EOF L ::= L ; SL ::= S

Grammar:

S ::= ( L )S ::= id = num

Input from lexer:

( L )

yet to read

Parsing Table

SHIFT


47/58




A ::= S EOF L ::= L ; SL ::= S

Grammar:

S ::= ( L )S ::= id = num

Input from lexer:

S

yet to read

Parsing Table

REDUCES ::= ( L )


48/58




A ::= S EOF L ::= L ; SL ::= S

Grammar:

S ::= ( L )S ::= id = num

Input from lexer:

A

Parsing Table

SHIFT

REDUCEA ::= S EOF

ACCEPT


49/58




A ::= S EOF L ::= L ; SL ::= S

Grammar:

S ::= ( L )S ::= id = num

Input from lexer:

A

Parsing Table

A successful parse! Is this grammar LL(1)?


50/58

Shift-reduce algorithm

Parser keeps track of position in current input (what input to read next)

a stack of terminal & non-terminal symbols representing theparse so far

Based on next input symbol & stack, parser tableindicates shift: push next input on to top of stack

reduce R:

top of stack should match RHS of rule

replace top of stack with LHS of rule error

accept (we shift EOF & can reduce what remains on stack tostart symbol)


51/58

Shift-reduce algorithm (a detail)

The parser summarizes the current parse state using aninteger the integer is actually a state in a finite automaton the current parse state can be computed by running the automaton over

the current parse stack

Revised algorithm: Based on next input symbol & the parsestate (as opposed to the entire stack), parser table indicates shift s:

push next input on to top of stack and move automaton into state s

reduce R & goto s: top of stack should match RHS of rule replace top of stack with LHS of rule move automaton into state s

error accept


52/58


???????? EOF

shift-reduce parsingGrammar:

Input from lexer:

????

Like LL parsing, shift-reduce parsing does not always work.What sort of grammar rules make shift-reduce parsing impossible?

????


53/58


???????? EOF

shift-reduce parsingGrammar:

Input from lexer:

????

Like LL parsing, shift-reduce parsing does not always work.

Shift-Reduce errors: cant decide whether to Shift or Reduce

Reduce-Reduce errors: cant decide whether to Reduce by R1 or R2

????


54/58


shift-reduce errors

A ::= S EOF

Grammar:

S ::= S + SS ::= S * SS ::= id

Input from lexer: ???????? EOF

????


55/58


id + id * id EOF

shift-reduce errors

A ::= S EOF

Grammar:

S ::= S + SS ::= S * SS ::= id

Input from lexer:

S + S

reduce by rule (S ::= S + S) or

shift the * ???

notice, this is an ambiguousgrammar we are always going toneed some mechanism for resolvingthe outstanding ambiguity before parsing


56/58


shift-reduce errors

A ::= S id EOF

Grammar:

E ::= E ; EE ::= id

Input from lexer: id ; id EOF

E ;

S ::= E ;

reduce by rule (S ::= E ;) or

shift the id

id ; id ; id EOF

input might be this,making shifting

correct

some unambiguousgrammars cant be

parsed by LR(1)parsers either


57/58


( id) EOF

reduce-reduce errors

A ::= S EOF

Grammar:

S ::= ( E )S ::= E

Input from lexer:

( E )

reduce by rule ( S ::= ( E ) ) or

reduce by rule ( E ::= ( E ) )

E ::= ( E )E ::= E + EE ::= id


58/58

Summary

Top-down Parsing simple to understand and implement

you can code it yourself using nullable, first, followsets

excellent for quick & dirty parsing jobs

Bottom-up Parsing more complex: uses stack & table

more powerful

Bonus: tools do the work for you ==> ML-Yacc but you need to understand how shift-reduce & reduce-

reduce errors can arise

5-parserconstruction

Documents