chapter 4 grammars and parsing prof chung. 1 2008/03/18

Chapter 4 Grammars and Parsing

Prof Chung.

12008/03/18

Outlines

2

Context-Free Grammars Errors in Context-Free Grammars Transforming Extended BNF Grammars Parsers and Recognizers Grammar Analysis Algorithms

Context-Free Grammars: (1)Concepts and Notation

A context-free grammar G = (Vt, Vn, S, P) A finite terminal vocabulary Vt

The token set produced by scanner

A finite set of nonterminal vocabulary Vn Intermediate symbols

3

Denote symbols in Vt : a,b,c

Denote symbols in Vn : A,B,C


A context-free grammar G = (Vt, Vn, S, P) A start symbol S Vn that starts all derivations

Also called goal symbol

P, a finite set of productions (rewriting rules) of the form AX1X2Xm

AVn, Xi VnVt, 1i m

Ais a valid production

4


Other notations Vocabulary V of G,

V= VnVt

L(G), the set of string s derivable from S Context-free language of grammar G

Notational conventions Denote symbols in V : U,V,W, Denote strings in V* : ,,, Denote strings in Vt

* : u,v,w,

5


Derivation One step derivation If A, then

A One or more steps derivation

Zero or more steps derivation

If S , then is said to be sentential form of the CFG SF(G) is the set of sentential forms of grammar G

L(G) = {x Vt*|Sx}

L(G)=SF(G)Vt*

6


Left-most derivation top-down parsers 【

lm,

lm,lm 】 E.g. of leftmost derivation of F(V+V)

Right-most derivation (canonical derivation) Buttom-up parsers 【

rm,

rm,rm 】 E.g. of rightmost derivation of F(V+V)

7

Elm Prefix(E)

lm F(E)

lm F(V Tail)

lm F(V+E)

lm F(V+V Tail)

lm F(V+V)

EPrefix(E)EV TailPrefixFPrefixTailTail

G0

Erm Prefix(E)

rm Prefix(V Tail)

rm Prefix(V+E)

rm Prefix(V+V Tail)

rm Prefix(V+V)

rm F(V+V)

Same # of steps, but different orderSame # of steps, but different order


A parse tree rooted by the start symbol Its leaves are grammar symbols or

A phase at next page…

8


Phrase A phrase of a sentential form

is a sequence of symbols descended from a single non-terminal in the parse tree

Simple or prime phrase is a phrase that contains no smaller phrase

Simple phrase is a sequence of symbols directly derived from a non-terminal

Handle The handle of a sentential form

is the left-most simple phrase Simple phrases cannot overlap, so “leftmost” is unambiguous Handles are important

because they represent individual derivation steps, which can be recognized by various parsing techniques.

9

Sentential Form : F(V Tail)Simple Phrases : F,VTailHandle : F

Example


Regular grammars Limited to productions of the form The handle of a sentential form is the left-most simple phrase

Context-sensitive grammars Require that nonterminals be rewritten only when they appear in a particular

context.

Type-0 grammars Still more general and allow arbitrary patterns to be rewritten.

Context-sensitive and Type-0 grammars are more powerful than CFGs, but less useful.

This is that efficient parsers for these extended grammar classes do not exist, Without a parser there is no way to use a grammar definition to drive a compiler. CFGs represent a nice balance between generality and practicality.

10

AaBC

Rule ： α A β α δ β

Rule ： α β

Errors in Context-Free Grammars (1) Ambiguous

Grammars that allow different parse trees for the same terminal string

11

Equal ?

Errors in Context-Free Grammars (2) “Useless nonterminals ”

Is unreachable or derive no terminal string are termed useless.

Can be safely removed from a grammar without changing the defined by the

grammar. A grammar containing useless nonterminals is said to be

nonreduced. After useless nonterminals are removed, the grammar is

reduced.

Example : In G1, nonterminal C cannot be reached from S.

In G1, nonterminal B derives no terminal string.

12

S A | BA aB BbC c

G1

S A A a

G1Reduced

It is impossible to do : Decide whether a given CFG is ambiguous

Analyze Wrong language Production Exception No general comparison algorithm applicable to all CFGs

Errors in Context-Free Grammars (3)

13

Transforming Extened BNF Grammars Extended BNF BNF

Extended BNF allows Square bracket [ ] Optional list { }

for (each production P=A α [X1…Xn] β) {Create a new nonterminal, N.Replace production P with p’ = A α N βAdd the productions: N X1 … Xn and N

}

for (each production Q=B γ{Y1…Ym} δ) {Create a new nonterminal, M.Replace production Q with Q’ = B γ M δAdd the productions: M Y1 … Ym M and M

}

Algorithm to Transform Extended BNF Grammars into Standard Form

14

<id list> ID { , ID }

<id list>’ ID MM , ID M M

<id list> ID [ ID1 ID2 ]

<id list>’ ID NN , ID1 ID2 N

Parsers and Recognizers (1) Recognizer

An algorithm that does boolean-valued test “Is this input syntactically valid?

Parser Answers more general questions

Is this input valid? And, if it is, what is its structure (parse tree)?

15

Parsers and Recognizers (2) Two general approaches to parsing

Top-down parser Expanding the parse tree (via predictions) in a depth-first

manner Preorder traversal of the parse tree Predictive in nature LL (lm)

Buttom-down parser Beginning at its bottom (the leaves of the tree, which are

terminal symbols) and determining the productions used to generate the leaves

Post-order traversal of the parse tree LR (rm)

16

Parsers and Recognizers (3) Naming of parsing techniques

• Top-down LL

• Bottom-up LR

17

The way to parse token sequence

L: Leftmost R: Righmost

Grammar Analysis Algorithms (1) Goal of this section

Discuss a number of important analysis algorithms for Grammars

18

typedef struct gram {　　 symbol terminals[NUM_TERMINALS];　　 symbol nonterminals[NUM_NONTERMINIALS]; 　　　　　　　 symbol start_symbol;　　 int 　　　 num-productions;　　 struct prod {　　　　 symbol lhs;　　　　 int 　　　 rhs_length;　　　　 symbol rhs[MAX_RHS_LENGTH];　　 } produtcions[NUM_PRODUCTIONS];　　 symbol vocabulary[VOCABULARY];} grammar;

typedef struct prod production;typedef symbol terminal;typedef symbol nonterminal;

Grammar Analysis Algorithms (2) What nonterminals can derive ?

Ex : A BCD BC B An iterative marking algorithm

19

Marked_vocabulary mark_lambda(const grammar g) {

Step 1 : Initialize

　　 Step 2 : Nonterminals that derive trivially are marked.

Step 3 : Nonterminals requiring a parse tree height of two are found.

Step 4 : Continue finding nonterminals requiring parse trees of ever-increasing 　　 height, until no more nonterminals can be marked as deriving .}

Source Code in Next page.

【 Cause & GOAL 】Determining if a nonterminal can derive is not entirely trivial because the derivation may take more than one step.

Grammar Analysis Algorithms (3)typedef short boolean;typedef boolean marked_vocabulary[VOCABULARY];/* Mark those vocabulary symbols found to derive (directly or indirectly). */

Marked_vocabulary mark_lambda(const grammar g){

…….…………………………………………………………………….

for(v=0; v<VOCABBULARY; v++)derives_lambda[v] = FALSE;/* initially, nothing is marked */do {changes = FALSE;for (i=0;i<g.num_productions; i++) {p=g.productions[i];if( !derives_lambda[p.lhs]) {if(p.rhs_length ==0) {/*derives directly */changes = derives_lambda[p.lhs] = TRUE;continue;}/* does each part of RHS derive ? */rhs_derives_lambda = derives_lambda[p.rhs[0]];for(j=1;j<p.rhs_length;j++)rhs_derives_lambda = rhs_derives_lambda && deroves_lambda[p.rhs[j]];if(rhs_derives_lambda) {changes =TRUE; derives_lambda[p.lhs] =TRUE;

}

}}}while (changes);return derives_lambda;

}

20

static marked_vocabulary derives_lambda;boolean changes;/* any changes during last iteration? */boolean rhs_derives_lambda;/* does the RHS derive ? */symbol v; /* a word in vocabulary */production p; /* a production in the grammar */int i,j; /* loop variables */

More Example at next page…

Grammar Analysis Algorithms (4) Definition of C data structures and

subroutines first_set[X]

contains terminal symbols and X is any single vocabulary symbol All the terminal symbols that can begin a sentential form

derivable from If is the right-hand side of a production, then First() contains

terminal symbols that begin strings derivable from

follow_set[A] contains terminal symbols and in some sentential form A is a nonterminal symbol

21

Follow(A)={aVt|S* Aa } {if S + A then {} else }

First()={aVt| * a}{if * then {} else }


Grammar Analysis Algorithms (5)typedef set_of_terminal_or_lambda termsets;termset follow_set[NUM_NONTERMINAL];termset first_set[SYMBOL];marked vocabulary dervies_lambda = mark_lambda(g);/* mark_lambda(g) as defined above */

termset compute first (string_of_symbols alpha){

int i,k;termset result;

k= length (alpha);if( k==0 )

result = SET_OF ();else {

result = first_set[alpha[0]];for (i=1; i<k && ∈first_set[alpha[i-1]];i++)

result = result ∪( first_set[alpha[i]] –SET_OF());if(i==k && ∈first_set[alpha[k-1]]) result = result ∪ SET_OF();

}return result;

}

22

【 Cause】： For an arbitrary string , we can’t have the First and Follow sets pre-computed, so we use this function that return the set of terminals defined by First(). 【 Action】： If happens to be exactly one symbol long. 【 GOAL 】： This function will simply return first_set[].

It is a subroutine of fill_first_set()


G0


Grammar Analysis Algorithms (6)Extern grammar g;

void fill_first_set(void){

…………… (0)…………… (1)…………… (2)…………… (3)

}

23


G0

Step First Set

E Prefix Tail ( ) V F +

(1)First Loop

for(i=0; i<NUM_NONTERMINAL; i++) {　　 A=g.nonterminals[i];　　 if(derives_lambda[A])　　　　 first_set[A] = SET_OF();　　 else　　　　 first_set[A]= ;}

Nonterminal A;terminal a;production p;boolean changes;Int I, j;

01



…………… (0)…………… (1)…………… (2)…………… (3)

}

24


G0

Step First Set


(1)First Loop



01

{}



…………… (0)…………… (1)…………… (2)…………… (3)

}

25


G0

Step First Set


(1)First Loop



01

{} {}

Step First Set


(1)First Loop

(2)Second (nested) Loop

for(i=0;i<NUM_TERMINAL; i++) {　　 a=g.terminals[i];　　 first_set[a] = SET_OF (a);　　 for(j=0;j<NUM_NONTERMINAL; j++) {　　　　 A=g.nonterminals[j];　　　　 if(there exists a production A aβ)　　　　　　 first_set[A]=first_set[A] SET_OF(a);∪　　 }}



…………… (0)…………… (1)…………… (2)…………… (3)

}

26


G0



01

{}

2

{}

{} {} {(}

Step First Set


(1)First Loop





…………… (0)…………… (1)…………… (2)…………… (3)

}

27


G0



01

{}

2

{}

{} {} {(}

Step First Set


(1)First Loop





…………… (0)…………… (1)…………… (2)…………… (3)

}

28


G0



01

{}

2

{}

{} {} {(}

Step First Set


(1)First Loop





…………… (0)…………… (1)…………… (2)…………… (3)

}

29


G0



01

{}

2

{}

{} {} {)}

{(}

Step First Set


(1)First Loop





…………… (0)…………… (1)…………… (2)…………… (3)

}

30


G0



01

{}

2

{}

{} {} {)}

{(}

Step First Set


(1)First Loop





…………… (0)…………… (1)…………… (2)…………… (3)

}

31


G0



01

{}

2

{}

{} {} {)}

{(}

Step First Set


(1)First Loop





…………… (0)…………… (1)…………… (2)…………… (3)

}

32


G0



01

{}

2

{}

{} {} {V}

{(} {)}{V}

Step First Set


(1)First Loop





…………… (0)…………… (1)…………… (2)…………… (3)

}

33


G0



01

{}

2

{}

{} {} {V}

{(}{V} {)}

Step First Set


(1)First Loop





…………… (0)…………… (1)…………… (2)…………… (3)

}

34


G0



01

{}

2

{}

{} {} {V}

{(}{V} {)}

Step First Set


(1)First Loop





…………… (0)…………… (1)…………… (2)…………… (3)

}

35


G0



01

{}

2

{}

{} {} {F}

{(}{V} {)} {V}

Step First Set


(1)First Loop





…………… (0)…………… (1)…………… (2)…………… (3)

}

36


G0



01

{}

2

{}

{} {} {F}

{(}{V} {)} {V}{F,}

Step First Set


(1)First Loop





…………… (0)…………… (1)…………… (2)…………… (3)

}

37


G0



01

{}

2

{}

{F,} {} {F}

{(}{V} {)} {V}

Step First Set


(1)First Loop





…………… (0)…………… (1)…………… (2)…………… (3)

}

38


G0



01

{}

2

{}

{F,} {} {+}

{(}{V} {)} {V} {F}

Step First Set


(1)First Loop





…………… (0)…………… (1)…………… (2)…………… (3)

}

39


G0



01

{}

2

{}

{F,} {} {+}

{(}{V} {)} {V} {F}

Step First Set


(1)First Loop





…………… (0)…………… (1)…………… (2)…………… (3)

}

40


G0



01

{}

2

{}

{F,} {} {+}

{(}{V} {)} {V} {F}{+, }

do {　　 changes =FALSE;　　 for (i=0;i<g.num_productions; i++) {　　　　 p=g.productions[i];　　　　 first_set[p.lhs] = first_set[p.lhs] compute_first(p.rhs);∪　　　　 if( first_set changed )　　　　　　 changes = TRUE;　　 }} while (changes);

Step First Set

E Prefix Tail ( ) V F +(1)First Loop


(3)Third Loop, production 1




…………… (0)…………… (1)…………… (2)…………… (3)

}

41


G0



01

{}

2

{}

{F,} {+,} {+}

{(}{V} {)} {V} {F}

3

{V}{F,V,(} {F,} {(} {)} {V} {F} {+}

{+,}


Step First Set







…………… (0)…………… (1)…………… (2)…………… (3)

}

42


G0



01

{}

2

{}

{F,} {+}

{(}{V} {)} {V} {F}

3

{F,V,(} {F,} {(} {)} {V} {F} {+}

{+,}

{+,}


Step First Set







…………… (0)…………… (1)…………… (2)…………… (3)

}

43


G0



01

{}

2

{}

{F,} {+}

{(}{V} {)} {V} {F}

3

{F,V,(} {F,} {(} {)} {V} {F} {+}

{+,}

{+,}


Step First Set







…………… (0)…………… (1)…………… (2)…………… (3)

}

44


G0



01

{}

2

{}

{F,} {+}

{(}{V} {)} {V} {F}

3

{F,V,(} {F,} {(} {)} {V} {F} {+}

{+,}

{+,}


Step First Set







…………… (0)…………… (1)…………… (2)…………… (3)

}

45


G0



01

{}

2

{}

{F,} {+}

{(}{V} {)} {V} {F}

3

{F,V,(} {F,} {(} {)} {V} {F} {+}

{+,}

{+,}


Step First Set







…………… (0)…………… (1)…………… (2)…………… (3)

}

46


G0



01

{}

2

{}

{F,} {+}

{(}{V} {)} {V} {F}

3

{F,V,(} {F,} {(} {)} {V} {F} {+}

{+,}

{+,}


Step First Set







…………… (0)…………… (1)…………… (2)…………… (3)

}

47


G0



01

{}

2

{}

{F,} {+,} {+}

{(}{V} {)} {V} {F}

3

{F,V,(} {F,} {(} {)} {V} {F} {+}

{+,}


Step First Set







…………… (0)…………… (1)…………… (2)…………… (3)

}

48


G0



01

{}

2

{}

{F,} {+}

{(}{V} {)} {V} {F}

3

{F,V,(} {F,} {(} {)} {V} {F} {+}

{+,}

{+,}


Step First Set







…………… (0)…………… (1)…………… (2)…………… (3)

}

49


G0



01

{}

2

{}

{F,} {+}

{V} {)} {V} {F}

3

{F,V,(} {F,} {(} {)} {V} {F} {+}

{+,}

{+,}

{(}


Step First Set







…………… (0)…………… (1)…………… (2)…………… (3)

}

50


G0



01

{}

2

{}

{F,} {+}

{(}{V} {)} {V} {F}

3

{F,V,(} {F,} {(} {)} {V} {F} {+}

{+,}

{+,}


Step First Set







…………… (0)…………… (1)…………… (2)…………… (3)

}

51


G0



01

{}

2

{}

{F,} {+}

{V} {)} {V} {F}

3

{F,V,(} {F,} {(} {)} {V} {F} {+}

{+,}

{+,}

{(}


Step First Set







…………… (0)…………… (1)…………… (2)…………… (3)

}

52


G0



01

{}

2

{}

{F,} {+}

{V} {)} {V} {F}

3

{F,V,(} {F,} {(} {)} {V} {F} {+}

{+,}

{+,}

{(}



…………… (0)…………… (1)…………… (2)…………… (3)

}

53


G0

Step First Set

E Prefix Tail ( ) V F +(1)First Loop {} {}(2)Second (nested) Loop {V} {F,

}{+, }

{(}

{)}

{V}

{F}

{+}


{V, F, (}

{F, }

{+, }

{(}

{)}

{V}

{F}

{+}



for(i=0;i<NUM_TERMINAL; i++) {　　 a=g.terminals[i];　　 first_set[a] = SET_OF (a);　　 for(j=0;j<NUM_NONTERMINAL; j++) {　　　　 A=g.nonterminals[j];　　　　 if(there exists a production A a β)　　　　　　 first_set[A]=first_set[A] SET_OF(a);∪　　 }}


0

2 3

1



Simple Method}

54


G0

Look

First Set

E Prefix

Tail ( ) V F +

(1)

(2)

(3)

Simple Algorithm

{(} {)} {V} {F} {+}

{Prefix, V} {F, } {+, }

{F, V,(} {F, } {+, }

{F, V,(} {F, } {+, }

More Operation at next page…



Simple Method}

55


G0

Look

First Set

E Prefix

Tail ( ) V F +

(1)

Simple Algorithm

{Prefix, V} {F, } {+, }



Simple Method}

56


G0

Step

First Set

E Prefix

Tail ( ) V F +

(1)

(2)

Simple Algorithm

{Prefix, V} {F, } {+, }

{F, V, (} {F, } {+, }

Translate Non-terminal in First Set “Prefix” can be replaced by “F” or “” So….

EPrefix(E)TranslateEF(E)E(E)But In “E(E)”, the is empty…So catch more one…E(E)

The first set of “Prefix” and “Tail” do not have non-terminals!



Simple Method}

57


G0

Look

First Set

E Prefix

Tail ( ) V F +

(1)

(2)

(3)

Simple Algorithm

{(} {)} {V} {F} {+}

{Prefix, V} {F, } {+, }

{F, V,(} {F, } {+, }

{F, V,(} {F, } {+, }

More Operation at next page…

The first set of terminals are itself


void fill_follow_set(void){

…………… (0)…………… (1)…………… (2)

}

58

for(i=0; i<NUM_NOTERMINAL; i++) {　　 A = g.nonterminals[i];　　 follow_set[A] = ;}


G0

Step Follow Set

E Prefix Tail

(1) Initialization

noterminal A, B;int i;boolean changes;

0

1

{}



…………… (0)…………… (1)…………… (2)

}

59



G0

Step Follow Set

E Prefix Tail

(1) Initialization


0

1

{}



…………… (0)…………… (1)…………… (2)

}

60



G0

Step Follow Set

E Prefix Tail

(1) Initialization {}


0

1

do {　　 changes = FALSE;　　 for (each production A αBβ) {/* I.e. for each production and each occurrence of a nonterminal in its right-hand side.*/

　　　　 follow_set[B] = follow_set[B] (compute_first(∪ β) – SET_OF ());　　　　 if( ∈ compute_first(β))�　　　　　　 follow_set[B] = follow_set[B] follow_set[A];∪　　　　 if(follow_set[B] changed)　　　　　　 changes = TRUE;　　 }}while (changes);



…………… (0)…………… (1)…………… (2)

}

61



G0

Step Follow Set

E Prefix Tail(1) Initialization {} (2)Process Prefix in production 1

{}


0

1 2

{(}

changes = FALSETRUE





…………… (0)…………… (1)…………… (2)

}

62



G0

Step Follow Set


{} {(}

(2)Process E in production 1

{(}


0

1 2

changes = FALSETRUE

{, )}{}





…………… (0)…………… (1)…………… (2)

}

63



G0

Step Follow Set


{} {(}

(2)Process E in production 1

{(}

(4)Process Tail in production 2

{, )}

{(}


0

1 2

changes = FALSETRUE

{, )}{}

{, )}



…………… (0)…………… (1)…………… (2)

}

64

for(i=0; i<NUM_NOTERMINAL; i++) {　　 A = g,nonterminals[i];　　 follow_set[A] = ;}

do {　　 changes = FALSE;　　 for (each production A αBβ) {/* I.e. for each production and each occurrence of a nonterminal in its right-hand side.*/　　　　 follow_set[B] = follow_set[B] (compute_first(∪ β) – SET_OF ());　　　　 if( ∈ compute_first(β))�　　　　　　 follow_set[B] = follow_set[B] follow_set[A];∪　　　　 if(follow_set[B] changed)　　　　　　 changes = TRUE;　　 }}while (changes);


G0

Step Follow Set


{} {(} (3)Process E in production 1

{, )}

{(} (4)Process Tail in production 2

{, )}

{(} {, )}


0

1 2

More examples (1)

65

S aSeS BB bBeB CC cCeC d

The execution of fill_first_set()

Step First Set

S B C a b c d e

Step Follow Set

S B C

(1)First Loop


{a} {b} {c, d}

{a}

{b}

{c}

{d}

{e}(3)Third Loop,

production 2{a, b} {b} {c,

d}{a}

{b}

{c}

{d}

{e}(4)Third Loop,

production 4{a, b} {b, c,

d}{c, d}

{a}

{b}

{c}

{d}

{e}(5)Third Loop,

production 2{a, b, c, d}

{b, c, d}

{c, d}

{a}

{b}

{c}

{d}

{e}

(1)Initialization {}

(2)Process S in production 1

{e, }

(3)Process B in production 2

{e, }

{e, }


No Changes

(5)Process C in production 4

{e, }

{e, }

{e, }

(6)Process C in production 5

No Changes

More examples (2)

Step First Set

S A B a b c

66

S ABcA aA B bB

Step Follow Set

S A B

(1)First Loop {} {}


{a, }

{b, }

{a}

{b}

{c}(3)Third Loop,

production 1{a, b, c}

{a, }

{b, }

{a}

{b}

{c}

(1)Initialization {}

(2)Process A in production 1

{} {b, c}


{} {b, c} {c}

chapter 4 grammars and parsing prof chung. 1 2008/03/18

Documents

fexamplecontextfree

contextfree grammarserrors

contextfree grammar

particular context

rightmost derivation

parsers rm

parsers lm

grammar symbols