slide1 chapter 4 lexical and syntax analysis. slide2 outlines: in this chapter a major topics will...

slideslide11

Chapter 4Chapter 4Lexical and Syntax AnalysisLexical and Syntax Analysis

slideslide22

OutLines:OutLines:In this chapter a major topics will be In this chapter a major topics will be

discussed :discussed :• Introduction to lexical analysis, Introduction to lexical analysis,

including simple example.including simple example.• Discussing of parsing problem .Discussing of parsing problem .• The primary approach to parsing.The primary approach to parsing.• Recursive-descent implementation Recursive-descent implementation

technique for LL parser.technique for LL parser.• Discussing bottom-up parsing and LR Discussing bottom-up parsing and LR

parsing algorithm . parsing algorithm .

Lexical and Syntax Analysis: Chapter 4

IntroductionIntroduction

Three different approaches to Three different approaches to implementing programming implementing programming languages :languages :

1.1. Compilation.Compilation.

2.2. Pure interpretation.Pure interpretation.

3.3. Hybrid.Hybrid.


slideslide44

Compilation ApproachCompilation Approach

• Uses a program called a compiler, Uses a program called a compiler, which translates program written in which translates program written in high-level programming languages high-level programming languages into machine code .into machine code .


slideslide55

Interpretation ApproachInterpretation Approach

• Perform no translate .Perform no translate .

• Program are interpreted in their Program are interpreted in their original form by software original form by software interpreter .interpreter .

• Used for smaller system in which Used for smaller system in which execution is not critical. execution is not critical.


slideslide66

Hybrid ApproachHybrid Approach

• Translate program in high-level into Translate program in high-level into intermediate forms which are intermediate forms which are interpreter .interpreter .

• It takes much slower execution than It takes much slower execution than compiler system.compiler system.


slideslide77

Compilers task separate into two Compilers task separate into two parts:parts:

1.1. lexical analysis: deal with small-lexical analysis: deal with small-scale constructs (names , numeric scale constructs (names , numeric literals)literals)

2.2. Syntax analysis: deal with large-Syntax analysis: deal with large-scale constructs ( expressions , scale constructs ( expressions , statement , program unit )statement , program unit )


slideslide88

Reasons of separating lexical analysis Reasons of separating lexical analysis from Syntax analysisfrom Syntax analysis : :

1.1. SimplicitySimplicity

2.2. EfficiencyEfficiency

3.3. PortabilityPortability


slideslide99

Lexical AnalysisLexical Analysis

• Lexical Analysis is a part of syntax Lexical Analysis is a part of syntax analysisanalysis

• Lexical Analyzer collect characters Lexical Analyzer collect characters into logical grouping( lexemes) and into logical grouping( lexemes) and assigns internal codes ( tokens )assigns internal codes ( tokens )


slideslide1010

Consider following example of assign Consider following example of assign statement:statement:• Result = oldsum – value / 100 ;Result = oldsum – value / 100 ;

TokenToken lexemelexeme

IDENTIDENT ResultResult

ASSIGN-OPASSIGN-OP ==

IDENTIDENT oldsumoldsum

SUBSTRACT-OPSUBSTRACT-OP ––

IDENTIDENT valuevalue

DIVISION-OPDIVISION-OP //

INT-LITINT-LIT 100100

SEMICOLONSEMICOLON ;;


slideslide1111

Lexical Analysis process :Lexical Analysis process :

1.1. Skipping comments and blank out Skipping comments and blank out side lexemes .side lexemes .

2.2. Insert lexemes into the symbol Insert lexemes into the symbol table.table.

3.3. Detect syntactic errors in tokens .Detect syntactic errors in tokens .

4.4. Report errors to the user . Report errors to the user .


slideslide1212

Approaches to building a lexical Approaches to building a lexical analyzer:analyzer:

1.1. Write a formal description of the token Write a formal description of the token patterns using descriptive language patterns using descriptive language related to RE .related to RE .

2.2. Design state transition diagram that Design state transition diagram that describes the token patterns of language describes the token patterns of language and write program that implement and write program that implement diagramdiagram

3.3. Design state transition diagram that Design state transition diagram that describes the token patterns of language describes the token patterns of language and hand-construct a table-driven and hand-construct a table-driven implementation of state diagram.implementation of state diagram.


slideslide1313

State diagramState diagram include state, include state, transitions for each, and every token transitions for each, and every token pattern.pattern.

• Results very large and complex Results very large and complex diagramdiagram . .

Therefore aTherefore a lexical analyzerlexical analyzer need to need to recognizes only:recognizes only:

• NameName

• Reserved wordsReserved words

• Integer literalsInteger literals


slideslide1414

• Lexical analyzer useLexical analyzer use aa tabletable of of reserved words to determine which reserved words to determine which names are reserved words . names are reserved words .

• Can build a much more compact state Can build a much more compact state diagram.diagram.

• Use a single transition on any Use a single transition on any character.character.

• we define some sub programs for the we define some sub programs for the common tasks .common tasks .


slideslide1515

/ * lex – a simple lexical analyzer *// * lex – a simple lexical analyzer */

Int lex() {Int lex() {getchar();getchar();Switch ( charclass ) {Switch ( charclass ) {/* parse identifiers and reserved word *//* parse identifiers and reserved word */

Case LETTER:Case LETTER:addchar ();addchar ();getchar();getchar();While (charclass == LETTER II charclass == DIGIT) {While (charclass == LETTER II charclass == DIGIT) {Addchar();Addchar();Getchar();Getchar();}}Return lookup (lexeme);Return lookup (lexeme);Break;Break;


slideslide1616

/* parse integer literals*//* parse integer literals*/Case DIGIT:Case DIGIT:addchar ();addchar ();getchar();getchar();While (charclass ==DIGIT ){While (charclass ==DIGIT ){addchar ();addchar ();getchar();getchar();}}Return INT_LIT;Return INT_LIT;Break;Break;} /*end of switch */} /*end of switch */} /* end of function lex*/ } /* end of function lex*/


slideslide1717

A state diagram to recognize names, reserved A state diagram to recognize names, reserved word, and integer literals:word, and integer literals:


slideslide1818

Parsing problemParsing problem

• Syntax analysis is calledSyntax analysis is called parsingparsing..

• Two goals of Syntax analysis:Two goals of Syntax analysis:

• 1.1. Must check the input program to Must check the input program to determine it is syntactically correct.determine it is syntactically correct.

2.2. Produce either a complete parse tree Produce either a complete parse tree or at least trace the structure of the or at least trace the structure of the complete parse tree.complete parse tree.

• Parsers for pl construct parse trees for Parsers for pl construct parse trees for given programs. given programs.


slideslide1919

Syntax analyzer \parsers/ based in a Syntax analyzer \parsers/ based in a formal description of syntax of formal description of syntax of programming calls context-free programming calls context-free grammars, or BNF. grammars, or BNF.


slideslide2020

Advantages for using BNFAdvantages for using BNF : :

1.1. BNF description are clear and BNF description are clear and concise.concise.

2.2. Can be used as the direct basis for Can be used as the direct basis for the syntax analysis.the syntax analysis.

3.3. The implementations are easy to The implementations are easy to maintain .maintain .


slideslide2121

Notational convention for grammar Notational convention for grammar symbols:symbols:

1. 1. Terminal Symbols : lowercase letters the Terminal Symbols : lowercase letters the beginning of the alphabet ( a,b,c,……..)beginning of the alphabet ( a,b,c,……..)

2.2. Nonterminal Symbols: uppercase letters the Nonterminal Symbols: uppercase letters the beginning of the alphabet (A,B,C, …..) beginning of the alphabet (A,B,C, …..)

3.3. Terminal or Nonterminal symbols : Terminal or Nonterminal symbols : uppercase letters the end of the alphabet uppercase letters the end of the alphabet ( W,X,Y,Z)( W,X,Y,Z)

4. 4. String of Terminal - lowercase letters the end String of Terminal - lowercase letters the end of the alphabet (w,x,y,z)of the alphabet (w,x,y,z)

5.5. Mixed strings (terminal or/and nonterminal) Mixed strings (terminal or/and nonterminal)


slideslide2222

Categories Of Parsing Categories Of Parsing Algorithms:Algorithms:

1.1. Top-down: the tree is built from the Top-down: the tree is built from the root downward to the leaves.root downward to the leaves.

1.1. Bottom-up: the tree is built from the Bottom-up: the tree is built from the leaves upward to the root.leaves upward to the root.


slideslide2323

Complexity Of ParsingComplexity Of Parsing

• Parsing algorithms for grammar are Parsing algorithms for grammar are complex and inefficient.complex and inefficient.

• Complexity in algorithms is O( nComplexity in algorithms is O( n33), Means ), Means amount of time they take to order of the amount of time they take to order of the cube length string to be parsed. cube length string to be parsed.

• Algorithms must back up and reparse part Algorithms must back up and reparse part

of sentence being analyzed. of sentence being analyzed. • ReparsingReparsing :is required when the parser has :is required when the parser has

made a mistake in the parsing process. made a mistake in the parsing process. • backing upbacking up: is requires that part of the : is requires that part of the

parse tree being constructed (trace) must parse tree being constructed (trace) must be dismantled and rebuilt. be dismantled and rebuilt.


slideslide2424

Top down parserTop down parser

• Top-down parsers check to see if a Top-down parsers check to see if a string can be generated by a grammar string can be generated by a grammar by creating a parse tree starting from by creating a parse tree starting from the initial symbol and working downthe initial symbol and working down

• The parser uses symbol-look-ahead and The parser uses symbol-look-ahead and this approach without backtrackingthis approach without backtracking

• LL parsersLL parsers are examples of top-down are examples of top-down parsers.parsers.


http://en.wikipedia.org/wiki/LL_parser

slideslide2525

Recursive – Decent ParsingRecursive – Decent Parsing

• Recursive – Decent Parser consist of Recursive – Decent Parser consist of many recursive subprograms and many recursive subprograms and produce a parse tree in top-down produce a parse tree in top-down (descending) order(descending) order

• The syntax of These structure The syntax of These structure described with recursive grammar rule described with recursive grammar rule

• EBNF is suitable for Recursive – EBNF is suitable for Recursive – Decent Parser Decent Parser


slideslide2626

Consider the following EBNF Consider the following EBNF description of arithmetic expression description of arithmetic expression

• <expr> <term> { ( + l - ) <term> }<expr> <term> { ( + l - ) <term> }• <term> <factor>{ ( * l / ) <factor>}<term> <factor>{ ( * l / ) <factor>}• <factor> id l ( <expr> )<factor> id l ( <expr> )

• This grammar must ensure that code This grammar must ensure that code generation process( syntax analysis) generation process( syntax analysis) Produce code that adheres to the Produce code that adheres to the associativety rule of language associativety rule of language


slideslide2727

The LL Grammar ClassThe LL Grammar Class

• A A LL parserLL parser is a is a top-downtop-down parserparser for for a subset of the a subset of the context-free grammarscontext-free grammars. It parses the . It parses the input from input from LLeft to right, and eft to right, and constructs a constructs a LLeftmost derivationeftmost derivation of of the sentence the sentence


http://en.wikipedia.org/wiki/Top-down_parsing

http://en.wikipedia.org/wiki/Parser

http://en.wikipedia.org/wiki/Context-free_grammar

http://en.wikipedia.org/wiki/Context-free_grammar#Derivations_and_syntax_trees

http://en.wikipedia.org/wiki/Context-free_grammar#Derivations_and_syntax_trees

slideslide2828

Left RecursionLeft Recursion

• In recursive descent parsing, it is In recursive descent parsing, it is possible to implement a procedure possible to implement a procedure that may cause an infinite loop if the that may cause an infinite loop if the grammar that we called left grammar that we called left recursionrecursion


slideslide2929

Left Recursion Left Recursion

• Consider the following rule:Consider the following rule:

• A A + b {direct}A A + b {direct}

• A B a A {indirect}A B a A {indirect}

B A bB A b

• The Algorithm that remove both {direct, The Algorithm that remove both {direct, indirect} not covered here indirect} not covered here


slideslide3030

Pairwise disjoint ness test Pairwise disjoint ness test

• Test requires the ability to compute a Test requires the ability to compute a set of RHSs of a given nonterminal set of RHSs of a given nonterminal symbol in a grammarsymbol in a grammar

• If a nonterminal has more than RHSs If a nonterminal has more than RHSs the first terminal symbol must be the first terminal symbol must be unique. unique.


slideslide3131

Consider the following rule:Consider the following rule:

• A aB l bAb l cA aB l bAb l c• The FIRST set for RHSs are {a}, {b}, The FIRST set for RHSs are {a}, {b},

{c} which are clearly disjoint ,which {c} which are clearly disjoint ,which pass {Pairwise disjoint ness test} pass {Pairwise disjoint ness test}

• A aB l albA aB l alb• The FIRST set for RHSs are {a}, {a}The FIRST set for RHSs are {a}, {a} which are clearly not disjoint .which are clearly not disjoint .


slideslide3232

Left factoringLeft factoring

• A process that solve the problem of A process that solve the problem of left recursion and don’t pass “ pair left recursion and don’t pass “ pair wise disjoint ness test [ both RHS wise disjoint ness test [ both RHS begin with the same terminal]begin with the same terminal]

• We will take a look at left factoringWe will take a look at left factoring


slideslide3333

Consider the rule:Consider the rule:

<variable> ident l ident [<expr>]<variable> ident l ident [<expr>]

• { this rule don’t pass pair wise { this rule don’t pass pair wise disjoitness}disjoitness}

• We use left factoring to solve this We use left factoring to solve this problem. Tow rules can be replaced by :problem. Tow rules can be replaced by :

<variable> ident <variable> ident <new><new>

<new><new> εε l l [<expr>][<expr>]


slideslide3434

Bottom- Up Parsing Bottom- Up Parsing

• check to see a string can be generated check to see a string can be generated from a grammar by creating a parse tree from a grammar by creating a parse tree from the leaves, and working up.from the leaves, and working up.

• Left recursive acceptable to bottom-up Left recursive acceptable to bottom-up parsers parsers

• bottom-up parsers don’t include met bottom-up parsers don’t include met symbols symbols

• the process of bottom-up parsers produce the process of bottom-up parsers produce the reverse of a right most derivationthe reverse of a right most derivation

• LR parsersLR parsers are examples of bottom-up are examples of bottom-up parsers.parsers.


http://en.wikipedia.org/wiki/LR_parser

slideslide3535

Consider the following grammar :Consider the following grammar :

• E E + T l TE E + T l T

• T T * F l FT T * F l F

• F { E } l idF { E } l id


slideslide3636

• E => E + E => E + TT

=> E + T * => E + T * FF

=> E + T * => E + T * idid

=> E + => E + FF * id * id

=> E + => E + idid * id * id

=> => TT + id * id + id * id

=> => FF + id * id + id * id

=> => idid + id * id + id * id


slideslide3737

Shift-Reduce algorithmShift-Reduce algorithm

• Its another name of “ bottom up Its another name of “ bottom up parsers “ and specify two common parsers “ and specify two common action :action :

1.1. Shift : moves the next input token Shift : moves the next input token onto the parser’s stack.onto the parser’s stack.

2.2. Reduce : replace RHS {the handle} on Reduce : replace RHS {the handle} on top of the parser stack by its top of the parser stack by its corresponding LHScorresponding LHS


slideslide3838

LR Parsers LR Parsers

• Its an algorithm which most Its an algorithm which most commonly used bottom-up parsing commonly used bottom-up parsing approach for programming approach for programming languages .languages .

• It use parse stack, which contains It use parse stack, which contains grammar symbol and state symbol .grammar symbol and state symbol .


slideslide3939

Advantages for LR parsers :Advantages for LR parsers :

1.1. they work for nearly all grammars they work for nearly all grammars that describe programming languages that describe programming languages ..

2.2. they work on a larger class of they work on a larger class of grammars than other bottom-up grammars than other bottom-up algorithm and in the same efficient .algorithm and in the same efficient .

3.3. They can detect syntax errors as soon They can detect syntax errors as soon as its possible.as its possible.

4.4. The LR class of grammar Is a super The LR class of grammar Is a super set of the class parsable by LL parsersset of the class parsable by LL parsers


slideslide4040

Consider the following Consider the following grammargrammar1.1. E E + TE E + T

2.2. E TE T

3.3. T T * FT T * F

4.4. T FT F

5.5. F ( E )F ( E )

6.6. F idF id

slideslide4141

Action GOTO

state id + * ﴾ ﴿ $ E T F

0 S5 S4 1 2 3

1 S6 accept

2 R2 S7 R2 R2

3 R4 R4 R4 R4

4 S5 S4 8 2 3

5 R6 R6 R6 R6

6 S5 S4 9 3

7 S5 S4 10

8 S6 S11

9 R1 S7 R1 R1

10 R3 R3 R3 R3

11 R5 R5 R5 R5

slideslide4242

Stack Input Action

0 id + id * id $ Shift 5

0id5 + id * id + id * id $ Reduce 6 ﴾use GOTO[0,F]﴿

0F3 + id * id + id * id $ Reduce4 ﴾use GOTO[0,T]﴿

0T2 + id * id + id * id $ Reduce2 ﴾use GOTO[0,E]﴿

0E10E1 + id * id + id * id $ Shift 6

0E1+6 id * id id * id $ shift 5

0E1+6id5 * id * id $ Reduce 6 ﴾use GOTO[6,F]﴿0E1+6F3 * id * id $ Reduce 4 ﴾use GOTO[6,T]﴿0E1+6T9 * id * id $ Shift 7

0E1+6T9*7 id$ Shift 5

0E1+6T9*7id5 $ Reduce 6 ﴾use GOTO[7,F]﴿0E1+6T9*7F10 $ Reduce 3 ﴾use GOTO[6,T]﴿0E1+6T9 $ Reduce1 ﴾use GOTO[0,E]﴿0E10E1 $ Accept

slide1 chapter 4 lexical and syntax analysis. slide2 outlines: in this chapter a major topics will...

Documents

syntax analysis slide

portability lexical

program unit lexical

slide1 chapter

parsing problem

compiler system

primary approach

software interpreter