DESIGN OF LEXICAL

ANALYZER USING LEX

Lex helps to specify lexical analyzers by specifying regular expression

i/p notation for lex tool is lex language and the tool itself is refered to as lex compiler

Lex compiler transform i/p patterns into transition diagram and generates the code in a file lex.yy.c

USE OF LEX

Fig. Creating lexical analyzer using lex

i/p file is lex.l, describes the lexical analyzer to be generated

lex.l runs through lex compiler to produce a C program lex.yy.c that simulate transition diagram for i/p patterns

lex.yy.c is compiled by C compiler to produce a.out, the working lexical analyzer that can produce stream of tokens from sequence of i/p characters

a.out is a subroutine to parser. It returns an integer, which is a code for one of the possible token namesAttribute value is placed in a global variable

yylval, which is shared by lexical analyzer and parser

LEX SPECIFICATIONS The general format of Lex source is: {declarations}

%%{translation rules}%%{user subroutines}

Declaration: of variables, manifest constants( identifiers declared to stand for a constant), and regular definition

Translation rules: pattern {action} pattern-> regular expression that may use

regular definitions of declaration

sectionaction -> action lexical analyzer

should take when pattern pi matches a lexeme user subroutines: auxiliary procedures

needed by action

When called by parser , lexical analyzerBegins reading its remaining i/p until it finds the

longest prefix of i/p that matches one of the patterns Pi

Then execute associated action Ai (Typically Ai will return to the parser. If it does not, lexical analyzer proceeds to find additional lexemes until one of the corresponding actions causes a return to the parser)

lexical analyzer returns a single value , token name .

The shared integer variable yylval is used to pass additional information about the lexeme found

Int InstallID() { /*function to install lexeme, whose 1st character is pointed to by yytext, and whose length is yyleng, into the symbol table and returns a pointer to there */ }

Int installNum() {/* similar to InstallID() but puts numerical constants into separate table*/ }

The actions taken when id is matched are: InstallID() is called to place the lexeme found

in the symbol tableThis function returns a pointer to symbol table,

which is placed in global variable yylval, where it can be used by parser or a later component of compiler. installID() has two variables available to it that are set automatically by lexical analyzer generated by Lex: yytext : pointer to beginning of lexeme yyleng: length of lexeme found

token name ID is returned to parser action taken when lexeme matching

number is similar

CONFLICT RESOLUTION IN LEX When several prefixes of the i/p match one

or more patterns :Prefer the longest prefix to a shorter prefix

e.g: <= to < If longest possible prefix matches two or more

patterns, prefer the pattern listed first in Lex program

THE LOOKAHEAD OPERATOR Lex automatically reads one character

ahead of last character that forms the lexeme, and then retract the i/p so the lexeme itself is taken from i/p stream

Sometimes we want certain pattern to be matched to i/p only when it is followed by certain other characters

Use / in the pattern to indicate the end of the part of pattern that matches the lexemeWhat follows / is additional pattern that must

be matched before we can decide that token in question is seen. What matches second pattern is not part of the lexeme

e.g. IF(I,J)=3 (IF array name) IF (condition) THEN….

IF / \( .* \) {letter}