ling 438/538 computational linguistics sandiway fong lecture 24: 11/16

LING 438/538Computational Linguistics

Sandiway Fong

Lecture 24: 11/16

Administrivia

• Current Reading– Chapter 10: Parsing with Context-Free Grammars

• Lecture schedule– Tuesday 21st November

• Homework #6: Context-free Grammars and Parsing• due Tuesday 28th

– Thanksgiving break– Tuesday 28th November– Thursday 30th November

• Homework #7: Machine Translation• due December 7th• 538 Presentations

– Tuesday 5th December• Homework #7: Machine Translation• 538 Presentations

Administrivia

• 538 Presentations – details

• select a chapter• prepare a powerpoint (or PDF) presentation • your goal is to present a clear explanation and summary of the

assumptions, ideas and techniques used– you do not have to cover the entire chapter– you could choose to cover one or two sub-sections in depth– or survey the entire chapter

• offer a critique of the methods– e.g. disadvantages– e.g. are there better methods out there now? (Google is your friend)– or present examples for which the method may not work well on

• graded on how well you do this• you have 10 minutes for the presentation• answer questions

Administrivia

• Your choice – (pick any chapter from 11 to 20, inclusive)– send me an email with your top 3 choices– first come, first served – per order of emails as received in my mailbox– due date

• November 29th midnight• slides are due in my mailbox• (no advantage with respect to slides for the presentation date)

– (pick your presentation date)• November 30th• December 5th

Administrivia

• 10 Chapters Available– there are more than 10 of you

taking 538– chapters may be split into

two presentations

– 11: Features and Unification

– 12: Lexicalized and Probabilistic Parsing

– 13: Language and Complexity

– 14: Representing Meaning

– 15: Semantic Analysis– 16: Lexical Semantics– 17: Word Sense

Disambiguation and Information Retrieval

– 18: Discourse– 19: Dialogue and

Conversational Agents– 20: Natural Language

Generation

Administrivia

• Homework 5– due tonight (revised due date) – corpus.txt.zip

• (first version I put up on the website was somehow truncated,

• 2nd version is of the correct length, • data between <text> ... </text>

• there is still some noise in the data: – e.g. &amp and tables

• ignore them)

Today’s Topic

• Chapter 10: – Parsing with Context-Free Grammars

Top-Down Parsing

• we already know one top-down parsing algorithm– DCG rule system starting at the top node– using the Prolog computation rule

• always try the first matching rule

– expand x --> y, z.• top-down: x then y and z• left-to-right: do y then z• depth-first: expands DCG rules for y before tackling z

• problems– left-recursion

• gives termination problems

– no bottom-up filtering• inefficient• left-corner idea

Top-Down Parsing

• Prolog computation rule – is equivalent to

the stack-based algorithm shown in the textbook

– (figure 10.6)

• Prolog advantage– we don’t need

to implement this explicitly

– this is default Prolog strategy

Top-Down Parsing

• assume grammar

Top-Down Parsing

• example– does this flight

include a meal?

mismatch

➡

Top-Down Parsing


include a meal?

Top-Down Parsing

• Left Recursion– example

• np(np(D,N)) --> det(D), nominal(N).• det(d(a)) --> [a].• det(d(NP,'\'s')) --> np(NP), ['\'s'].• nominal(nom(N)) --> noun(N).• noun(n(man)) --> [man].

– query: NP a man• ?- np(X,[a,man],[]).• X = np(d(a),nom(n(man))) ? ; (goes into a loop)• Prolog interruption (h for help)? a• % Execution aborted

– query: NP a man’s man• ?- np(X,[a,man,'\'s',man],[]).• X = np(d(np(d(a),nom(n(man))),'\'s'),nom(n(man))) ? ;• Prolog interruption (h for help)? a• % Execution aborted

Sentential forms:npdet nominalnp ‘s nominal

Top-Down Parsing


• np(np(D,N)) --> det(D), nominal(N).• det(d(a)) --> [a].• det(d(NP,'\'s')) --> np(NP), ['\'s']. % left recursive rule• nominal(nom(N)) --> noun(N).• noun(n(man)) --> [man].

– query (trace)• | ?- np(X,[a,man],[]).• 1 1 Call: np(_410,[a,man],[]) ? • 2 2 Call: det(_998,[a,man],_993) ? s• ? 2 2 Exit: det(d(a),[a,man],[man]) ? • 3 2 Call: nominal(_999,[man],[]) ? • 4 3 Call: noun(_2183,[man],[]) ? s• 4 3 Exit: noun(n(man),[man],[]) ? • 3 2 Exit: nominal(nom(n(man)),[man],[]) ? • ? 1 1 Exit: np(np(d(a),nom(n(man))),[a,man],[]) ? • X = np(d(a),nom(n(man))) ? ;

Top-Down Parsing



– query (trace)• 1 1 Redo: np(np(d(a),nom(n(man))),[a,man],[]) ? • 2 2 Redo: det(d(a),[a,man],[man]) ? • 5 3 Call: np(_1417,[a,man],_1412) ? • 6 4 Call: det(_1818,[a,man],_1813) ? s• ? 6 4 Exit: det(d(a),[a,man],[man]) ? • 7 4 Call: nominal(_1819,[man],_1412) ? • 8 5 Call: noun(_3003,[man],_1412) ? s• 8 5 Exit: noun(n(man),[man],[]) ? • 7 4 Exit: nominal(nom(n(man)),[man],[]) ? • ? 5 3 Exit: np(np(d(a),nom(n(man))),[a,man],[]) ? • 9 3 Call: 'C'([],'\'s',_993) ?

DCG rules translated into Prolog:det(d(a), A, B) :- 'C'(A, a, B).det(d(A,'\'s'), B, C) :- np(A, B, D), 'C'(D, '\'s', C).

requires a following ‘s

Top-Down Parsing



– query (trace)• 9 3 Call: 'C'([],'\'s',_993) ? • 9 3 Fail: 'C'([],'\'s',_993) ? • 5 3 Redo: np(np(d(a),nom(n(man))),[a,man],[]) ? • 6 4 Redo: det(d(a),[a,man],[man]) ? • 10 5 Call: np(_2237,[a,man],_2232) ? • 11 6 Call: det(_2638,[a,man],_2633) ? s• ? 11 6 Exit: det(d(a),[a,man],[man]) ? • 12 6 Call: nominal(_2639,[man],_2232) ? s• 12 6 Exit: nominal(nom(n(man)),[man],[]) ? • ? 10 5 Exit: np(np(d(a),nom(n(man))),[a,man],[]) ? • 13 5 Call: 'C'([],'\'s',_1813) ?

now requires two following ‘s

strategy? one possibility• lookahead:

•before committing to recursion• det(d(NP,'\'s')) --> np(NP), ['\'s'].

•det(d(NP,'\'s')) --> checkinputaheadfor('\'s'), np(NP), ['\'s'].

Top-Down Parsing• back to textbook example• assume grammar

Top-Down Parsing

• example– does this flight include a meal?

• query– ?- s(X,

[does,this,flight,include,a,meal],[]).

– X = s(aux(does),np(det(this),nom(noun(flight))),vp(verb(include),np(det(a),nom(noun(meal)))))

Top-Down Parsing

• Prolog grammar• s(s(NP,VP)) --> np(NP), vp(VP).• s(s(Aux,NP,VP)) --> aux(Aux),

np(NP), vp(VP).• s(s(VP)) --> vp(VP).• np(np(D,N)) --> det(D),

nominal(N).• nominal(nom(N)) --> noun(N).• nominal(nom(N1,N)) --> noun(N1),

nominal(N).• np(np(PN)) --> propernoun(PN).• vp(vp(V)) --> verb(V).• vp(vp(V,NP)) --> verb(V),np(NP).• det(det(that)) --> [that].• det(det(this)) --> [this].• det(det(a)) --> [a].• noun(noun(book)) --> [book].• noun(noun(flight)) --> [flight].

• noun(noun(meal)) --> [meal].• noun(noun(money)) --> [money].• verb(verb(book)) --> [book].• verb(verb(include)) --> [include].• verb(verb(prefer)) --> [prefer].• aux(aux(does)) --> [does].• preposition(prep(from)) --> [from].• preposition(prep(to)) --> [to].• preposition(prep(on)) --> [on].• propernoun(propn(houston)) -->

[houston].• propernoun(propn(twa)) --> [twa].• nominal(nom(N,PP)) --> nominal(N),

pp(PP).• pp(pp(P,NP)) --> preposition(P),

np(NP).

Top-Down Parsing


include a meal?

• gain in efficiency– avoid computing the

first row of Figure 10.7

Top-Down Parsing

• no bottom-up filtering– left-corner idea– eliminate unnecessary top-down search– reduce the number of choice points (amount of branching)


• computation:1. s --> np, vp.

2. s --> aux, np, vp.

3. s --> vp.

– left-corner idea rules out 1 and 3

Left Corner Parsing

• need bottom-up filtering– filter top-down rule expansion using bottom-up information– current input is the bottom-up information– left-corner idea

• example– s(s(NP,VP)) --> np(NP), vp(VP).

– what terminals can be used to begin this phrase?– answer: whatever can begin NP– np(np(D,N)) --> det(D), nominal(N).– np(np(PN)) --> propernoun(PN).

– answer: whatever can begin Det or ProperNoun– det(det(that)) --> [that].– det(det(this)) --> [this].– det(det(a)) --> [a].– propernoun(propn(houston)) --> [houston].– propernoun(propn(twa)) --> [twa].

– answer: – {that,this,a,houston,twa} “Left Corner”

s /\ np vp /\det nominalpropernoun

Left Corner Parsing


• computation1. s(s(NP,VP)) --> np(NP), vp(VP). LC: {that,this,a,houston,twa}2. s(s(Aux,NP,VP)) --> aux(Aux), np(NP), vp(VP). LC: {does}3. s(s(VP)) --> vp(VP). LC: {book,include,prefer}

1. only rule 2 is compatible with the input2. match first input terminal against left-corner (LC) set for each possible

matching rule3. left-corner idea prunes away or rules out options 1 and 3

Left Corner Parsing

• DCG Rules1. s(s(NP,VP)) --> np(NP), vp(VP). LC: {that,this,a,houston,twa}

2. s(s(Aux,NP,VP)) --> aux(Aux), np(NP), vp(VP). LC: {does}

3. s(s(VP)) --> vp(VP). LC: {book,include,prefer}

• left-corner database facts1. % lc(rule#,[word|_],[word|_]).

2. lc(1,[that|L],[that|L]). lc(2,[does|L],[does|L]).

3. lc(1,this|L],[this|L]). lc(3,[book|L],[book|L]).

4. lc(1,[a|L],[a|L]). lc(3,[include|L],[include|L]).

5. lc(1,[houston|L],[houston|L]). lc(3,[prefer|L],[prefer|L]).

6. lc(1,[twa|L],[twa|L]).

• rewrite Prolog rules to check input against lc1. s(s(NP,VP)) --> lc(1), np(NP), vp(VP).

2. s(s(Aux,NP,VP)) --> lc(2), aux(Aux), np(NP), vp(VP).

3. s(s(VP)) --> lc(3), vp(VP).

Left Corner Parsing

• Summary:– Given a context-free DCG – Generate left-corner database facts

• lc(rule#,[word|_],[word|_]).

– Rewrite DCG rules to check input against lc1. s(s(NP,VP)) --> lc(1), np(NP), vp(VP).

– DCG rules are translated into underlying Prolog rules:1. s(s(A,B), C, D) :- lc(1, C, E), np(A, E, F), vp(B, F, D).

• This process can be done automatically (by program)• Note:

– not all rules need be rewritten– lexicon rules are direct left-corner rules– no filtering is necessary

• det(det(a)) --> [a].• noun(noun(book)) --> [book].

– i.e. no need to call lc as in• det(det(a)) --> lc(11), [a].• noun(noun(book)) --> lc(12), [book].

Left Corner Parsing

• Prolog query:• ?- s(X,[does,this,flight,include,a,meal],[]).• 1 1 Call: s(_430,[does,this,flight,include,a,meal],[]) ? • 2 2 Call: lc(1,[does,this,flight,include,a,meal],_1100) ? • 2 2 Fail: lc(1,[does,this,flight,include,a,meal],_1100) ? • 3 2 Call: lc(2,[does,this,flight,include,a,meal],_1107) ? • 3 2 Exit: lc(2,[does,this,flight,include,a,meal],[does,this,flight,include,a,meal]) ?• 4 2 Call: aux(_1112,[does,this,flight,include,a,meal],_1100) ? • 5 3 Call: 'C'([does,this,flight,include,a,meal],does,_1100) ? s• 5 3 Exit: 'C'([does,this,flight,include,a,meal],does,[this,flight,include,a,meal]) ? • 4 2 Exit: aux(aux(does),[does,this,flight,include,a,meal],[this,flight,include,a,meal]) ? • 6 2 Call: np(_1113,[this,flight,include,a,meal],_1093) ? • 7 3 Call: lc(4,[this,flight,include,a,meal],_3790) ? • ? 7 3 Exit: lc(4,[this,flight,include,a,meal],[this,flight,include,a,meal]) ? • 8 3 Call: det(_3795,[this,flight,include,a,meal],_3783) ? s• ? 8 3 Exit: det(det(this),[this,flight,include,a,meal],[flight,include,a,meal]) ? • 9 3 Call: nominal(_3796,[flight,include,a,meal],_1093) ? • 10 4 Call: lc(5,[flight,include,a,meal],_5740) ? s• ? 10 4 Exit: lc(5,[flight,include,a,meal],[flight,include,a,meal]) ?• 11 4 Call: noun(_5745,[flight,include,a,meal],_1093) ? s• ? 11 4 Exit: noun(noun(flight),[flight,include,a,meal],[include,a,meal]) ? • ? 9 3 Exit: nominal(nom(noun(flight)),[flight,include,a,meal],[include,a,meal]) ?

•? 6 2 Exit: np(np(det(this),nom(noun(flight))),[this,flight,include,a,meal],[include,a,meal]) ?

Left Corner Parsing• Prolog query (contd.):• 12 2 Call: vp(_1114,[include,a,meal],[]) ? • 13 3 Call: lc(8,[include,a,meal],_8441) ? s• ? 13 3 Exit: lc(8,[include,a,meal],[include,a,meal]) ? • 14 3 Call: verb(_8446,[include,a,meal],[]) ? s• 14 3 Fail: verb(_8446,[include,a,meal],[]) ? • 13 3 Redo: lc(8,[include,a,meal],[include,a,meal]) ? s• 13 3 Fail: lc(8,[include,a,meal],_8441) ? • 15 3 Call: lc(9,[include,a,meal],_8448) ? s• ? 15 3 Exit: lc(9,[include,a,meal],[include,a,meal]) ? • 16 3 Call: verb(_8453,[include,a,meal],_8441) ? s• ? 16 3 Exit: verb(verb(include),[include,a,meal],[a,meal]) ? • 17 3 Call: np(_8454,[a,meal],[]) ? • 18 4 Call: lc(4,[a,meal],_10423) ? s• 18 4 Exit: lc(4,[a,meal],[a,meal]) ? • 19 4 Call: det(_10428,[a,meal],_10416) ? s• 19 4 Exit: det(det(a),[a,meal],[meal]) ? • 20 4 Call: nominal(_10429,[meal],[]) ? • 21 5 Call: lc(5,[meal],_12385) ? s• ? 21 5 Exit: lc(5,[meal],[meal]) ? • 22 5 Call: noun(_12390,[meal],[]) ? s• ? 22 5 Exit: noun(noun(meal),[meal],[]) ? • ? 20 4 Exit: nominal(nom(noun(meal)),[meal],[]) ? • ? 17 3 Exit: np(np(det(a),nom(noun(meal))),[a,meal],[]) ? • ? 12 2 Exit: vp(vp(verb(include),np(det(a),nom(noun(meal)))),[include,a,meal],[]) ? • ? 1 1 Exit: s(s(aux(does),np(det(this),nom(noun(flight))),vp(verb(include),np(det(a),nom(noun(meal))))),

[does,this,flight,include,a,meal],[]) ?

•X = s(aux(does),np(det(this),nom(noun(flight))),vp(verb(include),np(det(a),nom(noun(meal))))) ?

Bottom-Up Parsing

• LR(0) parsing– An example of bottom-up tabular parsing– Similar to the top-down Earley algorithm

described in the textbook in that it uses the idea of dotted rules

Tabular Parsing

• e.g. LR(k) (Knuth, 1960)– invented for efficient parsing of programming languages – disadvantage: a potentially huge number of states can be generated when

the number of rules in the grammar is large– can be applied to natural languages (Tomita 1985)

• tables encode the grammar– grammar rules are compiled– no longer interpret the grammar rules directly

• Parser = Table + Push-down Stack– table entries contain instruction(s) that tell what to do at a given state

… possibly factoring in lookahead– stack data structure deals with maintaining the history of computation and

recursion

Tabular Parsing

• Shift-Reduce Parsing– example

• LR(0)– left to right – bottom-up – (0) no lookahead (input word)

• LR actions– Shift: read an input word

» i.e. advance current input word pointer to the next word

– Reduce: complete a nonterminal» i.e. complete parsing a grammar rule

– Accept: complete the parse» i.e. start symbol (e.g. S) derives the terminal string

Tabular Parsing

• LR(0) Parsing– L(G) = LR(0)

• i.e. the language generated by grammar G is LR(0)if there is a unique instruction per state(or no instruction = error state)LR(0) is a proper subset of context-free languages

– note• human language tends to be ambiguous• there are likely to be multiple or conflicting actions per state• can let Prolog’s computation rule handle it

– i.e. use Prolog backtracking

Tabular Parsing

• Dotted Rule Notation– “dot” used to indicate the progress of a parse

through a phrase structure rule– examples

• vp --> v . np means we’ve seen v and predict np

• np --> . d np means we’re predicting a d (followed by np)

• vp --> vp pp. means we’ve completed a vp

Tabular Parsing

• state– a set of dotted rules encodes the state of the parse

• kernel• vp --> v . np• vp --> v .

• completion (of predict)• np --> . d n• np --> . n• np --> . np cp

Tabular Parsing

• advance the dot– example: (d is next in the input)

• vp --> v . np• vp --> v . (eliminated)• np --> d . n• np --> . n (eliminated)• np --> . np cp

Tabular Parsing

• Dotted rules– example

• State 0:– s -> . np vp– np -> .d np– np -> .n– np -> .np pp

– possible actions• shift d and go to new state• shift n and go to new state

Tabular Parsing

• State 0: Shift D or N

S -> . NP VPNP -> . D NNP -> . NNP -> . NP PP

NP -> D . N

NP -> N .

State 0

State 2

State 1

Tabular Parsing

• State 1: Shift N


NP -> D . N

NP -> N .

State 0

State 2

State 1

NP -> D N .

State 3

Tabular Parsing

• Shift– take input word, and– place on stack

[D a] [N man] [V hit ] …

Input

Stack

Tabular Parsing

• Shift– take input word, and– place on stack

[V hit ] … [N man][D a ]

Input

Stack• state 3


NP -> D . N

NP -> N .

State 0

State 2

State 1

NP -> D N .

State 3

shift d

shift n

Tabular Parsing

• State 2: Reduce NP -> N


NP -> D . N

NP -> N .

State 0

State 2

State 1

NP -> D N .

State 3

Tabular Parsing

• Reduce NP -> N .– pop [N milk] off the stack, and– replace with [NP [N milk]] on stack

[V is ] … [N milk]

Input

Stack• State 2

Tabular Parsing

• Reduce NP -> N .– pop [N milk] off the stack, and

– replace with [NP [N milk]] on stack

[V is ] … [NP milk]

Input

Stack• State 2

Tabular Parsing

• State 3: Reduce NP -> D N .


NP -> N .

State 0

State 2

NP -> D . N

State 1

NP -> D N .

State 3

Tabular Parsing

• Reduce NP -> D N .– pop [N man] and [D a] off the stack– replace with [NP[D a][N man]]

[V hit ] … [N man][D a ]

Input

Stack• State 3

Tabular Parsing

[V hit ] … [NP[D a ][N man]]

Input

Stack• State 3

• Reduce NP -> D N .– pop [N man] and [D a] off the stack– replace with [NP[D a][N man]]

ling 438/538 computational linguistics sandiway fong lecture 24: 11/16

Documents

parsing example

grammar slide

mismatch slide

probabilistic parsing

contextfree grammars

default prolog strategy

administrivia homework

presentation date