statistical models for frame-semantic parsingmiriamp/fillmore-tribute/slides/dipanjan_das.pdf ·...

Statistical Models for Frame-Semantic Parsing

Dipanjan Das*

GoogleFrame Semantics in NLP: A Workshop in Honor of Chuck Fillmore

June 27, 2014

*Thanks to Desai Chen, Kuzman Ganchev, Karl Moritz Hermann, André Martins, Nathan Schneider, Noah Smith and Jason Weston

Frame-Semantic Parsing

I want to travel to Baltimore on Sunday

2

I want to travel to on Sunday

TRAVEL Encodes an event or scenario

Baltimore


3


TRAVEL Participant or role for the

frameTravelerTime

GoalBaltimore


4


TRAVEL Participant or role for the

frameTravelerTime

GoalBaltimore


DESIRING

Experiencer Event

5

Outline

• Why should we build statistical models for frame semantics?

• Overview of statistical methods

• Future directions

6

Outline




7

Motivation

8

Motivation

Deeper understanding beyond syntax

8

Motivation


Grouping of predicates and argumentsinto clusters

8

Motivation


Grounding of natural language to an ontology

Grouping of predicates and argumentsinto clusters

8

Motivation

Bengal ’s massive stock of food was reduced to nothing

Motivation

Bengal ’s massive stock of food was reduced to nothingN X A N ADP N V V ADP N

Store or financial entity?

Motivation


Store of what?Of what size?Whose store?

Motivation


What was reduced?To what?

Motivation

Bengal ’s massive stock of food wasreducedto nothingN X A N ADP N V V ADP N


STOREPossessor

DescResource

CAUSE_CHANGE_OF_POSITION_ON_A_SCALE

Value_2Item

Motivation

Motivation

Motivation

James Cameron directed Titanic

BEHIND_THE_SCENES

Artist Production

Motivation


BEHIND_THE_SCENES

Artist Production

James Cameron is Titanic’s director

BEHIND_THE_SCENES

ProductionArtist

Motivation


BEHIND_THE_SCENES

Artist Production


BEHIND_THE_SCENES

ProductionArtist

Grouping across syntactic categories

Motivation

James Cameron filmed Titanic

BEHIND_THE_SCENES

Artist Production


BEHIND_THE_SCENES

ProductionArtist

Grouping across different lemmas

Motivation


film.01

ARG0


director.01

ARG2ARG0

ARG1from

PropBank

fromNomBank

Motivation


film.01

ARG0


director.01

ARG2ARG0

ARG1from

PropBank

fromNomBank

Hard to align different lemmas across resources

Motivation


film.01

ARG0


director.01

ARG2ARG0

ARG1from

PropBank

fromNomBank

Very generic argument labels

Motivation


film.01

ARG0


director.01

ARG2ARG0

ARG1from

PropBank

fromNomBank

Argument labels do not match up

Motivation

James Cameron filmed Titanic in 1997

BEHIND_THE_SCENES

ArtistProduction

Motivation


BEHIND_THE_SCENES

ArtistProduction

�e.filmed.arg1(e, James Cameron)^filmed.arg2(e,Titanic)^filmed.in(e, 1997)

�e.filmed.arg1(e, /m/03 gd) ^ filmed.arg2(e, /m/0dr 4) ^ filmed.in(e, 1997)

Motivation


BEHIND_THE_SCENES

ArtistProduction

�e.filmed.arg1(e, James Cameron)^filmed.arg2(e,Titanic)^filmed.in(e, 1997)


Motivation�e.filmed.arg1(e, James Cameron)^filmed.arg2(e,Titanic)^filmed.in(e, 1997)


Motivation�e.filmed.arg1(e, James Cameron)^filmed.arg2(e,Titanic)^filmed.in(e, 1997)

�e.BEHIND THE SCENES.Artist(e, /m/03 gd)^BEHIND THE SCENES.Production(e, /m/0dr 4)^BEHIND THE SCENES.Time(e, 1997)

�e.BEHIND THE SCENES.Artist(e, /m/03 gd)^BEHIND THE SCENES.Production(e, /m/0dr 4)^BEHIND THE SCENES.Time(e, 1997)

FrameNet

Applications ofFrame-Semantic Parsing

Stance Classification

• Frame Semantics for Stance ClassificationHasan and Ng (CoNLL 2013)

• Two sided debates in an online forum

• Classification of stance

• Improvement over a baseline that uses bag of words and dependencies

25

Dialog Systems

• Unsupervised Induction and Filling of Semantic Slots for Spoken Dialogue Systems Using Frame-Semantic ParsingChen, Wang and Rudnicky (ASRU 2013)

• Annotation of dialog transcripts with frame-semantic structures

• Uses only a subset of frames

• Uses these annotations for slot induction

26

Stock Price Movement

• Semantic Frames to Predict Stock Price MovementXie et al. (ACL 2013)

• Predict the change in stock price from financial news

• Lots of features along with features based on frames and roles

• Shows improvements over other features

27

Summarization

• Generating Automated Meeting SummariesThomas Kleinbauer (PhD thesis, Saarland University)

• Part of a large system for generating meeting summaries

28

Outline




29

Structure of Lexicon and Data

PLACINGAgentCause

Goal

ThemeArea

Time


PLACINGAgentCause

Goal

ThemeArea

Time

frame


PLACINGAgentCause

Goal

ThemeArea

Time

roles

frame


PLACINGAgentCause

Goal

ThemeArea

Time

core roles

non-core roles

frame


PLACINGAgentCause

Goal

ThemeArea

Time

excludes relationship

core roles

non-core roles

frame


PLACINGAgentCause

Goal

ThemeArea

Time

excludes relationship

core roles

non-core roles

frame

archive.V, arrange.V, bag.V, bestow.V bin.V

predicates

PLACINGAgentCause

Goal

ThemeArea

Time

DISPERSAL

AgentCause

IndividualsDistance

Time

TRANSITIVE_ACTIONAgentCause

Patient

EventPlace

Time

INSTALLINGAgent

Component

Fixed_locationArea

Time

STORINGAgentLocation

ThemeArea

Time

STOREPossessorResource

SupplyDescriptor


PLACINGAgentCause

Goal

ThemeArea

Time

DISPERSAL

AgentCause

IndividualsDistance

Time

TRANSITIVE_ACTIONAgentCause

Patient

EventPlace

Time

INSTALLINGAgent

Component

Fixed_locationArea

Time

STORINGAgentLocation

ThemeArea

Time

STOREPossessorResource

SupplyDescriptor

Structure of Lexicon and Datainheritance

used by

Datasets

Benchmark Dataset(SemEval 2007)

665 frames720 role labels

8.4K unique predicate types

Training set:2.2K sentences

11.2K predicate tokens

Test set:120 sentences

1. 1K predicate tokens

LTH Frame-Semantic ParserJohansson and Nugues (2007)

Frame Identification Argument Filtering Argument Labeling

40



41


ambiguous



Find the best among all frames



Taken from an SVM classifier trained on

ambiguous predicates



To increase coverage, potential predicates were extracted from WordNet and automatically frame labels were selected for

them.


Frame Identification Accuracy

50.0

61.3

72.5

83.8

95.0

LTH

57.3

F-Measure




47

Argument Filtering


STORE

massive stock

ø

Argument Filtering


STORE

massive stock

ø

Bengal ’s Bengal massive stock of food

food massive Bengal ’s massive to nothing

to of ’s

Potential Arguments

Argument Filtering


STORE

massive stock

ø



to of ’s

Potential ArgumentsBinary SVM Classification



50

Argument Filtering


STORE

massive stock

ø

Multiclass SVM Classification

Argument Filtering


STORE

massive stock

ø



to of ’s

Potential ArgumentsMulticlass SVM Classification

Argument Filtering


STORE

massive stock

ø



to of ’s

Potential ArgumentsMulticlass SVM Classification

Possessor Resource

Descriptorø

Full Frame-Semantic Structure Prediction

35.0

43.8

52.5

61.3

70.0

LTH

44.4

F-Measure


SEMAFORDas, Chen, Martins, Schneider, Smith (2014)

Frame Identification Argument Identification

54



55

SEMAFOR: Frame Identification


Logistic regression with a latent variable


Logistic regression with a latent variable

Predicates evoking a frame in supervised data, e.g.cargo.N, inventory.N, reserve.N, stockpile.N, store.N, supply.N

evoke STORE


STORE stock.N

stockpile.N



STORE stock.N

stockpile.N


LexSem = {synonym}


STORE stock.N

stockpile.N


LexSem = {synonym}

If STORE

stockpile.Nsynonym LexSem


STORE stock.N

stockpile.N


LexSem = {synonym}

If STORE

stockpile.Nsynonym LexSem

comes from WordNet!


Datasets

New Data(FrameNet 1.5, 2010)














New Data(FrameNet 1.5, 2010)














Datasets


ResultsBenchmark

50.0

61.3

72.5

83.8

95.0

LTH SEMAFOR

6157.3

F-Measure

New Data

log-linear

ResultsBenchmark

50.0

61.3

72.5

83.8

95.0

LTH SEMAFOR

6157.3

F-Measure

New Data

log-linear

auto predicates


ResultsBenchmark

50.0

61.3

72.5

83.8

95.0

LTH SEMAFOR

6157.3

F-Measure

New Data

log-linear

auto predicates

50.0

61.3

72.5

83.8

95.0

SEMAFOR

83.0

Accuracy

log-linear

gold predicates


20.0

38.8

57.5

76.3

95.0

All Predicates

83.0

Accuracy

20.0

38.8

57.5

76.3

95.0

Unknown Predicates

23.1

Accuracy

Frame Identification


SEMAFOR: Handling Unknown Predicates

Knowledge of only 9,263 predicates in supervised data


However, English has lot more potential predicates(~65,000 in newswire English)



However, English has lot more potential predicates(~65,000 in newswire English)

Lexicon expansion using graph-based semi-supervised learning


How can label propagation help?

Example Graph

Seed predicates

Example Graph

Seed predicatesUnseen predicates

Example Graph


Graph Propagation

Example Graph


Graph Propagation

Continues till convergence...

15.0

28.8

42.5

56.3

70.0

Supervised Self-Training Graph-Based

42.7

18.923.1

Accuracy


SEMAFOR: Unknown Predicates



81

SEMAFOR: Argument Identification


STORE


STORE

Possessor

Resource

Supply

Use

Descriptor


stock

STORE

Possessor

Resource

Supply

Use

Descriptor

Bengal ’s

Bengal

massive stock

of food

food

massive

Bengal ’s massive

massive stock

ø


stock

STORE

Possessor

Resource

Supply

Use

Descriptor

Bengal ’s

Bengal

massive stock

of food

food

massive

Bengal ’s massive

massive stock

ø

Violates overlap constraints


Other types of structural constraints

PLACINGAgentCause

Goal

ThemeArea

Time

Mutual exclusion constraint




PLACINGAgentCause

Goal

ThemeArea

Time



If an agent places something, there cannot be a cause role in the sentence



PLACINGAgentCause

Goal

ThemeArea

Time



The waiter placed food on the table.

In Kabul, hauling water put food on the table.Cause

Agent



SIMILARITYDimension

Differentiating_fact

Entity_1

Entity_2

Degree

Requires constraint

difference.N, resemble.V,

unliike.A, vary.V



SIMILARITYDimension


Entity_1

Entity_2

Degree

Requires constraint


unliike.A, vary.V

A mulberry resembles a loganberry.

second entity

first entity



SIMILARITYDimension


Entity_1

Entity_2

Degree

Requires constraint


unliike.A, vary.V

A mulberry resembles.!


stock

STORE

Possessor

Resource

Supply

Use

Descriptor

Bengal ’s

Bengal

massive stock

of food

food

massive

Bengal ’s massive

massive stock

ø

A constrained optimization

problem


stock

STORE

Possessor

Resource

Supply

Use

Descriptor

Bengal ’s

Bengal

massive stock

of food

food

massive

Bengal ’s massive

massive stock

ø


A constrained optimization problem



a binary variable for each role, span tuple



a binary vector for all role, span tuples



Uniqueness



Prevents overlap



more structural constraints

An integer linear program (ILP)





Punyakanok, Roth and Yih

(2008)





Often, very slow solutions





Fast ILP solvers proprietary





Dual Decomposition


SEMAFOR: Frame-Semantic Parsing

Final Results

Benchmark

35.0

43.8

52.5

61.3

70.0

LTH SEMAFOR

46.542.0

F-Measure

New Data


Final Results

Benchmark

35.0

43.8

52.5

61.3

70.0

LTH SEMAFOR

46.542.0

F-Measure

New Data

auto predicates


Final Results

Benchmark

35.0

43.8

52.5

61.3

70.0

LTH SEMAFOR

46.542.0

F-Measure

New Data

auto predicates

35.0

43.8

52.5

61.3

70.0

SEMAFOR

64.5

F-Measure

gold predicates

Hermann, Das, Weston and GanchevACL 2014

111

112

Frame Identification with Embeddings


113


Discrete lexical features


114



Syntactic context words connected via a set of

dependency paths

amod

; massiveprep pobj ; food

nsubjpass auxpass ; was

... ; ...

115



amod



amod

;prep pobj ;

nsubjpass auxpass ;

Replace context words with off-the-shelf word

embeddings

... ; ... ... ; ...

116

Frame Identification with Embeddingsamod



amod

;prep pobj ;

nsubjpass auxpass ;

... ; ... ... ; ...

amod nsubj dobj>prep>pobjp<nsubjpass>auxpass...

Input sparse embedding vector with context blocks

117


Frame instance space

N ⇥ Rd

117



N ⇥ Rd

Joint SpaceRm

117



N ⇥ Rd

Joint SpaceRm

Frame Instance MapM : Rd ! Rm

117



N ⇥ Rd

Joint SpaceRm


Set of FrameNet labels

STORESTORING

STINGINESS...

...

117



N ⇥ Rd

Joint SpaceRm



STORESTORING

STINGINESS...

...

Y 2 RF⇥mLabel matrix

117



N ⇥ Rd

Joint SpaceRm



STORESTORING

STINGINESS...

...

Y 2 RF⇥mLabel matrix

learned learned

118


118


frame instances

118


frame instancesframe labels

119



?

test predicate

120



?

test predicate

120



?

121



?

15.0

28.8

42.5

56.3

70.0

Supervised Self-Training Graph-Based Embeddings

46.1542.7

18.923.1

Accuracy


Results on Unknown Predicates

SEMAFOR

75.0

78.8

82.5

86.3

90.0

Supervised Self-Training Graph-Based Embeddings

86.49

83.682.3

83.0

Accuracy


Results on All Predicates

SEMAFOR

60.0

62.5

65.0

67.5

70.0

Supervised Graph-Based Embeddings

68.7

64.564.1

F-Score


Final Results

SEMAFOR

Outline




125

Better Models

• Very little research on argument identification

• More non-local features

• Using distributed representations

• Generalization using PropBank resources

126

Data• Number of FrameNet annotated sentences ~30

times less than PropBank/Ontonotes.

• Number of argument labels ~30 times more

• To make systems usable, we need annotations

• inter-annotator agreement studies

• annotation guidelines127

Custom FrameNets

• Is a general FrameNet lexicon useful?

• often, FrameNet frames are too general or too specific

• is it possible to quickly build customized FrameNet lexicons for applications?

• is it possible to use PropBank-style frames to induce FrameNet-style frames?

128

Conclusions

• Lot of exciting work in predicate-argument structure prediction

• Semi-supervised methods improve coverage

• Systems trained on small amounts of FrameNet-style data shown to be useful

• More annotations will result in usable systems

129

Thank You

130

statistical models for frame-semantic parsingmiriamp/fillmore-tribute/slides/dipanjan_das.pdf ·...

Documents