learning narrative schemas nate chambers, dan jurafsky stanford university ibm watson research...

Learning Narrative SchemasLearning Narrative Schemas

Nate Chambers, Dan Jurafsky

Stanford University

IBM Watson Research Center Visit

Two Joint TasksTwo Joint Tasks

Events in a Narrative Semantic Roles

suspect, criminal, client, immigrant, journalist, government, …

police, agent, officer, authorities, troops, official, investigator, …

ScriptsScripts

• Background knowledge for language understanding

Restaurant Script

Schank and Abelson. 1977. Scripts Plans Goals and Understanding. Lawrence Erlbaum.

Mooney and DeJong. 1985. Learning Schemata for NLP. IJCAI-85.

• Hand-coded• Domain dependent

ApplicationsApplications

• Coreference• Resolve pronouns (he, she, it, etc.)

• Summarization• Inform sentence selection with event confidence scores

• Aberration Detection• Detect surprise/unexpected events in text

• Story Generation• McIntyre and Lapata, (ACL-2009)

• Textual Inference• Does a document infer other events

• Selectional Preferences• Use chains to inform argument types

The ProtagonistThe Protagonist

protagonist:

(noun)

1. the principal character in a drama or other literary work

2. a leading actor, character, or participant in a literary work or real event

Inducing Narrative RelationsInducing Narrative Relations

1. Dependency parse a document.

2. Run coreference to cluster entity mentions.

3. Count pairs of verbs with coreferring arguments.

4. Use pointwise mutual information to measure relatedness.

Chambers and Jurafsky. Unsupervised Learning of Narrative Event Chains. ACL-08

Narrative Coherence AssumptionVerbs sharing coreferring arguments are semantically connected

by virtue of narrative discourse structure.

Chain Example (ACL-08)Chain Example (ACL-08)

Schema Example (new)Schema Example (new)

Police, Agent, Authorities

Judge, OfficialProsecutor, Attorney

Plea, Guilty, InnocentSuspect, Criminal,Terrorist, …

Narrative SchemasNarrative Schemas

Integrating Argument TypesIntegrating Argument Types

• Use verb relations to learn argument types. • Record head nouns of coreferring arguments.

• Use argument types to learn verb relations.• Include argument counts in relation scores.

The typhoon was downgraded

Sunday as it moved inland from the

coast, where it killed two people.

downgrade-o, move-s, typhoon

move-s, kill-s, typhoon

downgrade-o, kill-s, typhoon

€

sim(ei,e j )

€

sim(ei,e j ,a)

Learning SchemasLearning Schemas

€

narsim(N,v j ) = maxC i

chainsim(Ci,< v j ,d >)d∈Dv j

∑

Argument InductionArgument Induction

SuspectGovernmentJournalistMondayMemberCitizenClient……

€

score( )

• Induce semantic roles by scoring argument head words.

Training DataTraining Data

• 1.2 million New York Times articles• NYT portion of the Gigaword Corpus• David Graff. 2002. English Gigaword. Linguistic Data Consortium.

• Stanford Parser• http://nlp.stanford.edu/software/lex-parser.shtml

• OpenNLP coreference• http://opennlp.sourceforge.net

• Lemmatize verbs and noun arguments.

Learned ExamplesLearned Examples

court, judge, justice, panel, Osteen, circuit, nicolau, sporkin, majority

law, ban, rule, constitutionality, conviction, ruling, lawmaker,

Learned ExamplesLearned Examples

company, inc, corp, microsoft, iraq, co, unit, maker, …

drug, product, system, test, software, funds, movie, …

Database of SchemasDatabase of Schemas

• ~500 unique schemas, 10 events each• Temporal ordering data• Available online soon.

EvaluationsEvaluations

• Compared to FrameNet• High precision when overlapping• New type of knowledge not included

• Cloze Evaluation• Predict missing events• Far better performance than vanilla distributional approaches

Future WorkFuture Work

• Improved information extraction• Extract information across multiple predicates.

• Knowledge Organization• Link news articles describing subsequent events.

• Core AI Reasoning• Automatic approach to learning causation?

• NLP specific tasks• Coreference, summarization, etc.

Thanks!Thanks!

• Unsupervised Learning of Narrative Schemas and their ParticipantsNathanael Chambers and Dan JurafskyACL-09, Singapore. 2009.

• Unsupervised Learning of Narrative Event ChainsNathanael Chambers and Dan JurafskyACL-08, Ohio, USA. 2008.

• Jointly Combining Implicit Constraints Improves Temporal OrderingNathanael Chambers and Dan JurafskyEMNLP-08, Waikiki, Hawaii, USA. 2008.

• Classifying Temporal Relations Between EventsNathanael Chambers, Shan Wang, Dan JurafskyACL-07, Prague. 2007.

Cloze EvaluationCloze Evaluation

1. Choose a news article at random.

2. Identify the protagonist.

3. Extract the narrative event chain.

4. Randomly remove one event from the chain.

• Predict which event was removed.

Cloze ResultsCloze Results

• Outperform the baseline distributional learning approach by 36%

• Including participants improves further by 10%

Comparison to FrameNetComparison to FrameNet

• Narrative Schemas• Focuses on events that occur together in a narrative.

• FrameNet (Baker et al., 1998)

• Focuses on events that share core roles.

Comparison to FrameNetComparison to FrameNet

• Narrative Schemas• Focuses on events that occur together in a narrative.• Schemas represent larger situations.

• FrameNet (Baker et al., 1998)

• Focuses on events that share core roles.• Frames typically represent single events.

Comparison to FrameNetComparison to FrameNet

1. How similar are schemas to frames?• Find “best” FrameNet frame by event overlap

2. How similar are schema roles to frame elements?• Evaluate argument types as FrameNet frame elements.

FrameNet Schema SimilarityFrameNet Schema Similarity

1. How many schemas map to frames?• 13 of 20 schemas mapped to a frame• 26 of 78 (33%) verbs are not in FrameNet

2. Verbs present in FrameNet• 35 of 52 (67%) matched frame• 17 of 52 (33%) did not match

FrameNet Schema SimilarityFrameNet Schema Similarity

traderisefall

Exchange

Change Position on a Scale

Two FrameNet FramesOne Schema

• Why were 33% unaligned?• FrameNet represents subevents as separate frames• Schemas model sequences of events.

FrameNet Argument SimilarityFrameNet Argument Similarity

2. Argument role mapping to frame elements.• 72% of arguments appropriate as frame elements

law, ban, rule, constitutionality,conviction, ruling, lawmaker, tax

INCORRECT

FrameNet frame: EnforcingFrame element: Rule

XX Event ScoringXX Event Scoring

€

chainsim(Ci,< acquit,subj >) =maxa

sim(< acquit,subj >,ei,a)i=1

n

∑

€

+score(Ci,a)

XX Argument InductionXX Argument Induction

• Induce semantic roles by scoring argument head words.

€

score( ) = (1− λ )pmi(ei,e j )j= i+1

n

∑i=1

n−1

∑ +λ log( freq(ei,e j , ))

How often do events shareany coreferring arguments?

How often do they shareargument ?

= criminal?

€

score( ) = sim(ei,e j , )j= i+1

n

∑i=1

n−1

∑

ResultsResults

Chains

Schemas

Typed Chains

Typed Schemas

10.1%

ResultsResults

1. We learned rich narrative structure.

• 10.1% improvement over previous work

2. Induced semantic roles characterizing the participants in a narrative.

3. Verb relations and their semantic roles can be jointly learned and improve each other’s results.

• Selectional preferences improve verb relation learning.

XX Semantic Role InductionXX Semantic Role Induction

• Supervised Learning• PropBank (Palmer et al., 2005),• FrameNet (Baker et al., 1998), • VerbNet (Kipper et al., 2000)

• Bootstrapping from a seed corpus• (Swier and Stevenson, 2004), (He and

Gildea, 2006)

• Unsupervised, pre-defined roles• (Grenegar and Manning 2006)

• WordNet inspired• (Green and Dorr, 2005), (Alishahi and

Stevenson, 2007)

SuspectGovernmentJournalistMondayMemberCitizenClient…