informa(on extrac(on · my mug broke into pieces immediately. the robot broke my favorite mug with...

Informa(onExtrac(on

WeiXu(many slides from Greg Durrett, Luheng He, Emma Strubell)

ThisLecture

‣ Howdowerepresentinforma(onforinforma(onextrac(on?

‣ OpenInforma(onExtrac(on

‣ Rela(onextrac(on

‣ Slotfilling

‣ Seman(crolelabeling/abstractmeaningrepresenta(on

IE:TheBigPicture

‣ Howdowerepresentinforma(on?Whatdoweextract?

‣ Slotfillers‣ En(ty-rela(on-en(tytriples(fixedontologyoropen)

‣ Seman(croles

‣ Abstractmeaningrepresenta(on

Seman(cRoleLabeling

‣ Findout5Wintext—“whodidwhattowhom,whenandwhere”

‣ Iden(fypredicate,disambiguateit,iden(fythatpredicate’sarguments

FigurefromHeetal.(2017)

Ques(on-AnswerDrivenSRL

FigurefromFitzGeraldetal.(2018)

Seman(cRoleLabeling

My mug broke into pieces immediately.

The robot broke my favorite mug with a wrench.

Semantic Role Labeling (SRL)

3 FigurefromHeetal.(2017)

Seman(cRoleLabeling

vsubj obj prep

subj v prep adv

Seman(cRoleLabeling

vsubj obj prep

subj v prep adv

thing broken

breaker instrument

pieces (final state) temporal

Seman(cRoleLabeling

vsubj obj prep

Frame: break.01role description

ARG0 breakerARG1 thing brokenARG2 instrumentARG3 pieces

ARG4 broken away from what?

subj v prep adv

thing broken

breaker instrument

pieces (final state) temporal

ARG0 ARG1 ARG2

ARG3ARG1 ARGM-TMP

�11 4

The Proposition Bank (PropBank)

Core roles: Verb-specific roles (ARG0-

ARG5) defined in frame files

Frame: break.01

role descriptionARG0 breakerARG1 thing broken

ARG2 instrument

ARG3 pieces

ARG4 broken away from what?

Frame: buy.01

role descriptionARG0 buyerARG1 thing bough

ARG2 seller

ARG3 price paid

ARG4 benefactive

role descriptionTMP temporalLOC locationMNR mannerDIR directionCAU causePRP purpose…

Adjunct roles: (ARGM-) shared

across verbs

Annotated on top of the Penn Treebank Syntax

PropBank Annotation Guidelines, Bonial et al., 2010

Paul Kingsbury and Martha Palmer.From Treebank to PropBank. 2002TheProposi(onBank(PropBank)

Seman(cRoleLabeling

‣ Iden(fypredicates(love)usingaclassifier(notshown)

‣ Iden(fyARG0,ARG1,etc.asataggingtaskwithaBiLSTMcondi(onedonlove

‣ Othersystemsincorporatesyntax,jointpredicate-argumentfinding

Residual/HighwayConnec(ons

�1323

input from the previous layer

recurrent input

from the prev. timestep

output to the next layer

References: Deep Residual Networks, Kaiming He, ICML 2016 Tutorial

Training Very Deep Networks, Srivastava et al., 2015

Non-linearity

ht shortcut

ht�1 F(ct�1,ht�1,xt)

residual netht + xt

gated highway network:

rt � ht + (1� rt) � xt

rt = �(f(ht�1,xt))

new output:

Model - (2) Highway ConnectionsDeep BiLSTM Tagger Highway Connections Variational

DropoutViterbi Decoding w\

Hard Constraints

input from the previous layer

recurrent input

from the prev. timestep

output to the next layer

References: Deep Residual Networks, Kaiming He, ICML 2016 Tutorial

Training Very Deep Networks, Srivastava et al., 2015

Non-linearity

ht shortcut

ht�1 F(ct�1,ht�1,xt)

residual netht + xt

gated highway network:

rt � ht + (1� rt) � xt

rt = �(f(ht�1,xt))

new output:

Model - (2) Highway ConnectionsDeep BiLSTM Tagger Highway Connections Variational

DropoutViterbi Decoding w\

Hard Constraints

SRLSystems

�1416

SRL Systems

syntactic features

candidate argument spans

labeled arguments

prediction

labeling

ILP/DP

sentence, predicate

argument id.

Pipeline Systems

Deep BiLSTM

Hard constraints

BIO sequence

prediction

sentence, predicate

*This work

Punyakanok et al., 2008 Täckström et al., 2015 FitzGerald et al., 2015

sentence, predicate

BIO sequence

prediction

Deep BiLSTM + CRF layer

Viterbi

context window features

End-to-end Systems

Collobert et al., 2011 Zhou and Xu, 2015 Wang et. al, 2015

He et al., 2017

The System on previous slide

(syntax-based)

[LISA]

10YearsofPropBankSRL

F1 Syntax-based

End-to-enddeep NN

[FitzGerald, Täckström, Ganchev, Das]

[Täckström, Ganchev, Das]

[Zhou & Xu]

[He, Lee, Lewis, Zettlemoyer] [He, Lee, Levy,

Zettlemoyer]

[Tan, Wang, Xie, Chen, Shi]

[Toutanova, Haghigi, Manning][Punyakanok, Roth, Yih]

FigurefromStrubelletal.(2018)

Seman(cRoleLabeling

‣ Canwecombinethetwoapproaches—incorporatesyntac(cinforma(onintoneuralnetworks?

‣Mul(-tasklearningwithrelatedtasks,e.g.,part-of-speechtagging,dependencyparsing…

‣ Syntac(cally-informedself-afen(on:usetheTransformertoencorethesentence;inonehead,tokenafendstoitslikelysyntac(cparents;innextlayer,tokensobserveallotherparents.

committee awards Strickland advanced opticswhoNobelp+1

Recall:Transformer(mul(-headself-afen(on)

Multi-head self-attention + feed forward

Layer 1

Layer p

Multi-head self-attention + feed forwardLayer J

committee awards Strickland advanced opticswhoNobel

Recall:Transformer(mul(-headself-afen(on)FigurefromStrubelletal.(2018)

Layer p

Feed Forward

biaffine parser

[Dozat and Manning 2017]committee awards Strickland advanced opticswhoNobel

optics advanced

who Strickland

awards committee

NobelA

Syntac(cally-InformedSelf-Afen(on

Syntactically-informed self-attention

Multi-head self-attention + feed forwardLayer 1

Layer p

Linguis(cally-InformedSelf-Afen(onFigurefromStrubelletal.(2018)

Layer p Syntactically-informed self-attention

Linguis(cally-InformedSelf-Afen(on

Layer 1 Multi-head self-attention + feed forward

Multi-head self-attention + feed forwardLayer r

NNP NN VBZ/PRED NNP WP VBN/PRED NN

FigurefromStrubelletal.(2018)

Layer 1

Syntactically-informed self-attentionLayer p

Multi-head self-attention + feed forward

Layer r Multi-head self-attention + feed forward

nn nsubj root dobj nsubj rcmod dobj

NNP NN VBZ/PRED NNP WP NNVBN/PRED

Layer 1 Multi-head self-attention + feed forward

Layer p

argspredicates

Bilinear

Layer p

argspredicates

Bilinear

Layer p

class labels

B-ARG0

argspredicates

Bilinear

Layer p

B-ARG0

argspredicates

Bilinear

Layer p

B-ARG0

argspredicates

Bilinear

Layer p

I-ARG0 B-V B-ARG1 O O O

B-ARG0

argspredicates

Bilinear

Layer p

I-ARG0 B-V B-ARG1 O O OLinguis(cally-InformedSelf-Afen(on

B-ARG0

argspredicates

Bilinear

Layer p

B-ARG0

argspredicates

Bilinear

Layer p

B-ARG0

argspredicates

Bilinear

I-ARG0 B-V B-ARG1 O O OO O O B-ARG0 B-R-ARG0 B-V B-ARG1

pospreds

labeledparse

GloVe ELMo

in-domain (dev) in-domain (dev)

He et al. 2017 81.5 ---

He et al. 2018 81.6 85.3

SA 82.39 85.26

LISA 82.24 85.35

+D&M 83.58 85.17

+Gold 86.81 87.63

Strubelletal.(2018)

WhySRLisdifficult?orNLPingeneral

‣ Syntac(cAlterna(onThe robot plays piano.

The music plays softly.

ARG0player

ARG1thing performed

ARG2instrument

ARGM-MNR

The cafe is playing my favorite song.

ARG1thing performed

ARG0player

subjects

objects

Syntactic Alternation PP Attachment Long-range Dependencies

‣ Preposi(onalPhrase(PP)Afachment

I eat [pasta] [with delight].ARG0eater

ARG1meal

ARGM-MNRmanner

I eat [pasta with broccoli].ARG0eater

ARG1meal

Syntactic Alternation Prepositional Phrase (PP) Attachment Long-range

Dependencies

We flew to Chicago.ARG1

passengerARGM-GOL

We remember the nice view flying to Chicago.ARG1

passengerARGM-GOL

We remember John and Mary flying to Chicago.ARG1

passengerARGM-GOL

Syntactic Alternation PP Attachment Long-range Dependencies‣ LongDependencies

‣ Evenharderforout-of-domaindataSRL is even harder for out-domain data …

“Dip chicken breasts into eggs to coat”

Active, Ser133-phosphorylated CREB effects transcription of CRE-dependent genes via interaction with the 265-kDa …

AbstractMeaningRepresenta(on

‣ Graph-structuredannota(on

‣ Nodesarevariableslabeledbyconcepts;edgesareseman(crela(onsAMR – Abstract Meaning Representation

“Bob ate four cakes that he bought.”(x2 / eat-01

:ARG0 (x1 / person

:name (n / name

:op1 "Bob")

:wiki "Bob_X")

:ARG1 (x4 / cake

:quant 4

:ARG1-of (x7 / buy-01

:ARG0 x1)))

e / eat-01

x4 / cakex1 / person

x7 / buy-01

"Bob_X"

FigurefromGruzi(setal.(2014)

AbstractMeaningRepresenta(on

Banarescuetal.(2014)

‣ SupersetofSRL:fullsentenceanalyses,containscoreferenceandmul(-wordexpressionsaswell

‣ F1scoresinthe60s:hard!

‣ Socomprehensivethatit’shardtopredict,buts(lldoesn’thandletenseorsomeotherthings…

Summariza(onwithAMR

Liuetal.(2015)

‣MergeAMRsacrossmul(plesentences

‣ Summariza(on=subgraphextrac(on

SlotFilling

SlotFilling‣Mostconserva(ve,narrowformofIE

IndianExpress—Amassiveearthquakeofmagnitude7.3struckIraqonSunday,103kms(64miles)southeastofthecityofAs-Sulaymaniyah,theUSGeologicalSurveysaid,reportsReuters.USGeologicalSurveyiniKallysaidthequakewasofamagnitude7.2,beforerevisingitto7.3.

magnitude time

epicenter

Speaker:AlanClark

“GenderRolesintheHolyRomanEmpire”

AllagherCenterMainAuditorium

Thistalkwilldiscuss…

speakertitle

location

FreitagandMcCallum(2000)

‣ Oldwork:HMMs,laterCRFstrainedperrole

SlotFilling:MUC

HaghighiandKlein(2010)

‣ Keyaspect:needtocombineinforma(onacrossmul(plemen(onsofanen(tyusingcoreference

SlotFilling:Forums

‣ Extractcode-relatedconceptsinsoowareengineeringforums,butnoteverythingthatlookslikeaconceptisaconcept(e.g.,windows)

avariablenameinthiscontext

JeniyaTabassum,MounicaMaddela,WeiXu,AlanRi8er.“CodeandNamedEn?tyRecogni?oninStackOverflow”inACL(2020)

Rela(onExtrac(on

Rela(onExtrac(on‣ Extracten(ty-rela(on-en(tytriplesfromafixedinventory

ACE(2003-2005)

DuringthewarinIraq,AmericanjournalistsweresomeKmescaughtinthelineoffire

Located_In

‣ Systemscanbefeature-basedorneural,lookatsurfacewords,syntac(cfeatures(dependencypaths),seman(croles

Na(onality

‣ Problem:limiteddataforscalingtobigontologies

‣ UseNER-likesystemtoiden(fyen(tyspans,classifyrela(onsbetweenen(typairswithaclassifier

HearstPaferns

Hearst(1992)

Xsuchas[list] ciKessuchasBerlin,Paris,andLondon.

‣ Syntac(cpafernsespeciallyforfindinghypernym-hyponympairs(“isa”rela(ons)

Berlinisacity

otherciKesincludingBerlin

otherXincludingY

‣ Totallyunsupervisedwayofharves(ngworldknowledgefortaskslikeparsingandcoreference(BansalandKlein,2011-2012)

DistantSupervision

Mintzetal.(2009)

[StevenSpielberg]’sfilm[SavingPrivateRyan]islooselybasedonthebrothers’story

‣ Iftwoen((esinarela(onappearinthesamesentence,assumethesentenceexpressestherela(on

‣ Lotsofrela(onsinourknowledgebasealready(e.g.,23,000film-directorrela(ons);usethesetobootstrapmoretrainingdata

Allisonco-producedtheAcademyAward-winning[SavingPrivateRyan],directedby[StevenSpielberg]

Director

DistantSupervision

Mintzetal.(2009)

‣ Learndecentlyaccurateclassifiersfor~100Freebaserela(ons‣ Couldbeusedtocrawlthewebandexpandourknowledgebase

DistantSupervision

FigurefromJiangetal.(2016)

‣ Inherentlyhavenoiseintrainingdata,needspecialmethods(e.g.,mul(-instancelearning)tohandlefalseposi(vesANDfalsenega(ves.

falseposi(ve

Mul(-instanceLearning

Xuetal.(2013),Pershinaetal.(2014),Tabassum(2016)

Nega%ve'Bags' Posi%ve'Bags''

A'bag'is'labeled'posi%ve,'if''there'is'at#least#one'posi%ve'example'

A'bag'is'labeled'nega%ve,'if''all'the'examples'in'it'are'nega%ve'

‣ Insteadoflabelsoneachindividualinstance,thelearneronlyobserveslabelsonbagsofinstances.

Mul(-instanceLearning

Xuetal.(2013),Pershinaetal.(2014),Tabassum(2016)

OpenIE

OpenInforma(onExtrac(on

‣ Typicallynofixedrela(oninventory

‣ “Open”ness—wanttobeabletoextractallkindsofinforma(onfromopen-domaintext

‣ Acquirecommonsenseknowledgejustfrom“reading”aboutit,butneedtoprocesslotsoftext(“machinereading”)

TextRunner‣ Extractposi(veexamplesof(e,r,e)triplesviaparsingandheuris(cs

BarackObama,44thpresidentoftheUnitedStates,wasbornonAugust4,1961inHonolulu

=>Barack_Obama,wasbornin,Honolulu

‣ TrainaNaiveBayesclassifiertofiltertriplesfromrawtext:usesfeaturesonPOStags,lexicalfeatures,stopwords,etc.

‣ 80xfasterthanrunningaparser(whichwasslowin2007…)

Bankoetal.(2007)

‣ Usemul(pleinstancesofextrac(onstoassignprobabilitytoarela(on

Exploi(ngRedundancy

Bankoetal.(2007)

‣ Concrete:definitelytrueAbstract:possiblytruebutunderspecified

‣ Hardtoevaluate:canassessprecision ofextractedfacts,buthowdoweknowrecall?

‣ 9Mwebpages/133Msentences

‣ 2.2tuplesextractedpersentence,filterbasedonprobabili(es

ReVerb

Faderetal.(2011)

‣Moreconstraints:openrela(onshavetobeginwithverb,endwithpreposi(on,becon(guous(e.g.,wasbornon)

‣ Extractmoremeaningfulrela(ons,par(cularlywithlightverbs

ReVerb

Faderetal.(2011)

‣ Foreachverb,iden(fythelongestsequenceofwordsfollowingtheverbthatsa(sfyaPOSregex(V.*P)andwhichsa(sfyheuris(clexicalconstraintsonspecificity

‣ Findthenearestargumentsoneithersideoftherela(on

‣ Annotatorslabeledrela(onsin500documentstoassessrecall

Takeaways

‣ Rela(onextrac(on:cancollectdatawithdistantsupervision,usethistoexpandknowledgebases

‣ Slotfilling:(edtoaspecificontology,butgivesfine-grainedinforma(on

‣ OpenIE:extractslotsofthings,buthardtoknowhowgoodorusefultheyare

‣ Cancombinewithstandardques(onanswering

‣ Addnewfactstoknowledgebases

‣ SRL/AMR:handleabunchofphenomena,butmoreorlesslikesyntax++intermsofwhattheyrepresent

‣Many,manyapplica(onsandtechniques

informa(on extrac(on · my mug broke into pieces immediately. the robot broke my favorite mug with...

Documents

mugger mug

the south bay mug - sbamug · used the better your score....

a yellow mug

mug design

sublimation mug printing business with mug heat press and...

i broke a mug

mug magazine #24

ta mug 01003

mug booklet

big mug big_love_2013_02_14

digital multi-mug press - listawood trade supplies · a...

oraciones concesivas. subj o indic

churchill mug

heart mug rug

hsca mug book

mug cake receta

_spinning mug ppt

coasters & mug wrap

etticut mug affair

mason jar melonade wine beer - montana mike's...