sentiment analysis - columbia university · • social media features *apoorv agarwal, fadi biadsy,...
TRANSCRIPT
![Page 1: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/1.jpg)
SentimentAnalysis
![Page 2: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/2.jpg)
Announcements• HW1isduetodayat11:59pm• HW2willcomeoutMonday• IfyouuselatedaysforHW1,MUSTSTATEITINYOURSUBMISSION.• Reading:Today:C4.2NLP• Monday:C8.1-8.3SpeechandLanguage,
8.1NLP• RecommendreadingchaptersinYoavGoldbergonneuralnetsandJurafskyandMarVnaswell.• BotharemoreintuiVve
![Page 3: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/3.jpg)
Today• SenVmentanalysistasks:definiVon• SenVmentresources• TradiVonalsupervisedapproach• Neuralnetapproach
![Page 4: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/4.jpg)
Whatissentiment?• ExpressionofposiVveornegaVveopinions• ..Towardsatopic,person,event,enVty• ..Towardsanaspect(e.g.,service,foodorambienceinarestaurant)
![Page 5: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/5.jpg)
Whysentimentanalysis?• SenVmentiscommoninonlinepla[orms• Peoplewriteabouttheirpersonalviewpoints
• UsefultounderstandwhatpeoplethinkaboutpoliVcalissues,poliVcalcandidates,importanteventsoftheday• UsefulforgeneraVngsummariesofreviews:restaurants,products,movies
![Page 6: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/6.jpg)
Thesentimentanalysistask(s)• SubjecVvevsobjecVve• Polarity:PosiVve,negaVveorneutral• DowehavesenVmenttowardsatarget?OraspectbasedsenVment?• What/whoisthesenVmentsource?
![Page 7: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/7.jpg)
SubjectivevsObjective• �Atseveraldifferentlayers,it’safascina3ngtale.[“Who’sSpyingonOurComputers”,GeorgeMelloanWallStJournal.(Bookreview)• BellIndustriesIncincreaseditsquarterlyto10centsfrom7centsashare.
ExamplesfromWeibeetal2004
![Page 8: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/8.jpg)
Positive/Negative/Neutral• FromUseNet:
• NegaVve:Ihadinmindyourfacts,Buddy,nothers.• PosiVve:Nicetouch.“Alleges”whateverfactspostedarenotinyourpersonaofwhatis“real”• Neutral:Marchappearstobeanes3matewhileearlieradmissioncannotbeen3relyruledout,"accordingtoChen,alsoTaiwan'schiefWTOnego3ator
ExamplesfromWeibeetal2004andRosenthal2014
![Page 9: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/9.jpg)
SubjectivePhrases• TheforeignministrysaidThursdaythatitwas“surprised,toputitmildly”bytheU.S.StateDepartment’scri0cismofRussia’shumanrightsrecordandobjectedinparVculartothe“odious”secVononChechnya.[MoscowTimes,03/08/2002]• Subjec3vityanalysisiden3fiestextthatrevealsanauthor’sthoughts,beliefsorotherprivatestates.
ExamplesfromWeibeetal2004
![Page 10: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/10.jpg)
SubjectivePhrasesandSources• TheforeignministrysaidThursdaythatitwas“surprised,toputitmildly”bytheU.S.StateDepartment’scri0cismofRussia’shumanrightsrecordandobjectedinparVculartothe“odious”secVononChechnya.[MoscowTimes,03/08/2002]• Whowassurprised?• WhowascriVcal?
ExamplesfromWeibeetal2004
![Page 11: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/11.jpg)
SentimenttowardsTarget• IpreUymuchenjoyedthewholemovie.Target=wholemovie,senVment=posiVve.• Bulgariaiscri3cizedbytheEUbecauseofslowreformsinthejudiciarybranch,thenewspapernotes.Target=Bulgaria,senVment=negaVve• Stanishevwaselectedprimeministerin2005.Sincethen,hehasbeenaprominentsupporterofhiscountry’saccessiontotheEU.Target=country’saccesstotheEU,senVment=posiVve
ExamplesfromBreck&Cardie
![Page 12: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/12.jpg)
Datasets(Sem-evaldatasetsalsoused)
2000sentencesineachcorpus
12
Corpus AverageWordCount
AverageCharacterCount
Subjec6vePhrases
Objec6vePhrases
VocabularySize
CharacterLengthRestric6ons
LiveJournal 14.67 66.47 3035(39%) 4747(61%) 4747 30-120
MPQA 31.64 176.68 3325(41%) 4754(59%) 7614 none
Twimer 25.22 118.55 2091(36%) 3640(64%) 8385 0-140
Wikipedia 15.57 77.20 2643(37%) 4496(63%) 4342 30-120
MPQA:extensivelyannotateddatasetbyStoyanav,CardieandWeibe2004.15opinionorientedqusVons,15factorientedquesVons.Alongwithtextspansfrom252arVcles.
RosenthalandMcKeown2013)
![Page 13: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/13.jpg)
ExampleSentences
LiveJournal iwillhavetosVcktomycanonfilmslrunVlinafewyearsicanaffordtoupgradeagain:)
MPQA ThesaleinfuriatedBeijingwhichregardsTaiwananintegralpartofitsterritoryawaiVngreunificaVon,byforceifnecessary.
Twimer RT@tashjade:That’sreallysad,CharlieRT“UnVltonightIneverrealisedhowfuckedupIwas”-CharlieSheen#sheenroast
Wikipedia PerhapsifreportedcriVcallybyawesternsourcebutcertainlynotbyanIsraelisource.
13
SubjecVve ObjecVve
![Page 14: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/14.jpg)
SentimentLexicons• GeneralInquirer• SenVWordNet• DicVonaryofAffect(DAL)
![Page 15: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/15.jpg)
DictionaryofAffectinLanguage• DicVonaryof8742wordsbuilttomeasuretheemoVonalmeaningoftexts• Eachwordisgiventhreescores(scaleof1to3)• pleasantness-alsocalledevaluaVon(ee)• acVveness(aa)• andimagery(ii)
C.M.Whissel.1989.Thedic6onaryofaffectinlanguage.InR.PlutchikandH.Kellerman,editors,EmoVon:theoryresearchandexperience,volume4,London.Acad.Press.
15
![Page 16: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/16.jpg)
Emoticons• 1000emoVconsweregatheredfromseverallistsavailableontheinternet• Wekeptthe192emoVconsthatappearedatleastonceandmappedeachemoVcontoasingleworddefiniVon
16
![Page 17: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/17.jpg)
Methods• Pre-processingsteps• EmoVconkeysandcontracVonexpansion• Chunkerandtagger*• LexicalFeatures*• SyntacVcFeatures*• SocialMediaFeatures
*ApoorvAgarwal,FadiBiadsy,andKathleenR.McKeown.2009.Contextualphrase-levelpolarityanalysisusinglexicalaffectscoringandsyntac6cn-grams.InProceedingsofEACL’09
17
![Page 18: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/18.jpg)
PreprocessingLiveJournal [i]/NPsub[willhavetos6ck]/VPobj[to]/PPobj[mycanonfilmslr]/NPobj[unVl]/
PPobj[in]/PPobj[afewyears]/NPsub[i]/NPsub[canaffordtoupgrade]/VPobj[again:)]/NPsub
MPQA [Thesale]/NPsub[infuriated]/VPobj[Beijing]/NPobj[which]/NPsub[regards]/VPsub[Taiwan]/NPobj[anintegralpart]/NPsub[of]/PPobj[itsterritoryawaiVngreunificaVon,]/NPobj[by]/PPobj[force]/NPsub[if]/obj[necessary.]/sub
Twimer [RT@tashjade:]/NPobj [That]/Npobj[is]/VPsub[really]/sub[sad,]/sub[CharlieRT]/NPobj[”]/NPobj[UnVl]/PPobj[tonight]/NPsub[I]/NPsub[never]/sub[realised]/VPsub[how]/sub[fucked]/VPsub[up]/PPobj[I]/NPsub[was]/VPsub[”]/obj[-CharlieSheen#sheenroast]/NPobj
Wikipedia [Perhaps]/sub[if]/obj[reported]/VPsub[cri6cally]/sub[by]/PPobj[awesternsourcebut]/NPsub[certainlynot]/sub[by]/PPobj[anIsraelisource.]/NPsub
18Xuan-HieuPhan,CRFChunker:CRFEnglishPhraseChunkerhmp://crfchunker.sourceforge.net/,2006
![Page 19: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/19.jpg)
LexicalFeatures• POSTags*• N-grams*• Performedchi-squarefeatureselecVononthen-grams
*ApoorvAgarwal,FadiBiadsy,andKathleenR.McKeown.2009.Contextualphrase-levelpolarityanalysisusinglexicalaffectscoringandsyntac6cn-grams.InProceedingsofEACL’09
19
![Page 20: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/20.jpg)
SyntacticFeatures• Usethemarkedupchunkstoextractthefollowing:*• n-grams:1-3words• POS:NP,VP,PP,JJ,other• PosiVon:target,right,lev• SubjecVvity:subjecVve,objecVve• Minandmaxpleasantness
*ApoorvAgarwal,FadiBiadsy,andKathleenR.McKeown.2009.Contextualphrase-levelpolarityanalysisusinglexicalaffectscoringandsyntac6cn-grams.InProceedingsofEACL’09
20
![Page 21: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/21.jpg)
SocialMediaFeatures
21
![Page 22: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/22.jpg)
SingleCorpusClassiLication
BalancedUnbalanced
• LogisVcRegressioni
• 10runsof10-foldcross-validaVon
• StaVsVcalsignificanceusingthet-testwithp=.001
22
Experiment LiveJournal MPQA Twimer Wikipedia
n-gramsize 100 2000 none none
majority 58% 59% 64% 63%
JustDAL 76.5% 75.7% 83.6% 80.4%
DicVonaries+SM 77.1% 76.1% 84% 81.4%
Wordnet 76.7% 75.6% 84% 80.7%
Wordnet+SM 77.1% 76.1% 84.2% 81.4%
DicVonaries 76.6% 75.7% 83.9% 80.7%
SM 77% 76.1% 83.7% 81.2%
Experiment LiveJournal MPQA Twimer Wikipedia
n-gramsize 100 200 none none
majority 50% 50% 50% 50%
JustDAL 74.7% 75.7% 81.9% 79.3%
DicVonaries+SM 76.7% 76.2% 82.6% 80.2%
Wordnet 75.1% 75.8% 82.4% 79.1%
Wordnet+SM 76.6% 75.3% 82.6% 80.3%
DicVonaries 75.3% 75.8% 82.4% 79.1%
SM 76.2% 76.3% 82.2% 80.4%
![Page 23: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/23.jpg)
SocialMediaErrorAnalysis• Wikipedia• PunctuaVonwasusefulasafeaturefordeterminingthataphraseisobjecVveifitisasmallphrase.However,severalsubjecVvephraseswereincorrectlyclassifiedbecauseofthis 23
0%
20%
40%
60%
80%
100%
Wikipedia
subjecVveobjecVve
![Page 24: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/24.jpg)
SocialMediaErrorAnalysis• Twimer• EllipseshelpindicatethatasentenceisobjecVve.Theaccuracyimprovedfrom82%to92%forsentenceswiththisfeature• AllothersocialmediafeatureswereincorrectlyclassifiedasobjecVve/subjecVvedependingonthesocialmediapreference. 24
0%
20%
40%
60%
80%
100%
TwiZer
subjecVveobjecVve
![Page 25: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/25.jpg)
SocialMediaErrorAnalysis• LiveJournal• OutofVocabularywordsandpunctuaVonwerethemostusefulsocialmediafeatures.• InalldatasetsthepunctuaVonfeaturecausedcloseto50/50exchangebutthefeaturewasbestinLiveJournal. 25
0%
20%
40%
60%
80%
100%
Livejournal
subjecVveobjecVve
![Page 26: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/26.jpg)
NeuralNetworkApproachestoSentiment• TakeastandardRNN• Takealabeleddataset(e.g.,IMDBsenVmentdataset)• IniValizewithpre-trainedwordembeddings(wordtovecorglove)• UsesigmoidtopredictbinarysenVmentlabels:posiVvevsnegaVve.
![Page 27: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/27.jpg)
Languageismadeupofsequences• Sofarwehaveseenembeddingsforwords• (andmethodsforcombiningthroughvectorconcatenaVonandarithmeVc)
• Buthowcanweaccountforsequences?• Wordsassequencesoflemers• Sentencesassequencesofwords• Documentsassequencesofsentences
![Page 28: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/28.jpg)
RecurrentNeuralNetworks• Representarbitrarilysizedsequencesinfixed-sizevector• GoodatcapturingstaVsVcalregulariVesinsequences(ordermamers)• IncludesimpleRNNs,Longshort-termmemory(LSTMs),GatedRecurrentUnit(GRUs)
![Page 29: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/29.jpg)
[Thangetal.2013] [Bowmanetal.2014]
Learningwordmeaning Logicalentailmentfromtheirmorphsusingcompositional
semanticsviaRNNs
![Page 30: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/30.jpg)
MachineTranslation(Sequences)
• Sequence-to-sequence• Sutskeveretal.2014
![Page 31: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/31.jpg)
RNNAbstraction• RNNisafuncVonthattakesanarbitrarylengthsequenceasinputandreturnsasingledoutdimensionalvectorasoutput• Input:x1:n=x1x2…xn(xiεRd-in)• Output:ynεRd-out
OOutputvectoryusedforfurtherpredicVon
![Page 32: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/32.jpg)
RNNCharacteristics• CancondiVonontheenVresequencewithoutresorVngtotheMarkovassumpVon• Cangetverygoodlanguagemodelsaswellasgoodperformanceonmanyothertasks
![Page 33: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/33.jpg)
RNNsaredeLinedrecursively• BymeansofafuncVonRtakingasinputastatevectorhi-1andaninputvectorxi• Returnsanewstatevectorhi• ThestatevectorcanbemappedtoanoutputvectoryiusingasimpledeterminisVcfuncVon• AndfedthroughsovmaxforclassificaVon.
![Page 34: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/34.jpg)
wx
wh
RecurrentNeuralNetworks
x1
h0
h1
ℎ↓𝑡 =𝜎( 𝑊↓ℎ ℎ↓𝑡−1 + 𝑊↓𝑥 𝑥↓𝑡 )
σ
SlidefromRadev
![Page 35: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/35.jpg)
wx
wh
RNN
x1
h0
h1
ℎ↓𝑡 =𝜎(𝑊↓ℎ ℎ↓𝑡−1 + 𝑊↓𝑥 𝑥↓𝑡 )𝑦↓𝑡 =𝑠𝑜𝑓𝑡𝑚𝑎𝑥( 𝑊↓𝑦 ℎ↓𝑡 )
σ
y1so[maxwy
SlidefromRadev
![Page 36: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/36.jpg)
RNN
wx
wh
x1
h0
h1
σwx
wh
x2
h2
σwx
wh
x3
h3
σ
y3so[max
The cat sat
wy
SlidefromRadev
![Page 37: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/37.jpg)
UpdatingParametersofanRNN
wx
wh
x1
h0
h1
σwx
wh
x2
h2
σwx
wh
x3
h3
σ
y3so[max
The cat sat
Cost
wy
BackpropagaVonthroughVme
SlidefromRadev
![Page 38: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/38.jpg)
Example• Foreachsentenceinthetrainingcorpus,classify,comparetogoldstandardandcomputeloss,backpropagate.• Recallthatwemayusemini-batchessothatwe’renotback-propagaVngforeachexample
• Ihadinmindyourfacts,Buddy,nothers.
![Page 39: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/39.jpg)
RNN–Ihadinmindyourfacts,buddy,nothers.
wx
wh
x1
h0
h1
σwx
wh
x2
h2
σwx
wh
x3
h3
σ
y3sigmoid
I had in
wx
wh
x3
h3
σ
mind …
Inthisoverview,wreferstotheweightsButtherearedifferentkindsofweightsLet’sbemorespecific
![Page 40: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/40.jpg)
RNN–Ihadinmindyourfacts,buddy,nothers.
wx
U
x1
h0
h1
wx
U
x2
h2
σwx
U
x3
h3
σ
y3sigmoid
I had in
wx
U
x3
h3
σ
mind …
σ
![Page 41: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/41.jpg)
![Page 42: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/42.jpg)
![Page 43: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/43.jpg)
![Page 44: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/44.jpg)
RNN–Ihadinmindyourfacts,buddy,nothers.
wx
U
x1
h0
h1
wx
U
x2
h2
σwx
U
x3
h3
σ
y3sigmoid
I had in
wx
U
x3
h3
σ
mind …
σ
![Page 45: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/45.jpg)
RNN–Ihadinmindyourfacts,buddy,nothers.
wx
U
x1
h0
h1
wx
U
x2
h2
σwx
U
x3
h3
σ
y3sigmoid
I had in
wx
U
x3
h3
σ
mind …
ht=σ(Uwxt)ht-1
σ
![Page 46: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/46.jpg)
RNN–Ihadinmindyourfacts,buddy,nothers.
wx
wh
x1
h0
h1
σwx
wh
x2
h2
σwx
wh
x3
h3
σ
Sigmoid
I had in
wx
wh
x3
h3
σ
mind …
Y=posiVve?Y=negaVve?Finalembeddingrunthroughthesigmoid
funcVon->[0,1]1=posiVve0=negaVveOvenfinalhisusedaswordembeddingforthesentence
![Page 47: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/47.jpg)
UpdatingParametersofanRNN
wx
wh
x1
h0
h1
σwx
wh
x2
h2
σwx
wh
x3
h3
σ
y3sigmoid
Cost
wy
BackpropagaVonthroughVmeGoldlabel=0(negaVve)AdjustweightsusinggradientRepeatmanyVmeswithallexamples
SlidefromRadev
I had in
![Page 48: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/48.jpg)
ProblemwithRNN• Vanishinggradients• BytheVmeweback-propogateallthewaythroughthenetwork,theweightsapproachzero->vanishinggradient• Errorsignals(gradients)inlaterstepsdiminishquicklyanddonotreachearlierinputsignals• ->Hardtocapturelong-distancedependencies
![Page 49: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/49.jpg)
Whatisalongdistancedependency?• ThestudentswerelisteningtoKathyMcKeownspeak.• Thestudentsin451CSBuildingwerelisteningtoKathyMcKeownspeak
![Page 50: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/50.jpg)
GatedArchitectures• RNN:ateachstateofthearchitecture,theenVrememorystate(h)isreadandwrimen• Gate=binaryvectorgε{0,1}• Controlsaccesston-dimensionalvectorx�g
• Consider• Readsentriesfromxspecifiedbyg• Copiesremainingentriesfroms(orhaswe’vebeenlabelingthehiddenstate)
![Page 51: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/51.jpg)
Example:gatecopiesfromposiVons2and5intheinputRemainingelementscopiedfrommemory
![Page 52: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/52.jpg)
LSTMSolution§ UsememorycelltostoreinformaVonateachVmestep.
§ Use“gates”tocontroltheflowofinformaVonthroughthenetwork.§ Inputgate:protectthecurrentstepfromirrelevantinputs
§ Outputgate:preventthecurrentstepfrompassingirrelevantoutputstolatersteps
§ Forgetgate:limitinformaVonpassedfromonecelltothenext
[slidesfromCatherineFinegan-Dollak]
![Page 53: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/53.jpg)
TransformingRNNtoLSTM
𝑢↓𝑡 =𝜎( 𝑊↓ℎ ℎ↓𝑡−1 + 𝑊↓𝑥 𝑥↓𝑡 )
wx
wh
x1
h0
u1
σ
[slidesfromCatherineFinegan-Dollak]
![Page 54: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/54.jpg)
TransformingRNNtoLSTM
wx
wh
x1
h0
u1
σ
c0
[slidesfromCatherineFinegan-Dollak]
![Page 55: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/55.jpg)
TransformingRNNtoLSTM
𝑐↓𝑡 = 𝑓↓𝑡 ⊙ 𝑐↓𝑡−1 + 𝑖↓𝑡 ⊙ 𝑢↓𝑡
wx
wh
x1
h0
u1
σ
c0
+ c1
f1
i1
[slidesfromCatherineFinegan-Dollak]
![Page 56: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/56.jpg)
TransformingRNNtoLSTM
𝑐↓𝑡 = 𝑓↓𝑡 ⊙ 𝑐↓𝑡−1 + 𝑖↓𝑡 ⊙ 𝑢↓𝑡
wx
wh
x1
h0
u1
σ
c0
+ c1
f1
i1
[slidesfromCatherineFinegan-Dollak]
![Page 57: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/57.jpg)
TransformingRNNtoLSTM
𝑐↓𝑡 = 𝑓↓𝑡 ⊙ 𝑐↓𝑡−1 + 𝑖↓𝑡 ⊙ 𝑢↓𝑡
wx
wh
x1
h0
u1
σ
c0
+ c1
f1
i1
[slidesfromCatherineFinegan-Dollak]
![Page 58: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/58.jpg)
wx
wh
x1
h0
u1
σ
c0
+ c1
f1
i1
TransformingRNNtoLSTM
𝑓↓𝑡 =𝜎( 𝑊↓ℎ𝑓 ℎ↓𝑡−1 + 𝑊↓𝑥𝑓 𝑥↓𝑡 )f1
x1
h0
σwhf
wxf
[slidesfromCatherineFinegan-Dollak]
![Page 59: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/59.jpg)
wx
wh
x1
h0
u1
σ
c0
+ c1
f1
i1
TransformingRNNtoLSTM
𝑖↓𝑡 =𝜎( 𝑊↓ℎ𝑖 ℎ↓𝑡−1 + 𝑊↓𝑥𝑖 𝑥↓𝑡 )
x1
h0
σwhi
wxi
i1
[slidesfromCatherineFinegan-Dollak]
![Page 60: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/60.jpg)
TransformingRNNtoLSTM
wx
wh
x1
h0
u1
σ
c0
+ c1
f1
i1h1
tanh
o1
ℎ↓𝑡 = 𝑜↓𝑡 ⊙ tanh𝑐↓𝑡
[slidesfromCatherineFinegan-Dollak]
![Page 61: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/61.jpg)
LSTMforSequences
wx
wh
x1
h0
u1
σ
c0
+c1
f1
i1h1
tanh
o1wx
wh
x2
u2
σ
+c2
f2
i2h2
tanh
o2wx
wh
x2
u2
σ
+c2
f2
i2h2
tanh
o2
The cat sat
[slidesfromCatherineFinegan-Dollak]
![Page 62: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/62.jpg)
Bi-LSTM for sentiment!• Pre-trained Word Embeddings !!
Word Embeddings
..LSTM LSTMLSTMLSTM
p(x)
Feedforward
LSTM
x1
LSTM
x2
LSTM
x3
LSTM
xn
..
Right LSTM Left LSTM
Average
+
SoftmaxSentiment predictions
BiLSTM ..
62
![Page 63: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/63.jpg)
Bi-LSTM enriched with sentiment!word embeddings !!
Sentiment Word Embeddings
..BiLSTM
..
LSTM LSTMLSTMLSTM
p(x)
Feedforward
LSTM
x1
LSTM
x2
LSTM
x3
LSTM
xn
..
Right LSTM Left LSTM
Average
SoftmaxSentiment predictions
Vsentiment(x1)Bilingual Sentiment Scores
Vsentiment(x2) Vsentiment(x3) Vsentiment(xn)
63
![Page 64: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/64.jpg)
RecursiveDeepModelsforSemanticCompositionalityoveraSentimentTreebank• Socheretal,Stanford2013hmps://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf• Problemwithpreviouswork:difficultyexpressingthemeaningoflongerphrases• Goal• TopredictsenVmentatthesentenceorphraselevel• CaptureeffectofnegaVonandconjuncVons• SenVmentTreebank• RecursiveNeuralTensorNetwork
![Page 65: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/65.jpg)
SentimentTreebank• Moviereviewexcerptsfromromentomatoes.com(Pang&Lee2005)• 10,662sentences• ParsedbyStanfordParser(Klein&Manning2003)• 215,154phrases• EachphraselabeledforsenVmentusingAmazonMechanicalTurk(AMT)• 5classesemerge:negaVve,somewhatnegaVve,neutral,somewhatposiVve,posiVve
![Page 66: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/66.jpg)
Example--verynegaVve++veryposiVve- NegaVve+posiVve0neutral
![Page 67: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/67.jpg)
RecursiveNeuralModels
![Page 68: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/68.jpg)
RNN:RecursiveNeuralNetwork
WaretheweightstolearnWεf=tanh
![Page 69: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/69.jpg)
MV-RNNMatrixvectorRNN• Introduceweightmatrixassociatedwitheachnon-terminal(P2foradjP)andterminal(Afora)• a=not,b=very,c=good
![Page 70: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/70.jpg)
RNTN:RecursiveNeuralTensorNetwork• TheMV-RNNhastoomanyparameterstolearn(sizeofvocabulary)• CanwegetcomposiVonalitywithreducedparameters?• P1=f([ab]u1u2a)u3u4b=f([ab]u1a+u2b)u3a+u4b
=f(u1aa+u2ab+u3ab+u4bb)
![Page 71: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/71.jpg)
Results
![Page 72: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/72.jpg)
Positive–“mostcompelling”
![Page 73: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/73.jpg)
Negative–“leastcompelling”
![Page 74: Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy, and Kathleen R. McKeown. 2009. Contextual phrase-level polarity analysis using](https://reader035.vdocuments.us/reader035/viewer/2022062506/5f869722d721a4374a6e601e/html5/thumbnails/74.jpg)
HandlingConjunctions