Orphée DeClercqHUSO2016- EMOSEDENovember15,2016

Fine-grainedSentimentAnalysisAnOverviewoftheTaskanditsMainChallenges

AFFORDABLEFOODFRIENDLYSERVICE

Sentimentanalysis˚Early2000s:Wiebe(2000)Pangetal.(2002)…è newswiretext

RiseofWeb2.0applications2010-2016:• 20,000GoogleScholar• 731papersinWoSè user-generatedcontent

Sentimentanalysis• Opinionpolls,surveys• SentimentanalysisonUGC:

Ø Totrackhowabrandisperceivedbyconsumers(Zabin &Jefferies,2008)

Ø Formarket(Sprenger etal.,2014),electionprediction(Bermingham &Smeaton,2011)

Ø Todeterminethesentimentoffinancialbloggerstowardscompaniesandtheirstocks(O’Hareetal.,2009)

Ø Byindividualswhoneedadviceonpurchasingtherightproductorservice(Dabrowski etal.,2010)

Ø Bynonprofitorganizations,e.g., forthedetectionofsuicidalmessages(Desmet,2014)

Ø …

SentimentanalysisCoarse-grained:documentorsentence=POS |NEG | NEUTRAL

àDoesnotallowtodiscoverwhatpeople likeanddislikeexactly.

àNotonlyinterested ingeneralsentimentaboutacertainproduct,butalsointheiropinionsaboutspecificfeatures,partsorattributesofthatproduct.

Fine-grained: “almostallreal-lifesentimentanalysissystemsinindustryarebasedonthislevelofanalysis”(Liu,2015,p.10).

ABSAAspect-based(orfeature-based)sentimentanalysissystems

focusonthedetectionofallsentimentexpressionswithinagivendocumentandtheconceptsandaspects (orfeatures)towhich

theyrefer.

• VanHee etal.(2014):Coarse-grainedSAonTwitter• DeClercq etal.(2015):ABSA(Englishresto)• DeClercq (2015):SemEval ABSA(Dutchresto)• DeClercq andHoste (2016):ABSA(Dutchresto,smartphones)• Pontiki etal.(2016):SemEval ABSA8languages,4domains• 2016-2017:valorisation project(variousdomains,languages)

ABSAThebestresearch=teamresearch

Overview① Introduction

② TaskDefinition

③ DatasetsandAnnotation

④ Subtasks

Ø AspectTermExtraction

Ø AspectTermCategorization

Ø AspectTermPolarityClassification

⑤ Challenges

⑥ Conclusion

Overview① Introduction

② TaskDefinition

③ DatasetsandAnnotation

④ Subtasks

Ø AspectTermExtraction

Ø AspectTermCategorization

Ø AspectTermPolarityClassification

⑤ Challenges

⑥ Conclusion

TaskdefinitionThereexistmanyreference works(Pang&Lee,2008,Liu2012,Liu2015):DefinitionofanopinionbyLiu(2012):

“Anopinionisaquintuple,(ei;aij ;sijkl;hk;tl),whereei isthenameofanentity,aij isanaspect ofei,sijkl isthesentimentonaspectaij ofentity

ei,hk istheopinionholder,andtl isthetimewhentheopinionisexpressedbyhk.Thesentimentsijk ispositive,negative,orneutral,or

expressedwithdifferentstrength/intensitylevels.”(pp.19-20)

è Automaticallyderivingquintuples=fivedifferenttasks

Taskdefinition

1.Entityextraction+ categorization

Extractallentityexpressionsinadocumentcollection,andcategorizeorgroupsynonymousentityexpressionsintoentityclusters.

ENTITYCATEGORY

2.Aspectextraction+categorizationExtractallaspectexpressionsoftheentities,andcategorizetheseaspectexpressionsintoclusters.Theseaspectscanbebothexplicitandimplicit.

AmbienceFood Service

Restaurant

Extractopinionholdersforopinionsfromtextorstructureddataandcategorizethem.

3.Opinionholderextraction+categorization

Extractthetimeswhenopinionsaregivenandstandardizedifferenttimeformats

4.Timeextraction+standardization

5.AspectsentimentclassificationDeterminewhetheranopiniononanaspectispositive,negativeorneutral,orassignanumericsentimentratingtotheaspect.

AmbienceFood Service

Restaurant

J J

J

J

Taskdefinition

Derivedquintuples:• (Uma,Food,positive,ReviewerX,May-31-2016)• (Uma,Ambience, positive,ReviewerX,May-31-2016)• (Uma,Service,positive,ReviewerX,May-31-2016)• (Uma,Restaurant,positive,ReviewerX,May-31-2016)

• (Uma,Food,positive,ReviewerX,May-31-2016)• (Uma,Ambience, positive,ReviewerX,May-31-2016)• (Uma,Service,positive,ReviewerX,May-31-2016)• (Uma,Restaurant,positive,ReviewerX,May-31-2016)

èABSAofcustomerreviews:Ø AspectExtractionØ AspectCategorizationØ Aspectsentimentclassification

Taskdefinition:customerreviews

SemEval TaskDescription(Pontiki etal.,2014,

2015,2016)

META-DATA

Overview① Introduction

② TaskDefinition

③ DatasetsandAnnotation

④ Subtasks

Ø AspectTermExtraction

Ø AspectTermCategorization

Ø AspectTermPolarityClassification

⑤ Challenges

⑥ Conclusion

CustomerreviewsPreviousresearchMoviereviews(Thet etal.2010),electronicproducts(HuandLiu2004,BrodyandElhadad 2010),restaurants(Ganu etal.2009).

è Difficulttocompare

SemEval sharedtaskOnlinedatacompetition:everyoneworksonthesamedata.

èBettertocompareèStateoftheart

SemEval benchmarkdataèThreerunsofthetask(2014,2015&2016)è Lotsofdataindifferentdomains&languages

AnnotationGuidelinesareavailableonline:http://goo.gl/wOf1dX

Threesteps:

I.Allexplicitandimplicittargets-thewordorwordsreferringtoaspecificentityoraspect- areannotated.II.Thesetargetsareassignedtodomain-specificpredefinedclustersofaspectcategories.III.Sentimentexpressedtowardseveryaspectisindicated.

Annotation

ExperimentaldataTrainandtestsplithavebeencreatedforallSemEvaldatasets

èFocusonDutch(restaurantreviews)300reviewsfortraining(development)100reviewsfortesting(held-out)

èExplainthepipelinewedevelopedèStateoftheartapproachesandresultsonEnglish(restaurantreviews)

Overview① Introduction

② TaskDefinition

③ DatasetsandAnnotation

④ Subtasks

Ø AspectTermExtraction

Ø AspectTermCategorization

Ø AspectTermPolarityClassification

⑤ Challenges

⑥ Conclusion

ASPECT TERM EXTRACTION

ASPECT CATEGORY CLASSIFICATION

Term Extraction with TExSIS

Lexical• Bag-of-words

Features for category classification

Preprocessing(LeTs)

TermhoodUnithood

AdditionalFiltering

Features for polarity classification

Subjectivity Heuristic

Semantic• Cornetto• DBpedia• Semantic roles

ASPECT POLARITY CLASSIFICATION

Lexical• Token and character n-grams• Sentiment lexicons• Word-shape

PipelineforDutch:overviewTastypaella,butrudewaiter.

paella à FOOD_qualitywaiterà SERVICE_general

FOOD_qualitySERVICE_general

AspectTermExtractionExtract allaspectexpressionsoftheentities.

Subjectivity Heuristic

Term Extraction with TExSISPreprocessing

(LeTs)TermhoodUnithood

AdditionalFiltering

Onlywhensubjective!Lexicons• Pattern(DeSmedt &Daelemans,

2012)• Duoman (Jijkoun &Hofman,2009)

TExSIS =hybridsystemcombininglinguisticandstatisticalinformation(Macken etal.,2013)

Linguistic =whichwords?• PreprocessingusingLeTs (VandeKauter etal.,2013)• PoS patterns(i.e.nouns,nounphrases)

Statistical =aretheyterms?• Termhood,unithood measures (LL,c-value)

Additionalfiltering…

AspectTermExtractionTExSISoutput:

Aftera[goodappetizer]our[mother]ordereda[pizzamargherita],whichwasdivine!

…Additionalfiltering• Subjectivity(basedonsamelexicons)• Semantic

– Cornetto(Vossen etal.2013):synsets lookforhypernym-synonymlinks.

– DBPedia (Mendesetal.2011):tagtermswithDBPedia Spotlightandlookforcategories.

AspectTermExtraction

Additionalfilteringoutput:Afteragood[appetizer]ourmotherordereda[pizza

margherita],whichwasdivine!

AspectTermExtractionResultsTrainingdatasplitindevtrain (250)anddevtest (50)Bestsettingonheld-outtestset(100).Evaluationmetrics:precision,recallandF-1

AspectTermExtractionStateoftheartEnglishSupervisedmachinelearningapproachesmostsuccessfulSequential labelingtask(IOB2annotation~NER)

Toh andSu(2016)=topsystem• CRFclassifier• NEfeatures• Additional featuresfromRNN(Liu,Joty &Meng,2015)• 72.34

AspectTermCategorizationCategorizeallextractedaspectexpressions.

Classificationtask• Predefinedcategories• Multiclassproblem:

Ø MaincategoriesØ Subcategories

Lexical• Typicalbag-of-words: tokenunigram

Lexico-semantic• Cornetto(insynset orhypernym/hyponymofmaincats)• DBPedia (belongtouniquecategories)

Semanticroles• Termevokessemanticrole,whichrole(Thefoodtasted goodvsThefoodjustcosttoomuch)

ASPECT CATEGORY CLASSIFICATION

Features for category classification

Lexical• Bag-of-words

Semantic• Cornetto• DBpedia• Semantic roles

AspectTermCategorizationResultsTen-foldcrossvalidationontrainingdata.LibSVMRound1:graduallyaddingmorefeaturesRound2:jointoptimization,featuregroupsvsindividualfeaturesBestresultsonheld-outtestAccuracy

AspectTermCategorizationStateoftheartEnglishSupervisedmachinelearningapproachesmostsuccessful

Toh andSu(2016)=topsystem• Individualbinaryclassifierstrainedoneachcategory

(combined)• Lexicalbagofwords(unigram,bigram)• Lexical-semantic:clusterslearnedfromlargereferencecorpus• Additional featuresfromCNN(Severyn&Moschitti,2015)• 73.031

AspectPolarityClassificationDeterminewhetheropinionisPOS |NEG |NEUTRAL

Three-wayclassification

Tokenandcharactern-gramfeaturesunigram,bigramandtrigram(tok)&trigram,fourgram (char)

Sentimentlexicon• DuoMan andPatternlexicon,matchespos,neg,neut

Word-shape• UGCcharacteristics,character ofpunctuationflooding

(cooooool!!!!!),lasttokenhaspunct,capitalizedtokens

Features for polarity classification

Lexical• Token and character n-grams• Sentiment lexicons• Word-shape

AspectPolarityClassificationResultsTen-foldcrossvalidationontrainingdata.LibSVMDefault:allfeaturesJointoptimization:individualfeatureselectionBestresultsonheld-outtestsetAccuracy

AspectPolarityClassificationStateoftheartEnglishSupervisedmachinelearningapproachesmostsuccessful

Brun,Perez&Roux(2016)=topsystem• Ensembleclassifiers• Syntacticparser=basicfeatures (prepro +NER+syntax)• Semanticcomponentadded(basedondesignatedpolarity&

semanticlexicons)• 88.126

ABSAè AcceptableresultsforEnglishonallthreesubtasks.èDutch:subtasks1and2stillquitechallengingèSametrueforother languagesorotherdomains!!

Note:Inreality,thesearenotseparatetasksè errorpercolation

e.g.forDutchpolarityclassification,accuracydropsto39.70

Overview① Introduction

② TaskDefinition

③ DatasetsandAnnotation

④ Subtasks

Ø AspectTermExtraction

Ø AspectTermCategorization

Ø AspectTermPolarityClassification

⑤ Challenges

⑥ Conclusion

DomainadaptationFocusonconsumerreviews• Product-oriented• Aspectexpressions:nounsornounsphrases• Willalmostalwaysincludeanopinion

Inreality• Non-opinionatedtextco-occurswithopinionatedtext(skewed)

• Verbalexpressionsoravarietyofwordscanbeusedtorefertocertainaspects.E.g.politicaltweets,discussionforums,…

User-generatedcontent

• Differentfromstandardtext.• Highlyexpressive:emoticons,flooding(cooool!!)BUT• Fullofmisspellings,grammaticalerrors,abbreviations,…è hinderstandardNLPtools.

Itssokeeeeewl, lol

è polarityclassification:importanceoflexicalfeatures

User-generatedcontent• Normalization(VanHee etal.,underreview)

è Helps,especiallyforunseendata

CreativelanguageuseItwassoniceofmydadtocometomygraduationparty#notGoingtothedentistforarootcanal.Yay,can’twait!!!!

• Sarcasm,irony,humour andmetaphor.• NLP=difficulttointerpretthis

è Interestingresearchemerging.SemEval 2015taskonirony(Ghoshetal.,2015),howevertoomuchfocusonhashtags.VanHee etal.(2016)proposealternative:papertoappearatCOLING2016.

Requiresdeepunderstanding“Sentiment analysisrequiresadeepunderstandingoftheexplicitandimplicit,regularandirregular,andsyntacticandsemantic

languagerules.”(Cambriaetal.,2013)

• Explicitsentiment:seemseasybutwordsareneverusedinisolation– Negation,modifiers(intensifiers,diminishers,…)è crucial!

• Implicitsentiment:morecomplex,readbetweenthelines.Evenfactualstatementscanevokedifferentopinions.

• Coreference:crucialbutnotmuchresearch.

Overview① Introduction

② TaskDefinition

③ DatasetsandAnnotation

④ Subtasks

Ø AspectTermExtraction

Ø AspectTermCategorization

Ø AspectTermPolarityClassification

⑤ Challenges

⑥ Conclusion

ConclusionWhatisaspect-basedsentimentanalysis?

• Taskdefinition• Benchmarkdatasets(SemEval)• Stateoftheartapproaches(customerreviews)• Challenges

(AB)SAisfarfromsolved

MuchmoretoberesearchedMaybeweshouldcooperate

J

Orphée DeClercqorphee.declercq@ugent.be

https://www.lt3.ugent.be/people/orphee-de-clercq/@OrfeeDC

ReferencesBermingham,A.,&Smeaton,A.F.(2011).Onusingtwittertomonitorpoliticalsentimentandpredictelectionresults.Psychology,2-10.Brody,S.&Elhadad,N.(2010).Anunsupervisedaspect-sentiment model foronlinereviews,ProceedingsofNAACL-2010,pp.804-812.Brun,C.,Perez,J.& Roux,C.(2016).XRCEatSemEval-2016Task5:Feedbacked EnsembleModelingonSyntactico-SemanticKnowledgeforAspectBasedSentimentAnalysis,ProceedingsofSemEval-2016,pp.277–281.Cambria,E.,Schuller,B.,Xia,Y.& Havasi,C.(2013).Newavenuesinopinionminingandsentiment analysis, IEEEIntelligentSystems,vol.28,no.2,2013,pp.15–21.Dabrowski,M.,Acton,T.,Jarzebowski,P.&O’Riain,S.(2010).Improvingcustomerdecisionsusingproductreviews- CROM- CarReviewOpinionMiner, inProceedingsofWEBIST-2010,pp.354–357.DeClercq,O.,VandeKauter,M.,Lefever,E.,&Hoste,V.(2015).Applying hybridterminology extraction toaspect-based sentiment analysis. Proceedings of SemEval2015,pp.719–724.DeClercq,O.(2015). Tippingthe scales:exploring the added value of deep semanticprocessing on readability prediction andsentiment analysis.PhD,GhentUniversity.

DeClercq,O.,&Hoste,V.(2016).Rude waiter but mouthwatering pastries!Anexploratory study into Dutchaspect-based sentimesnt analysis, Proceedings of LREC2016, pp.2910–2917.DeSmedt,T.andDaelemans,W.(2012).Vreselijk mooi!Terribly beautiful:asubjectivitylexicon forDutchadjectives,Proceedings of LREC-2012,pp.3568-3572.Desmet,B.(2014)“Findingtheonlinecryforhelp:automatictextclassificationforsuicideprevention,”PhD,GhentUniversity.Ganu,G.,Elhadad,N.& Marian,A.(2009).Beyondthestars:improvingratingpredictionsusingreviewtextcontent,ProceedingsofWebDB-2009,pp.1-6.Ghosh,A.,Li,G.,Veale,T.,Rosso,P,Shutova,E.,Barnden,J.&Reyes,A.(2015).Semeval-2015task11:Sentimentanalysisoffigurativelanguageintwitter,inProceedingsofSemEval 2015,pp.470–478.Hu,M.andLiu,B.(2004).Miningandsummarizingcustomerreviews,ProceedingsofKDD-2004,pp.168-177.Jijkoun,V.andHofmann,K.(2009).Generatinganon-Englishsubjectivity lexicon:Relationsthatmatter,ProceedingsEACL-2009,pp.398-405.Liu,B.(2012).Sentimentanalysisandopinionmining,SynthesisLecturesonHumanLanguageTechnologies, vol.5,no.1,pp.1–167.

Liu,B.(2015)SentimentAnalysis - MiningOpinions,Sentiments,andEmotions.CambridgeUniversityPress.Liu,P.,Joty,S.&Meng,H.(2015).Fine-grainedopinionminingwithrecurrentneuralnetworksandwordembeddings,ProceedingsEMNLP-2015,pp.1433–1443.O’Hare,N.,Davy,M.,Bermingham,A.,Ferguson,P.Sheridan,P.Gurrin,C.&Smeaton,A.F.(2009).Topic-dependentsentiment analysisoffinancialblogs,ProceedingsofTSA-2009,pp.9–16.Macken,L.,Lefever,E.&Hoste,V.(2013).TExSIS:BilingualTerminologyExtractionfromParallelCorporaUsingChunk-basedAlignment,Terminology 19(1),pp.1-30.Mendes,P.N.,Jakob,M.,Garca-Silva,A.&Bizer,C.(2011).DBpedia Spotlight:Sheddinglightonthewebofdocuments,Proceedingsofthe7thInternationalConferenceonSemanticSystems(I-Semantics-2011),pp.1-8.Pang,B.,Lee,L.&Vaithyanathan,S(2002).Thumbsup?:Sentimentclassificationusingmachine learningtechniques,ProceedingsofEMNLP-2002,pp.79-86.Pang,B.&Lee,L.(2008).Opinionminingandsentiment analysis,FoundationsandTrendsinInformationRetrieval,vol.2,no.1-2,pp.1–135.

Pontiki,M.,Galanis,D.,Pavlopoulos,J.,Papageorgiou,H.,Androutsopoulos I.&Manandhar,S.(2014).Semeval-2014task4:Aspectbasedsentimentanalysis,ProceedingsofSemEval-2014,pp.27–35.Pontiki,M.,Galanis,D.,Papageorgiou,H.,Manandhar,S.&Androutsopoulos,I.(2015).Semeval-2015task12:Aspectbasedsentimentanalysis,ProceedingsofSemEval-2015,pp.486–495.Pontiki,M.,Galanis,D.,Papageorgiou,H.,Androutsopoulos,I.,Manandhar,S.,AL-Smadi,M.,Al-Ayyoub,M.,Zhao,Y.,Qin,B.,DeClercq,O.,Hoste,V.,Apidianaki,M.,Tannier,X.,Loukachevitch,N.,Kotelnikov,E.,Bel,N.,Jiménez-Zafra,S.M.&Eryiğit,G.(2016).Semeval-2016task5:Aspectbasedsentiment analysis,ProceedingsofSemEval-2016,pp.19–30.Severyn,A.&Moschitti,A.(2015)UNITN:TrainingDeepConvolutionalNeuralNetworkforTwitterSentimentClassification, ProceedingsofSemEval-2015,pp.464–469.Sprenger,T.O.,Tumasjan,A.,Sandner P.G.&Welpe, I.M.(2014).Tweetsandtrades:Theinformationcontentofstockmicroblogs,EuropeanFinancialManagement,vol.20,no.5,pp.926–957.Thet,T.T.,Na,J.-C.andKhoo,C.S.(2010).Aspect-basedsentiment analysisofmoviereviewsondiscussionboards,JournalofInformationScience36(6),823-848.Toh,Z.&Su,J.(2016).NLANGPatSemEval-2016Task5:ImprovingAspectBasedSentimentAnalysisusingNeuralNetworkFeatures,ProceedingsofSemEval2016,pp.282–288.

VandeKauter,M.,Coorman,G.,Lefever,E.,Desmet,B.,Macken,L.&Hoste,V.(2013),LeTs Preprocess:Themultilingual LT3linguisticpreprocessingtoolkit,ComputationalLinguistics intheNetherlandsJournal3,103- 120.VanHee,C.,VandeKauter,M.,DeClercq,O.,Lefever,E.,&Hoste,V.(2014).LT3:sentiment classification inuser-generated content using arich feature set,Proceedingsof SemEval 2014, pp.406–410.VanHee,C.,Lefever,E.,&Hoste,V.(2016).Exploring the realization of irony inTwitterdata,Proceedings LREC 2016,pp.1795–1799.VanHee,C.,Lefever,E.,&Hoste,V.(2016).Mondaymorning are myfave :)#not.Exploring the automatic recognition of irony inEnglishtweets.Toappear atCOLING2016.Vossen,P.(1998)EuroWordNet:amultilingual databasewithlexicalsemanticnetworksforEuropeanLanguages,Kluwer,Dordrecht.Zabin,J.&Jefferies,A.(2008).Socialmediamonitoringandanalysis:Generatingconsumerinsightsfromonlineconversation,AberdeenGroupBenchmarkReport,AberdeenGroup,Tech.Rep.

SurveypaperinHUSOProceedings:DeClercq,O.(2016).TheManyAspectsofFine-GrainedSentimentAnalysis:AnOverviewoftheTaskanditsMainChallenges,ProceedingsoftheInternationalConferenceonHumanandSocialAnalytics (HUSO).