LISPNYCMay12,2015

WhatisaWiseCrowd?•  SirFrancisGalton,Naturear=cle1907,VoxPopuli–  Guesstheweightoftheox?–  Takethemedianofguessesfromacrowd

•  Applieswellto–  Quan=tyes=mates–  Spa=alreasoning(trafficflow)

•  JamesSurowiecki,2004,TheWisdomofCrowds

1.  Independentanswers2.  Diverseexper=se3.  Local(decentralized)accesstoknowledge4.  Amethodtoaggregatetheanswers

5/12/16 LispNYC 1

WhyDoContentAssessment?•  Peopleselectcontentformanypurposes–  Students,towriteessays–  Journalists,toreportbreakingevents–  Automatedinforma=onsystems,tosummarize:

•  Webpages•  Newsar=cles•  Opinionpolls•  ...

•  Peoplemakejudgmentsaboutgoodcontent

–  Toteachstudentsreadingandwri=ng–  Tochoosewhichnews/opinionsourcetotrust–  Toevaluateautomatedsummarizers

5/12/16 LispNYC 2

Example:AggregateContentModel•  Asinglebreakingnewsevent•  Fournewssources–  Independent–  Diverseexper=se(+/-medical)

–  Decentralizedknowledgeaccess–  Howtoaggregatemeaning????

•  Pyramidmethod:aprocedureto–  Aggregatecontentfrommanysources

–  Generatesemergentdis=nc=onofcontentimportance

5/12/16 LispNYC 3

FirstReportsofEbolaOutbreakinAfrica•  FourearlyU.S.reports:health(WHO,CDC)andnews(NYT,WashPost)orgs

–  Somecontentincommonamongall4–  Whatcontentemergesfromthewisecrowd?

•  WHO3/23/14:TheMinistryofHealth(MoH)ofGuineahasno=fiedWHOofarapidlyevolvingoutbreakofEbolavirusdiseaseinforestedareassoutheasternGuinea.Asof22March2014,atotalof49casesincluding29deaths(casefatalityra=o:59%)hadbeenreported.ThecaseshavebeenreportedinGuekedou,Macenta,NzerekoreandKissidougoudistricts....

•  NYT3/24/14:ThefirstoutbreakofEbolafeverintheWestAfricanna=onofGuineahaskilledatleast59peopleandmaybespreadingintonearbycountries,interna=onalhealthagencieswarnedonMonday.Guinea’sHealthMinistrysaidmostofthe80knowncasesofthediseasewereinborderareasnearSierraLeoneandLiberia;earlyreportsofthreecasesinConakry,Guinea’scapital,wereunfounded,expertssaid....

•  CDC3/25/14:AccordingtotheWorldHealthOrganiza=on(WHO),theMinistryofHealth(MoH)ofGuineahasreportedanoutbreakofEbolahemorrhagicfeverinfoursoutheasterndistricts:Guekedou,Macenta,NzerekoreandKissidougou.ReportsofsuspectedcasesintheneighboringcountriesofLiberiaandSierraLeonearebeinginves=gated....

•  WashingtonPost3/25/14:AnoutbreakofthedeadlyEbolavirusisbelievedtohavekilledatleast59peopleinGuineaandmayalreadyhavespreadtoneighboringLiberia,healthofficialssaidMonday.HealthworkersinGuineaaretryingtocontainthespreadofthedisease,whichcausessevereinternalbleeding.InLiberia,healthofficialssaidtheyareinves=ga=ngfivedeathsakeragroupofpeoplecrossedtheborderfromGuineainsearchofmedicaltreatment.TheEbolavirusleadstoseverehemorrhagicfeverinitsvic=msandhasnovaccineorspecifictreatment....

5/12/16 LispNYC 4

EmergentContent:Weightedby“Votes”•  4of4votes–  AnoutbreakofEbolaoccurredinsoutheasternGuinea–  AsoflateMarch,therewere30-60deaths

•  3of4votes–  Therewereover80suspectedcases–  EbolaisspreadingtoLiberiaandSierraLeone

•  2of4votes–  Ebolastartedin4districtsofGuinea–  Fourhealthcareworkerswereearlyvic=ms

–  ...•  1of4votes–  Ebolahasnovaccine–  ....5/12/16 LispNYC 5

Assessment:WeightedContent10ContentUnitswith�2votes;total=26points•  CDC:16points(61.5%)•  WHO:22points(84.6%)•  NYT:22points(84.6%)•  WashingtonPost:10points(38.5%)

5/12/16 LispNYC 6

Example:MiddleSchoolScienceText(wc=1273)

What is Matter?

Matter Matter is the “stuff” that all objects and substances in the universe are made of. Because all matter takes up space

(has volume) and contains a certain amount of material (has mass), all matter can be detected and measured. You can observe some types of matter easily with your senses. For example, you can see or feel things like rocks,

trees, bicycles, and different kinds of animals. And you can see and smell things like smoke from a fire. Other types of matter are a little more difficult to observe. . . .

Properties of Matter You know that a piece of cork is different from a piece of clay. Cork will break if you squeeze it hard, but clay will

flatten or bend into a new shape. If you had a scale handy, you would find that a piece of cork weighs less than a piece of clay the same size. If you dropped both objects in water, you would see that the cork floats, but the clay sinks. Characteristics like these, that help us identify or classify matter, are called properties.

All matter has both physical properties and chemical properties. . . .

Physical and Chemical Changes . . . Energy . . . Forms of Energy . . .

5/12/16 LispNYC 7

Summary:StudentfromtoptiercollegeMatterisalltheobjectsandsubstancesthattakeupspacearoundus.Mattercan

bedetectedandmeasuredbecauseitcontainsvolumeandmass.Trees,rocks,animals,dustmitesandairareallconsideredexamplesofmatter.Propertiesareusedtohelpusidentifymatter.Physicalandchemicalpropertiesarethetwomaintypesofpropertiesusedtoclassifymatter.Physicalproper=esareobservedwithoutchangingtheiden=tyoftheobject.Forexample,claycanbemoldedintodifferentshapesandsizesbutthemoldingdoesnotchangethesubstancesinclay.Chemicalproper=esofmanerarecharacterizedbytheirabilitytochangeintoanewsubstancewithdifferentproper=es.Forexample,paperburningintoashandironsabilitytorust.Thereisalsoaphysicalandchemicalchangethatexistsinmaner.Thedifferencebetweenaphysicalandchemicalchangeisthatachemicalchangewillcreateanewsubstancewhileallthechemicalproper=esinaphysicalchangewillnotchange.

5/12/16 LispNYC 8

Summary:Studentfromcommunitycollege

Manerarewhatallobjectsandsubstancesaremadeupof.Allmanerhasenergy,volumeandmass.Inordertoclassifymaner,wehavehavephysicalandchemicalproper=esarethedensity,duc=lity,malleability,boilingpoint,mel=ngpoint,electricalconduc=vitysolubility,andI'msuremuchmore.Energycanbetransferredfromoneobjectorsystemtoanother,butenergycan'tbecreatednordestroyed.Mechanicalenergyarekine=cenergyandpoten=alenergykine=cenergydealswiththeenergywithobjectwhileitsmoving.Poten=alenergyistheenergyandobjecthasbecauseofitsposi=onorshape.

5/12/16 LispNYC 9

Machine-generatedSummaryManeristhestuffthatallobjectsandsubstancesintheuniversearemadeof.Thedustmitesthatliveinyourupholsteredfurnitureandrugsareanexampleofmanerthatistoosmalltoseewiththenakedeye.Anotherexampleofmanerthat'shardtodetectisair,theinvisiblegasthatsurroundsyou.Thechartlistssomecommonphysicalproper=esofmaner.Squeezingchangestheshapeofclaybutdoesnotchangewhattheclayismadeof.Rus=ngoccurswhenironreactswithoxygentoproduceironoxide.Chemicalproper=esdescribemanerbasedonitsabilitytochangeintoanewkindofmanerwithdifferentproper=es.Twophysicalproper=esofthepaperitssizeandshapehavechanged,butnotitschemicalproper=es.Mechanicalenergyistheenergyanobjecthasbecauseofitsmo=onorposi=on.Therearetwokindsofmechanicalenergy:kine=candpoten=al.Missing:Howtoclassifymaner

5/12/16 LispNYC 10

Outline1.  HumanComprehensionModel2.  State-of-the-ArtinContentAssessment

3.  PyramidMethodandReliability4.  AutomatedPyramidMethod

5.  Educa=onalRubricsandAutoma=on

6.  ExampleRubric:MainIdeasofWhatisMa9er

7.  Correla=onofPyramidScoresandInterven=onRubric

5/12/16 LispNYC 11

Kintsh&vanDijk,1970s•  Cogni=vemodelofreadercomprehension–  Specificmentalprocessestransformproposi=onalcontentintoamentalmodel(situa=onmodel)

–  Readersdrawinferences•  Situa=onmodel

–  ≠listofallproposi=ons+inferences–  reflectedinabilitytosummarize

5/12/16 LispNYC 12

1.1.HumanComprehension

Students’SummarizationSkills•  Developovermanyyears(Brown&Day,1983;Brown,Day&Jones,

1983;Day1986;...)•  Studieddevelopmentofsummariza=onmacrorules(inspiredby

Kintsch&VanDijk)–  Dele=on:omitunimportantorredundantinforma=on

–  Superordina=on:subs=tutecategorynameforinstances

–  Selec=on:reusetopicsentencesfromsourcetext–  Construc=on:createnoveltopicsentences

5/12/16 LispNYC 13

1.1.HumanComprehension

RubricsforContentSelection•  Waystoiden=fy“important”content–  RatersAiden=fyideaunits,ratersBjudgeimportanceon4pointscale(Brown&Day,1983)

–  RatersAiden=fyideaunits,ratersBeliminate25%,ratersCeliminate50%,ratersDeliminate75%;importantifnevereliminated(Brown,Day&Jones,1983)

–  Ratersjudgeimportanceofeverysentenceinsource(Day,1986)

5/12/16 LispNYC 14

1.1.HumanComprehension

ROUGE•  Automatedcontentevalua=on•  Widelyusedforlargescaleevalua=onofmachinesummarizers

•  Method:n-gramrecall:–  Collect“reference”summaries

–  forngramsfrom1toN,countnumberofmatchesofasummaryngramtoareferencesummary

–  Normalizebynumberofngramsinreferences

5/12/16 LispNYC 15

12.State-of-the-Art

ROUGEforEducation?•  Macro-levelevalua=on:Howwelldosystemsdoonaverage?–  Scoreoneormoresystemsonmul=pletasks

–  ROUGEmacro-levelevalua=oncorrelateswithhumanresponsivenessscores

•  Micro-levelevalua=on:Howwelldoesasystemdoonagiventask?–  Scoreoneormoresystemsononetask

–  Poorcorrela=onwithhumanresponsivenessscores

5/12/16 LispNYC 16

12.State-of-the-Art

Illustration•  Sourcetext:WhatisManer?(1273words)•  Wisecrowd:5mastersstudents,TeachersCollege

•  Subjects:communitycollegestudents•  Instruc=ons:–  Answer7ques=ons:Whathelpstoclassifyma9er?

– Writeasummaryofoneortwoparagraphs...astatementmostlyinyourownwordsthatcontainstheimportantinforma=on...

5/12/16 LispNYC 17

3.PyramidMethod

SampleStudentEssays•  WiseCrowdExample:Matterisalltheobjectsandsubstancesthattakeupspacearoundus.Mattercanbedetectedandmeasuredbecauseitcontainsvolumeandmass.Trees,rocks,animals,dustmitesandairareallconsideredexamplesofmatter.Propertiesareusedtohelpusidentifymatter.Physicalandchemicalpropertiesarethetwomaintypesofpropertiesusedtoclassifymatter....

•  StudentExample(Target)

Manerarewhatallobjectsandsubstancesaremadeupof.Allmanerhasenergy,volumeandmass.Inordertoclassifymaner,wehavehavephysicalandchemicalproper=esarethedensity,duc=lity,malleability,boillingpoint,mel=ngpoint,electricalconduc=vitysolubility,andI'msuremuchmore.Energycanbetransferred...

5/12/16 LispNYC 18

3.PyramidMethod

PyramidMethod:Step11.  ConstructthecontentmodelinDUCViewGUI

1.  Collectexamplesfromexperts(thewisecrowdN=5)2.  Annotatetheexamplestoiden=fycontributorstoCUs,i.e.,sharedcontent

5/12/16 LispNYC 19

3.PyramidMethod

PyramidMethod:Step11.  ConstructthecontentmodelinDUCViewGUI

1.  Collectexamplesfromexperts(thewisecrowdN=5)2.  Annotatetheexamplestoiden=fycontributorstoCUs,i.e.,sharedcontent

5/12/16 LispNYC 20

3.PyramidMethod

4thContributor

PyramidMethod:Step11.  ConstructthecontentmodelinDUCViewGUI

1.  Collectexamplesfromexperts(thewisecrowdN=5)2.  Annotatetheexamplestoiden=fycontributorstoCUs,i.e.,sharedcontent

5/12/16 LispNYC 21

3.PyramidMethod

4thContributor5thContributor

PyramidMethod:Step11.  ConstructthecontentmodelinDUCViewGUI

1.  Collectexamplesfromexperts(thewisecrowdN=5)2.  Annotatetheexamplestoiden=fycontributorstoCUs,i.e.,sharedcontent

5/12/16 LispNYC 22

3.PyramidMethod

CULabel4thContributor5thContributor

PyramidContentModel•  Weightsrangefrom1toN,thenumberofmodelsummaries•  Weightspar==ontheCUsintoapyramid

–  EachnextweighthasfewerCUs(powerlawdistribu=on)

5/12/16 LispNYC 23

3.PyramidMethod

PyramidMethod:Step22.  Assessastudent’sessayinDUCViewGUI

1.  Iden=fymodelCUsthatoccurthestudent’sessay(target)

2.  SumtheweightsofthetargetCUs3.  Normalizethesum

5/12/16 LispNYC 24

3.PyramidMethod

PyramidMethod:Step22.  Assessastudent’sessayinDUCViewGUI

1.  Iden=fymodelCUsthatoccurthestudent’sessay(target)

2.  SumtheweightsofthetargetCUs3.  Normalizethesum

5/12/16 LispNYC 25

3.PyramidMethod

MatchsummarywordstomodelCU

Normalization•  Model(60CUs)•  Target(5CUs):Oneofeachweight

–  w(5,CU1),w(4,CU4),w(3,CU12),w(2,CU26),w(1,CU45)–  Sumtargetweights:5+4+3+2+1=15

•  Quality(Precisionanalog):HowhighlyweightedarethetargetCUs?–  Maximumsumfor5CUs=23(3×5+2×4)–  15/23=0.65

•  Coverage(Recallanalog):Howcloseisthetargettoanaveragewisemodel?

–  Average#modelCUs:26–  Maximumsumfor26CUs:88–  15/88=0.17

•  F-measure:Harmonicmeanofrecallandprecision:0.41

5/12/16 LispNYC 26

3.PyramidMethod

HowReliableisIt?Stableandconsistentmeasurement1.  Howbigshouldthewisecrowdbe?

2.  Howconsistentarepyramidsfromdifferentannotators?

3.  Howconsistentaretheassessmentsfromdifferentannotators?

4.  Howconsistentlydodifferentpyramids,withdifferentassessors,ranksummariza=onsystems?

5/12/16 LispNYC 27

3.PyramidMethod

HowManyModelSummaries?Threemethods,sameanswer:≈5Groundtruth:pyramidscoreswith9modelsCompareresultswith1..8modelswithgroundtruth•  Method1

–  3kindsoferror•  Falseequivalence•  Falsenon-equivalence•  Reversalofranking

–  Howmanymodelsun=lp(error)≤0.1?–  Answer:5

•  Method2–  Howmanymodelsun=lscorescorrelatewithgroundtruth?–  Answer:5

•  Method3–  Howmanymodelsun=lnomisrankinganypairofsummaries?–  Answer:5

5/12/16 LispNYC 28

3.PyramidMethod

Method3

5/12/16 LispNYC 29

3.PyramidMethod

AnnotatorAgreement:ModelKrippendorff’sAlpha+set-baseddistancemetric“Real”annotators(par=cipantsinDUC)

5/12/16 LispNYC 30

DUC2006 Set-BasedAgrD0608 0.84

D0615 0.81

D0624 0.83

D0629 0.89

D0640 0.75

3.PyramidMethod

AnnotatorAgreement:AssessmentKrippendorff’sAlpha+set-baseddistancemetricRecruitedannotators(non-par=cipants)

Comparisonwithexpert

5/12/16 LispNYC 31

Novice MeanAgr Range Sd

A 0.78 [0.64,0.87] 0.08

B 0.78 [0.46,0.97] 0.14

C 0.77 [0.58,0.86] 0.09

3.PyramidMethod

SystemRanking•  Comparedifferentevalua=onsofsamesummariza=onsystems–  Differentpyramids

–  Differentsetsofassessors

•  16systemson5summariza=ontasks

•  ANOVAwithTukey’sHonestSignificantDifference–  120systempairs–  Only2/120pairsdifferedinrela=verank–  Onewasanoutlier

5/12/16 LispNYC 32

3.PyramidMethod

HighlyReliable•  Withatleast5models•  Onpyramidannota=on

•  Onassessmentannota=on•  Onsystemrankings

5/12/16 LispNYC 33

3.PyramidMethod

PyrScoreForeachstudentsummary(target):1.  Enumerateallcandidatephrasesineachsentence

(segmenta=on)2.  RepresentmeaningofeachtargetphraseandeachCU3.  Measureseman=csimilarityofeachcandidatetoeachCUto

findbestmatchforeachngram4.  Foreverytargetsentence,rankthe(segmenta=on,CUpairs)5.  Applyaweightedsetcoveralgorithm(Sakaietal.,2003)to

selectthebest“coveringset”ofCUsthatuseseachCUonlyonce

5/12/16 LispNYC 34

4.AutomatedPyramid

2.RepresentMeaning•  Distribu=onalseman=cstocapturelatentmeaning–  Basedonfrequencyofwordcontextsinalargecorpus– Mo=va=on:differentwordscanhavesimilarmeaningsMatter has volume Matter contains volume Matter takes up space Matter occupies space Matter takes up space defined as volume Matter is composed by volume Matter is something that takes up space Matter is something that occupies space . . .

2/26/15 VirginiaTech 35

4.AutomatedPyramid

2.DistributionalSemantics

Word S1 S2 S3

occupy 1 0 1

take_up 0 1 0

maner 1 1 1

space 1 1 1

energy 0 0 0

5/12/16 LispNYC 36

LatentSeman=cAnalysis(LSA)•  Similardistribu=onsforsynonymouswords:occupy, take_up•  Donotco-occurinthesamesentences(rows1-2)•  Co-occurwiththesameotherwords(rows3-5)

•  Matrixreduc=on:reducesize,retain/enhanceinforma=on

4.AutomatedPyramid

2.WeightedMatrixFactorization•  Latentmodelspooronsentencesimilarity

–  Sentencecontextsaretoosmall

– Wordsimilaritymethodsusuallydobener(eg.editdistance)•  WMF:alatentmodeldesignedforsentencesimilarity

–  LikeLSA/PLSA/LDA,observedwordssignalpresenceofcontext–  Alsousesunobservedwordstosignalabsenceofcontext

•  Implementa=on

–  Eachphrasemeaningisavocabularyvectorofweights–  Compare2vectorswithcosinesimilarity

5/12/16 LispNYC 37

4.AutomatedPyramid

3.MeasureCosineSimilarity•  CUID106,weight4=5stringstocomparewith

Label: Matter has volume and mass. Contributor 1. matter contains both volume and mass

Contributor 2. matter takes up space defined as volume . . . Contributor 3. Matter is anything that has mass and takes up space Contributor 4. matter contains volume and mass

•  Calculatesimilarityofcandidatetolabel,andtoeachcontributor•  Possiblevaluestocountasamatch

– Maximum

– Minimum– Mean

– Median

5/12/16 LispNYC 38

4.AutomatedPyramid

4.RankCUSetsperSentence

5/12/16 LispNYC 39

SEGMENTATIONSOne of: 11 unigramsenergy | is | a | property | of | matter | and | all | matter | has | it

Ten of: 1 bigram, 10 unigramsenergy is | a | property | of | matter | and | all | matter | has | it

energy | is a | property | of | matter | and | all | matter | has | it

. . .One of: one 11-gramenergy is a property of matter and all matter has it

(SEGMENTATION, CU SET) PAIRSenergy is a property of matter | and | all matter has it

{{112, 3, 1.00 }, {}, {103, 4, 0.85 }}

energy is a property of matter | and all matter has it

{{112, 3, 1.00 }, {103, 4, 0.84 }}

4.AutomatedPyramid

Rankbyavgcosinexavgngramlengthxavgweight

5.ApplyWeightedSetCover•  Graphtraversal:giventopN,choose–  Onenodepersentence(solidedges)–  OnenodeperCU(dashededges)–  NodesweightedbyaveragesumofCUweights

•  Solu=on:Nodes2and6

5/12/16 LispNYC 40

4.AutomatedPyramid

S1seg2:{112,3,1}

S1seg1:{110,4,0.5}

S1seg3:{101,2,0.75}

S2seg2:{101,2,0.60}

S2seg3:{102,4,0.81}

S2seg1:{110,4,0.76}1

32 5 6

4

PEAK•  OpenIEextracts(S,P,O)triplesfromtext–  Sentence:Alllivingthings,man-madeorfrommothernature

itself,aremadeoutofma9er

–  Triples:Noisy;mul=pleoutputspersentence•  Alllivingthings,itselfaremadeout,ofmaner•  Alllivingthings,aremadeoutof,maner

•  Foralltriples,ADWassignssimilarityscorestoall15pairsfromtwotriples:{S1,P1,O1,S2,P2,O2}

•  Constructahypergraphfromtheresults

–  Nodes:{S1,P1,O1,S2,P2,O2,...,Sn,Pn,On}–  Edges:ADWscore≥0.5

–  Hyperedges:Sametriple

5/12/16 LispNYC 41

4.AutomatedPyramid

PEAKHypergraph

5/12/16 LispNYC 42

4.AutomatedPyramid

Salientnodes(reddonedcircles):degree≥3(basedon5modelsummaries)

S1 P1 O1

S3 P3 O3

S2 P2 O2

Sent-1,T-1

Sent-2,T-2

Sent-3,T-1

SimilarityClasses

5/12/16 LispNYC 43

4.AutomatedPyramid

Findbestcontributorfromeachsummary(highestaveragesimilarity)

PEAKPyramid

s p o

Anchor matter is all the objects and substances

Contributor 1 matter is all the objects and substances

Contributor 2 matter is identified as being present everywhere

Contributor 3 matter as the stu↵ in all objects and substances

Contributor 4 matter makes up all objects or substances

5/12/16 LispNYC 44

4.AutomatedPyramid

PEAKAssessment

5/12/16 LispNYC 45

4.AutomatedPyramid

Hungarian,orMunkres-Kuhnassignmentalgorithm•  Bipar=teGraph•  Assignnodesonlek(modelCUs)tonodesonright(sentences)to

op=mize“cost”

ReadingandWritingSkills•  Verballiteracyversuscontentassessment

–  Readingassessmentthroughsummaries/essaysWhatcountsasgoodcomprehension?

–  Wri=ngassessmentthroughsummaries/essaysWhatcountsasvaluablecontent?

•  DisciplinaryknowledgeassessedthroughessaysWhatcountsasagoodscienceexplana=on?

5/12/16 LispNYC 46

5.Reading/Wri=ngRubrics

Rubrics•  Nostandardmethodtoiden=fygoodcontent–  Researchersdeveloptheirownrubrics–  Difficulttocomparequan=ta=veresultsacrossstudies

•  Labor-intensive–  Itera=onoverrubricdesign–  Reliabilitytes=ng–  Trainingraters– Manualapplica=onis=me-consuming

•  Verylinleanempttoautomateeduca=onalrubrics

–  LSA:Butcher&Kintsch,2001–  Rose&VanLehn,2005–  Gerardetal.,2016

5/12/16 LispNYC 47

5.Reading/Wri=ngRubrics

Perinetal.2013•  Study:ContextualizedCurricularSupport•  16communitycollegeclassrooms•  MiddleschooltextWhatisMa9er?

–  Buildbackgroundknowledgeforlatercollege-leveltext–  Intactpassage

•  Studentsreadpassage•  Answered7ques=onsaboutthemainideas•  Wroteasummaryinresponseto:

5/12/16 LispNYC 48

6.MainIdeasRubric

Writeasummaryofoneortwoparagraphs.Asummaryisastatementmostlyinyourownwordsthatcontainstheimportantinforma=oninthepassage.Pleasewriteclearly!

WhatisMatter?Rubric•  MainIdeasscore–  Apanelofexpertsiden=fied14mainideas

–  Pyramidcontentmodel:60Cus•  3weight5•  7weight4•  13weight3•  15weight2•  22weight1

–  All14mainideasareinthepyramidweights≥3•  3/3weight5CUs•  4/7weight4CUs•  6/13weight3CUs

5/12/16 LispNYC 49

6.MainIdeasRubric

ExperimentDATA•  A:140studentsummariesforWhatisMa9ertask

•  B:Subsetof40:manualpyramidassessment

Ques=ons:

1.  Doesthemanualwisecrowdcontentassessmentcorrelatewiththemainideasscore?(B)

2.  Dotheautomatedwisecrowdmethodscorrelatewiththemanualmethod?(B)

3.  Dotheautomatedwisecrowdmethodscorrelatewiththemainideasscore?(A)

5/12/16 LispNYC 50

7.Correla=on

Results:Question1(N=40)

WiseCrowd:Manual MainIdeas

Rawsum 0.89

Quality 0.77

Coverage 0.89

Comprehensive 0.87

5/12/16 LispNYC 51

7.Correla=on

Answer:Yes

Doesthemanualwisecrowdcontentassessmentcorrelatewiththemainideasscore?

Results:Question2(N=40)

5/12/16 LispNYC 52

7.Correla=on

Dotheautomatedwisecrowdmethodscorrelatewiththemanualmethod?

WiseCrowd:Manual PyrScore Peak

Rawsum 0.90 0.81

Quality 0.76 0.47

Coverage 0.90 0.81

Comprehensive 0.87 0.21

Answer:Yes

Results:Question3(N=140)

5/12/16 LispNYC 53

7.Correla=on

Dotheautomatedwisecrowdmethodscorrelatewiththemainideasscore?

Score PyrScore PEAK

Rawsum 0.84 0.69

Quality 0.58 0.22

Coverage 0.84 0.69

Comprehensive 0.83 0.36

Answer:•  PyrScore:Yes•  PEAK:Somewhat

Summary•  Ourapproach:aweightedcontentmodel–  Notallmeaningshaveequivalentstatus

–  Anemergentrepresenta=on•  Advantagesoverotherautomatedmethods

–  Veryreliablewithonly5models(!)fortraining

–  Automatedandhumanmethodscorrelateverywell–  Interpretablescores(quality/coverage)–  Caniden=fyspecificideasmoderatelywell

5/12/16 LispNYC 54

8.Conclusion

Publications•  Pyramidmethod

Nenkova,Ani;Passonneau,RebeccaJ.;McKeown,Kathleen.2007.Thepyramidmethod:incorporaEnghumancontentselecEonvariaEoninsummarizaEonevaluaEon.ACMTransacEonsonSpeechandLanguageProcessingVol.4,no.2,ArEcle4(May2007).

•  ReliabilityPassonneau,RebeccaJ.2010.FormalandfuncEonalassessmentofthepyramidmethodfor summarycontentevaluaEon.NaturalLanguageEngineering16:107-131.Copyright CambridgeUniversityPress.

•  PyrScorePassonneau,RebeccaJ.;Chen,Emily;Guo,Weiwei;Perin,Dolores.2013.Automated PyramidScoringofSummariesusingDistribuEonalSemanEcs.Proceedingsofthe2013 AnnualMeeEngoftheAssociaEonforComputaEonalLinguisEcs. August4-9,Sofia,Bulgaria.

•  PEAKYang,Qian;Passonneau,RebeccaJ.;deMelo,Gerard.2016.PEAK:PyramidEvaluaEonvia

AutomatedKnowledgeExtracEon.ProceedingsoftheThirEethAAAIConferenceonArEficialIntelligence,February12-17,2016,Phoenix,AZ.

•  ComparisontoMainIdeasRubricRebeccaJ.Passonneau,AnanyaPoddar,GauravGite,AlisaKrivokapic,QianYang,andDoloresPerin.

WiseCrowdContentAssessmentandEducaEonalRubrics.SubmibedtoInternaEonalJournalofArEficialIntelligenceinEducaEon.

5/12/16 LispNYC 55