CS124/LINGUIST180FromLanguagestoInformation

DanJurafsky

StanfordUniversity

IntroductionandCourseOverview

FromLanguagestoInformation

Automaticallyextractingmeaningandstructurefrom:◦Humanlanguagetextandspeech(news,socialmedia,etc.)◦Socialnetworks◦Genomesequences

Interactingwithhumansvialanguage◦Dialogsystems/Chatbots◦QuestionAnswering◦RecommendationSystems

CommercialWorld

1.Extractinginformationfromlanguage

InformationRetrieval

6,586,013,574 websearcheseveryday(byoneestimate)Text-basedinformationretrievalisthuslikelythemostfrequentlyusedpieceofsoftwareintheworldHowdoesitwork?CanyoubuildanIRengine?ProgrammingAssignment4:Search!

ExtractingSentimentandSocialMeaning

Lotsofmeaningisinconnotation"connotation: an idea or feeling that a word invokes in addition to its literal or primary meaning."

Extractingconnotationisgenerallycalledsentimentanalysis

SentimentAnalysis

ExtractingSocialMeaningfromSpeech

Uncertainty(studentsintutoring)Annoyance◦callerstodialogsystems:DeceptionEmotionIntoxicationFlirtation,Romanticinterest◦ McFarland,Jurafsky,Ranganath

Whatdoyoudoforfun?Dance?Uh,dance,uh,Iliketogo,likecamping.Uh,snowboarding,butI'mnotgood,butI

liketogoanyway.Youlikeboarding.Yeah.Iliketodoanything.LikeI,I'mupforanything.Really?Yeah.Areyouopen-mindedaboutmosteverything?Noteverything,butalotofstuff-Whatisnoteverything[laugh]Idon'tknow.Thinkofsomething,andI'llsayifIdoitornot.[laugh]Okay.[unintelligible].Skydiving.Iwouldn'tdoskydivingIdon'tthink.YeahI'mafraidofheights.F:Yeah,yeah,metoo.M:[laugh]Areyouafraidofheights?F:[laugh]Yeah[laugh]

Whatdoflirtersdo?Womenwhenflirting:◦raisepitchceiling◦laughatthemselves◦say“I”Menwhenflirting:◦raisetheirpitchfloor◦laughattheirdate(teasing?)◦say“you”and“youknow”◦don’tusewordsrelatedtoacademics

Rajesh Ranganath, Dan Jurafsky, and Daniel A. McFarland. 2013. Detecting friendly, flirtatious, awkward, and assertive speech in speed-dates. Computer Speech and Language. 27:1, 89-115

Unlikelywordsformaleflirting

academiainterviewteacherphdadvisorlabresearchmanagementfinish

SentimentinRestaurantReviewsDanJurafsky,VictorChahuneau,BryanR.Routledge,andNoahA.Smith.2014.Narrativeframingofconsumersentimentinonlinerestaurantreviews.FirstMonday19:4

Thebartender...absolutelyhorrible...wewaited10minbeforeweevengotherattention...andthenwehadtowait45- FORTYFIVE!- minutesforourentrees…stalkthewaitresstogetthecheque…shedidn'tmakeeyecontactorevenbreakherstridetowaitforaresponse…

900,000Yelpreviewsonline

A very bad (one-star) review:

Whatisthelanguageofbadreviews?Negativesentimentlanguagehorribleawfulterriblebaddisgusting

Pastnarrativesaboutpeoplewaited,didn’t,washe,she,his,her,manager,customer,waitress,waiter

Frequentmentionsofweand us...we wereignoreduntilwe flaggeddownawaitertogetour waitress…

OthernarrativeswiththislanguageAgenreusing:Pasttense,we/us,negative,peoplenarratives

Textswrittenbypeoplesufferingtrauma◦ JamesPennebaker lab◦ Pasttenseasdistancing◦ Useof“we”:seekingsolaceincommunity

1-starreviewsaretraumanarratives!Thelessonofreviews:It’sallaboutpersonalinteraction

Whataboutpositivereviews?Sex,Drugs,andDessert

• orgasmicpastry• sexyfood• seductivelysearedfois gras

� addictedtopeppershooters� garlicnoodles…mydrugofchoice� thefriesarelikecrack

ComputationalBiology:ComparingSequences

SLIDE STUFF FROM SERAFIMBATZOGLOU

AGGCTATCACCTGACCTCCAGGCCGATGCCCTAGCTATCACGACCGCGGTCGATTTGCCCGAC

-AGGCTATCACCTGACCTCCAGGCCGA--TGCCC---| | | | | | | | | | | | | x | | | | | | | | | | |

TAG-CTATCAC--GACCGC--GGTCGATTTGCCCGACSequencecomparisoniskeyto• Findinggenes• Determiningfunction• UncoveringevolutionaryprocessesThisisalsohowspellcheckerswork!

We'lllearn:editdistancealgorithms(Quiz1)

SocialNetworks

Thenetworkformedbyyourfriendsorotherrelationsofflineoronline◦Canwecomputepropertiesofthesenetworks?◦Extractinformationfromthem?

Highschooldating

PeterS.Bearman,JamesMoodyandKatherineStovel Chainsofaffection:ThestructureofadolescentromanticandsexualnetworksAmericanJournalofSociology 11044-91(2004)ImagedrawnbyMarkNewman

Whatisthestructureofsocialrelations?Imagineagraphofhighschool

• peoplearenodes• linksareromanticrelationships

Whatwilltheshapeofthisgraphbe?Adenselyconnectedgraph?Aline?Acycle?

2.Interactingwithhumansvialanguage

QuestionAnswering:IBM’sWatson

RecommendationEngines

PersonalAssistants

Whyislanguageinterpretationhard?

Ambiguity

Resolvingambiguityishard

AmbiguityFindatleast6meaningsofthissentence:

I made her duck

AmbiguityFindatleast6meaningsofthissentence:

I made her duckIcookedwaterfowlforherbenefit(toeat)IcookedwaterfowlbelongingtoherIcreatedthe(plaster?)waterfowlsheownsIcausedhertoquicklylowerherheadorbodyIrecognizedthetrueidentityofherspywaterfowlIwavedmymagicwandandturnedherintoundifferentiatedwaterfowl

AmbiguityisPervasive

IcausedhertoquicklylowerherheadorbodyPartofspeech:“duck”canbeaNounorVerb

Icookedwaterfowlbelongingtoher.Partofspeech:“her”ispossive pronoun(“ofher”)“her”isdativepronoun(“forher”)

Imadethe(plaster)duckstatuesheownsWordMeaning:“make”canmean“create”or“cook”

AmbiguityisPervasive

Grammar:make canbe:Transitive:(verbhasanoundirectobject)

Icooked[waterfowlbelongingtoher]Ditransitive:(verbhas2nounobjects)

Imade[her](into)[undifferentiatedwaterfowl]Action-transitive(verbhasadirectobject+verb)Icaused[her][tomoveherbody]

AmbiguityisPervasive:Phonetics!!!!!ImateorduckI’meightorduckEyemaid;herduckAyemate,herduckImaidherduckI’maidherduckImateherduckI’mateherduckI’mateorduckImateorduck

Moredifficulties:Non-standardlanguage

Great job @justinbieber! Were SOO PROUD of what youve accomplished! U taught us 2 #neversaynever& you yourself should never give up either♥

Andneologisms:unfriendretweetbromance

Makingprogressonthisproblem…

Thetaskisdifficult!Whattoolsdoweneed?◦Knowledgeaboutlanguageandtheworld◦Awaytocombineknowledgesources

Howwegenerallydothis:◦probabilisticmodelsbuiltfromlanguagedataP(“maison”® “house”)highP(“L’avocat général”® “thegeneralavocado”)low

ModelsandToolsRegularExpressionsEditdistanceandalignmentWordembeddings◦vector/neuralmodelsofmeaning

Languagemodels(wordprediction)MachineLearningclassifiers◦NaïveBayes◦LogisticRegression 33

Networkalgorithms◦PageRank

Recommendationalgorithms◦Collaborativefiltering

Linguistictools◦Sentimentlexicons

CourselogisticsinbriefInstructor:DanJurafskyTAs:WillHamilton(headTA)

Time:TuTh 3:00-4:20,420-040cs124.stanford.edu

JeffPykeKellyShenStephanieTangLucyWangRobVoigt

RobinJiaRafaelMusaKateParkCharissa Plattner

JanetteChengTimDozatAshkon FarhangiGasparGarcia

EvidenceBasedPedagogy!

35

http://www.knewton.com/flipped-classroom/

Whytheflippedclassroom(1)Masterylearning:LearnuntilyoumasterBenjaminBloom,1968

Bloom'smasterylearningPersonalized,goal-drivenpractice,drivenbyfeedback1. Watch(andre-watch)lecturesatyourownpaceand

learnwhenit'sbestforyou2. Videoshaveembeddedminiquizzes.Ifyougetitwrong,

itgivesyoufeedbackaboutwhyyoumisunderstood.3. Youhave2chancesateachweeklyTuesdayQuizzes,so

youcangobacktothelectureandretakethem.4. Withprogrammingassignmentsyoucanseeyour

performanceonthetrainingsettoseewhatyou'redoingwrong!

Whythevideoshaveembeddedquizzes:“summative”vs“formative”assessment

Summativeassessment◦Finalexams:goalisgrading

Formativeassessment◦Alongtheway:goalisforyou tofindoutwhatyoudon’tknowsoyoucanlearn

Whytheflippedclassroom(2)

Attentionspan:everyonespacesoutduringlonglectures◦Middendorf andKalish,1995,Johnstone andPercival1976,Burns1985

“theclassstarted1:00.Thestudentsittinginfrontofmetookcopiousnotesuntil1:20.Thenhejustnoddedoff…motionless,witheyesshutforaboutaminuteandahalf,penstillpoised.Thenheawokeandcontinuedhisrapidnote-takingasifhehadn’tmissedabeat.”Studentrememberedonlythefirst15-20minutes

Whytheflippedclassroom(3)Activelearning:Beinchargeofyourlearning◦Obviouslymostimportant:programmingassignments◦Activelearning(“constructivism”),learningbydoing

Collaborativelearning:Learnfromeachother◦Useclasstimeforgroupactivities,workedproblems◦“Smallgroupactivelearning”

cs124:Semi-flippedclassroomLecturesonvideo:Iexpectyouto:◦Watchvideolectures(and/orandreadtextbookchapters)◦Onaverageabout90minutesofvideocontenteachweek◦ Somepeoplewatchitspeededup

Somelectureslive:◦ 8lecturesand1groupsessionarerequired(onfinalexam,novideos)◦ Iwillalsore-lecture(double-cover)afewofthevideos◦ somepeopleliketheengagementofin-classlectures

In-classgroupsessions(“activelearning”)◦ Optionalbutrecommended

LogisticsMoreSpecificallyOnlineVideoLectureswithembeddedquizzes(beforeclass)WeeklyonlineReviewQuizzes(Tueoffollowingweek)RoughlyweeklyPythonhomeworks (Frioffollowingweek)FinalExam(TuesdayMarch203:30-6:30)Classsessions:Allencouraged;8 livelecturesrequired◦Fulllectures◦Mini-lectures◦Groupworkedproblems

TheOpenPlatform:EdX!

https://lagunita.stanford.edu/abouthttp://edx.readthedocs.io/projects/edx-guide-for-students/en/latest/index.htmlhttps://open.edx.org/about-open-edx

LearningGoals

Attheendofthiscourse,youwillbeableto:

45

Learninggoals

Writeefficientregularexpressionstosolveanykindoftext-basedextractiontask

46

Learninggoals

Applytheeditdistancealgorithmtoallsortsoftextsequenceproblems

47

Learninggoals

Buildasupervisedclassifiertosolveproblemslikesentimentclassification

48

Learninggoals

Buildasearchengine

49

Learninggoals

Buildarecommendationengine

50

Learninggoals

Buildacomputationalmodelofwordmeaning(usinglexiconsandembeddings)

51

Learninggoals

Buildachatbot

52

Learninggoals

UnderstandandimplementPageRank

53

Thisclassistheundergradintroto:

Win2018:cs224NNaturalLanguageProcessingw/DeepLearningWin2018:cs246MiningMassiveDataSetsSpr 2018:cs222UNaturalLanguageUnderstandingAut 2018:cs224WAnalysisofNetworks

Spr 2019:cs276InformationRetrievalandWebSearch

TBD:cs224SSpokenLanguageProcessing

54

Syllabus

http://web.stanford.edu/class/cs124

Comingupnextclass(Thursday)

Unixforpoetsgrepsort

PA1:SpamLord!

Writeregularexpressionstospreadevilthroughoutthegalaxy!Byextractingemailaddressesandphonenumbersfromtheweb!jur a fs ky at st anford dot e d u

GoesliveFriday!

57

ActionItemsBeforeThursday!1)Readthesyllabuswebpageatcs124.stanford.edu

2)SignupforpiazzaandedX◦ ForedX,you'llneedtofirstsigninwithyourSUnet IDat suclass.stanford.edu,andthenclickontheEdX buttonatthetopofthecs124.stanford.eduwebpage

3)Watchthefirsthalfofthisweek’svideos(“BasicTextProcessing”)beforeclass!

4)Downloadthisfiletoyourlaptop

http://cs124.stanford.edu/nyt_200811.txt.gz58