learning about information searchers from eye-‐tracking

8/3/2019 Learning about Information Searchers from Eye- ‐Tracking

http://slidepdf.com/reader/full/learning-about-information-searchers-from-eye-tracking 1/40

Jacek GwizdkaDepartment of Library and Information Science

School of Communication and Information

Rutgers University

Monday, April 4, 2011

LearningaboutInforma0onSearchers

fromEye-Tracking

CONTACT:

www.jsg.tel



OverallResearchGoals

• Characteriza*onandenhancementofhumaninforma<on

interac<onmediatedbycompu<ngtechnology

• Characteriza*on:cogni<veandaffec<veuserstates–

tradi<onally:li>leaccesstothemental/emo<onalstatesofuserswhiletheyareengagedinsearchprocess

• Implicitdatacollec<onaboutsearchers’cogni<veand

affec<vestatesinrela<ontoinforma<onsearchphases

• Enhancement :Personaliza<onandAdapta<on

3



Example:ImplicitCharacteriza0onof

Cogni0veLoadonWebSearch

4

Qformulate

query

L view

search

results list

Bbookmark

page

ST A R T

END

C view

contentpage

97% 58% 30%

42%

95%

7%

27%35%

higher peak

cognitive load: Chigher average

cognitive load: Q & B

(Gwizdka, JASIST, 2010)



Eye-Tracking?

• Earlya>emptslateXIXc.;early1950’s-amoviecamera

andhand-coding(Fi>s,Jones&Milton1950)

• Nowcomputerizedand“easytouse”

– infraredlightsourcesandcameras

– sta<onaryandmobile

5



Eye-tracking–fundamentalassump0ons

• Top-downvs.bo>om-upcontrol

– inbetween:languageprocessing(higher-level)controlswheneyes

move,visualprocessing(lower-level)controlswhereeyesmove

(Reichleetal.,1998)

• Eye-mindlinkhypothesis:a>en<oniswhereeyesare

focused(Just&Carpenter,1980;1987)

• Overtandcoverta>en<on

• A>en<oncanmovewithnoeyemovementBUTeyes

cannotmovewithouta>en<on

6



DatafromEye-trackingDevices

• eyegazepoints

– eyegazepointsinscreencoordinates+distance

– eyefixa<onsinscreencoordinates+validity

– pupildiameter• [headposi<on3D,distancefrommonitor]

• 50/60Hz;300Hz;1000-2000Hzeye-trackers

• common:60Hz:onedatarecordevery16.67ms

7



Eye-TrackingCan…

• Eyetrackingcanallowiden<fica<onofthespecificcontent

acquiredbythepersonfromWebpages

• Eyetrackingenableshighresolu<onanalysisofsearcher’s

ac<vityduringinterac<onswithinforma<onsystems• Andmore…

8

Example: composing answer and

from information on a Web page

(video)



RelatedWorkinInforma0onScience

• Interac<onwithsearchresults

– Interac<onwithSERPs(Grankaetal.,2004;Lorigoetal.,2007;2008)

– Effectsresultspresenta<on(Cutrelletal.,2007;Kammereral.,2010)

– Relevancedetec<on(Buscher,etal.2009)

– ImplicitFeedback(Fu,X.,2009);Queryexpansion(Buscher,etal.2009)

• Relevancedetec<on

– Pupillometry(Oliveira,Aula,Russell,2009)

• Detec<onoftaskdifferencesfromeye-gazepa>erns – Reading/reasoning/search/objectmanipula<on(Iqbal&Bailey,2004)

– Informa<onalvs.transac<onaltasks(Terai,etal.,2008)

– Taskdetec*onisalsooneofourresearchinterests9



Experiment1:Journalismtasks–OpenWebSearch

• 32journalismstudents

• 4journalis<ctasks(realis<c,createdbyjournalismfacultyandjournalists)

• Tasks:

– advancedobituary(OBI) – interviewprepara<on(INT) – copyedi<ng(CPE) – backgroundinforma<on(BIC)

10

Note: OBI vs. CPE are most dissimilar

Task facets:

• product: factual vs. intellectual• level: whole document vs. segment• nature of task goal• complexity – number of steps needed



Experiment1–ResearchQues0ons

• Canwedetecttasktype(differencesintaskfacets)from

implicitinterac<ondata(e.g.,eye-tracking)?

• Howdoweaggregateinforma<onfromeye-trackingdata?

11



Eye-gazepaMerns

• Eye-trackingresearchhave

frequentlyanalyzedeye-gaze

posi<onaggregates('hotspots’)

– spa<otemporal-intensity–heatmaps – alsosequen<al–scanpaths

12

• Higher-orderpa>erns:

– readingmodels



ReadingEyePaMerns

• Readingandscanninghaveeasilydis<nguishedpa>ernsof

fixa<onsandsaccades.(Rayner&Fischer,1996)

• LexicalProcessingofWords

– Readingresearchhasestablishedwordavailabilityisafunc<onoffixa<ondura<on:

– Orthographicrecogni<on:40-50ms

• <metomovedatafromeyestomind

– Phonologicalrecogni<on:55-70ms

– Lexicalavailability(typical):113ms–150ms(Rayner,1998)• Unfamiliarorcomplexmeaningsrequirelongerprocessing

– Eyesdonotsaccadeun<lthewordhasbeenprocessed

13



ScanFixa0onsvs.ReadingFixa0ons

• Scanningfixa<onsprovidesomeseman<cinforma<on,

limitedtofoveal(1°visualacuity)visualfield(Rayner&Fischer,1996)

• Fixa<onsinareadingsequenceprovidemoreinforma<on

thanisolated“scanning”fixa<ons: – informa<onisgainedfromthelargerparafoveal(5°beyondfoveal

focus)region(Rayneretal.,2003)(asymmetrical,indirofreading)

– richerseman<cstructureavailablefromtextcomposi<ons

(sentences,paragraphs,etc.)

• Someofthetypesofseman<cinforma<onavailableonly

throughreadingsequencesmaybecrucialtosa<sfytask

requirements.

14



Readingodels

• WeimplementedtheE-ZReaderreadingmodel(Reichleetal.,

2006)

– Inputs:(eyefixa<onloca<on,dura<on)

– Fixa<ondura<on>113ms –thresholdforlexicalprocessing(Reingold&Rayner,2006)

– Thealgorithmdis<nguishesreadingfixa<onsequencesfromisolated

fixa<ons,called'scanning'fixa<ons

– Eachlexicalfixa<onisclassifiedto(S,R)(Scan,Reading)

– Thesesequencesusedtocreateastatemodel

15



Readingodel–StatesandCharacteris0cs

• Twostates:transi<onprobabili<es

• Numberoflexicalfixa<onsanddura<on

16



ExampleReadingSequence

17



Results:SearchTaskEffectonReading/Scanning

Taskeffectsontransi<onprobabili<esSR&RS(allsubjects&pages)

18

(Cole, Gwizdka, Liu, Bierig, Belkin & Zhang, 2010)

• For OBI, INT searchers

biased to continue reading • For CPE to continuescanning

Searchers are adoptingdifferent reading strategies for

different task types



Results:SearchTaskFacetsandTextAcquisi0on

Forhighlya>endedpages

19

Total Text Acquisition on

SERPs and Content

per page

Total Text Acquired on

SERPs and Content



Results:SearchTaskFacetsandStateTransi0ons


20

State Transitions on

SERPs per page

State Transitions on

Content pages per page

Read ScanRead Scan Scan Read

Scan Read



TaskFacetsEffects-Summary


21

(Cole, Gwizdka, Liu, Bierig, Belkin & Zhang, submitted, 2011)



Scan<->ReadTransi0onProbabili0esin2Experiments

• Person’stendencytoreadscanrelatedto

scanread?(i.e.,isprelatedtoq?)

• p~1-q

correlation (Spearman ρ): 0.914 and 0.830

Journalistic tasks (N=32) Genomics tasks (N=40)



Experiment1:Conclusions

• Searchers’reading/scanningbehavioraffectedbytask

• Tasksfacetscanbe“detected”fromeye-trackingdata(from

readingmodelproper<es)

• Readingmodelscanbebuiltonthefly(duringsearch)real-<meobserva<onsofeyemovementscanbeusedbyadap<ve

searchsystems

• Challenge:Lackofbaselinedataaboutreadingmodelsofindividuals

23



Experiment2:ResultListvs.OverviewTag-Cloud

37par<cipants

Everydayinforma<onseekingtasks(travel,shopping…)

-twolevelsoftaskcomplexity

Twouserinterfaces

24

1. List UI2. Overview UI

(Tag Cloud)



Experiment2:UserAc0onsinTwoInterfaces

25

1. List

2. Overview

Tag Cloud



Experiment2:ResearchQues0ons

• Doesthesearchresultsoverviewbenefitusers?

• Taskeffects?

• Individualdifferences-cogni<veabilityeffects?

26



GeneralResults

• Searchresultsoverview(“tagcloud”)benefitedusers

– madethemfaster

– facilitatedformula<onofmoreeffec<vequeries

• Morecomplextaskswereindeedmoredemanding–required

moresearcheffort

27

(Gwizdka, Information Research, 2009)



TaskandUIandReadingodeldifferences

• Complextasksrequiredmorereadingeffort

– Longermaxreadingfixa<onlengthandmorereadingfixa<onregressions

• OverviewUIrequiredlesseffort

– Scanningmorelikely(S-Shigher;S-Rlower;R-Shigher)

– Readingscanpathlengthtotallowerbuttotalscanpathswerelonger

(includingscanning)

– Lessandshortermeanfixa<onsperpagevisited

28

List Overview



TaskandUIInterac0onandReadingmodeldata

• ForcomplextasksUIeffect

– Higherprobabilityofshortreading

sequencesinOverviewUI

• ForsimpletasksUIeffect

– Shorterlengthofreadingscanpathsper

pageandlessfixa<onsperpage

• Task&UIinterac<on

– Speedofreading:

• forcomplextasksfasterreadinginOverviewthaninListUI

• forsimpletasksfasterinListthaninOverviewUI

29



UserInterfaceFeatures–IndividualDifferences

• Twousers,sameUIandtask

30



IndividualDifferences–LeastEffort?

• Highercogni<veabilitysearcherswerefasterinOverviewUI

andonsimpletasks(samenumberofqueries)

• Higherabilitysearchersdidmoreinmoredemandingsitua<ons

– highersearcheffortdidnotseemtoimprovetaskoutcomes

31

F(144,1)=4.2; p=.042 F(144,1)=3.1; p=.08

For tasks and working memory (WM)



TaskandWorkingemory–Eye-trackingData

• HighWMlesslikelytostayinScanningstate

• HighWMhigherreadingspeed(scanpath/totalfixa<ondura<on)

• Numberanddura<onofreadingsequencesdiffers – (borderline:0.05p0.1)

• ForhighWMsearchers:

– forsimpletaskslessreading

– forcomplexmorereading

• ForlowWMnosuchdifference!

32



Experiment2:Conclusions

• OverviewUIwasfaster–reflectedinsomeeye-tracking

measures

• Taskcomplexitydifferencesreflectedinsomeeye-tracking

measures

• Someeffectsofcogni<veabili<esoninterac<on

– viola<onoftheleasteffortprinciplenotfullyexplainedyet

33



34



CurrentProject:

CanWeImplicitlyDetectRelevanceDecisions?

• Startwithpupillometry

– inforelevance(Oliveria,Russell,Aula,2009)

– low-leveldecision<ming(Einhäuser,etal.2010)

• AlsolookatEEG,GSR

FundedbyGoogleResearchAward

EEG

GSR

Eye trackingpupilanimation

• Implicitcharacteriza<onofInforma<onSearchProcess

usingphysiologicaldevices

• Canwedetectwhensearchersmakeinforma<onrelevancedecisions?



Summary&Conclusions

• Eyetrackingenableshighresolu<onanalysisofsearcher’s

ac<vityduringinterac<onswithinforma<onsystems

• Eyetrackingcansupportiden<fica<onofsearchtasktypes• Eyetrackingreflectsdifferencesinsearcherperformanceon

userinterfaces

• Eyetrackingreflectsindividualdifferencesbetween

searchers

• Highpoten<alforimplicitdetec<onofsearcherstates

36



SomeChallenges

• High-resolu<ondata(low-level)

• Howdowecreatehigher-levelpa>erns?

• Howdowedetectthemcomputa<onally?

• Howdowedealwithind.diffs(baselinedata)?

37(Lorigo et al., 2008)(Terai et al., 2008)

(Iqbal & Bailey, 2004)



High-resolu0onEye-trackingisComingSoontoYou

• Eyetrackingtechnologyisdeclininginpriceandin2-3years

couldbepartofstandarddisplays.

– Alreadyinluxurycarsandsemi-trucks(sleepdetec<on)

– Computerswithbuiltineye-tracking

38

Tobii / Lenovo

proof of concept laptop

March 2011



Thankyou! Ques<ons?

JacekGwizdkacontact:h>p://jsg.tel

PoODLEProject:Personaliza<onoftheDigitalLibraryExperience

IMLSgrantLG-06-07-0105-07

h>p://comminfo.rutgers.edu/research/poodle

orforshort:h>p://bit.ly/poodle_project

PoODLEPIs:NicholasJ.Belkin,JacekGwizdka,XiangminZhang

Post-Doc:RalfBierig,

PhDStudents:JingjingLiu,(nowAsstProf.),ChangLiu,MichaelCole



OtherEye-trackingDerivedeasures

• Speedoftextprocessing

– readingspeed;scanpathlength/totalfixa<ondura<on

• Perceptualspan – numberofcharactersprocessedata<me

– Englishetc.~19characters(Rayner,1998)

– decreasesinnormalreadingwhenuserencounterscogni<vely-

demandingcontent

40

learning about information searchers from eye-‐tracking

Documents