CLT 2009, Heerlen, 03.03.09

The Effects of an Exploratory Web Search Interface

on Performance and Cognitive Load

Yvonne Kammerer,Peter Pirolli,

Rowan Nairn,Peter Gerjets

y.kammerer@iwm-kmrc.de

CLT 2009, Heerlen, 03.03.09

The World Wide Web – A Huge Information Source

Ill-structered,multi-faceted

problemsOpen-endedinformation

goals

CLT 2009, Heerlen, 03.03.09

LookupInvestigate Learn

Fact retrievalKnown item searchVerificationQuestion answering

…

Knowledge acquisitionComprehensionInterpretationComparison

…

EvaluationAnalysisDiscoverySynthesis

…

Explore

Exploratory Search

Marchionini (2006)

CLT 2009, Heerlen, 03.03.09

Exploratory Search Process – Continuous Process

Submission of a tentative query

Decision making aboutwhich paths to follow

Exploration of returned hits

Selection of information

Query refinements

Learningtaking placeduring the process

White, Marchionini, & Muresan (2006)

multiple iterations

CLT 2009, Heerlen, 03.03.09

Exploratory Search Process

Difficulties for searchers:

• Search goal is vague

Uncertainty about what to search for and how to search

• Huge amount of information presented

Uncertainty about which information is relevant

CLT 2009, Heerlen, 03.03.09

Exploratory Search Interfaces

Aims:• Support searchers in their exploratory search

• Help searchers explore complex information spaces

• Help searchers learn about new topics: go beyond fact-finding

• Improve searchers’ queries

• Support searchers’ in navigation and decision-making

CLT 2009, Heerlen, 03.03.09

Features of Exploratory Search Interface

other keywords relatedto this web page

arrowsto give

relevance feedback

relatedkeywordslist

Providing related keywords

User-providedrelevance feedback

CLT 2009, Heerlen, 03.03.09

Baseline Version

CLT 2009, Heerlen, 03.03.09

Hypotheses

1. Performance:Exploratory search interface > Baseline interfaceExploratory search interface supports searchers’ performance (search outcome and informal learning)

• Cognitive load: Exploratory search interface > Baseline interfaceInteraction with exploratory search interface increases searchers’ cognitive load.

germane cognitive load?

CLT 2009, Heerlen, 03.03.09

Experimental Design

Factor 1: Search Interface

Factor 2: Task Domain

Future Architecture CLPerformance

CLPerformance

Global Warming

Web Mashups

Exploratory interface (n=15)(related keywords,

relevance feedback)

Baseline interface (n=15)(no exploratory search

features)

CLPerformance

CLPerformance

CLPerformance

CLPerformance

+interactivefeature

use

Participants:22 male, 8 femalePARC members and Stanford studentsaverage age: 31.9 years (range: 21-54)

CLT 2009, Heerlen, 03.03.09

Performance Measures: Tasks

1. Page Collection tasks: Collecting as many relevant pages as possible to specific pre-defined queries

2. Summarization tasks: Writing general summaries about the domain of search

3. Keywords Formulation tasks: Formulating keywords for search

CLT 2009, Heerlen, 03.03.09

Cognitive Load Measure: NASA Task Load Index

5 sub scales from 0=very low to 100=very high :• mental demand • time pressure • perceived success• mental effort • stress level

Aggregation of all 5 sub scales (with perceived success inverted):• Overall Cognitive Load measure

0=very low cognitive load to 100=very high cognitive load

Hart & Staveland, 1988

Cronbach‘s alpha= .83

CLT 2009, Heerlen, 03.03.09

Domain 1:

Page Collection Task (6 min)Summarization Task (12 min)

Keywords Task (2 min)

NASA-TLX

Instructions and Training Tasks

Pre-Questionnaire(demographics, prior knowledge)

Domain 2: …

Domain 3: …

Duration: approx. 1.5 hUsability ratings

Differences in prior knowledge about Web mashups

covariate

Experimental Procedure

CLT 2009, Heerlen, 03.03.09

Results: Performance Data – Summarization Tasks

Summarization task performance

Interface:F (1, 26) = 8.91, p = .01, η2 = .26

Mashups Knowledge:ns.

Interface x Mashups Knowledge:F (1, 26) = 3.68, p = .07, η2 = .12

0

1

2

3

4

5

futurearchitecture globalwarming mashups

num

ber o

f sum

mar

y as

pect

s

baseline exploratory

p=.11

*

**

CLT 2009, Heerlen, 03.03.09

Relationship: Mashups Summaries and Mashups Knowledge

r=.46(*)

r=-.18

CLT 2009, Heerlen, 03.03.09

Results: Performance Data – Keywords Tasks

Keywords task performance

Interface:ns.

WM Knowledge:ns.

Interface x Mashups Knowledge:F (1, 26) = 5.06, p = .03, η2 = .16

0

2

4

6

8

10

12

futurearchitecture globalwarming mashups

num

ber o

f key

wor

ds

baseline exploratory

CLT 2009, Heerlen, 03.03.09

Relationship: Mashups Keywords and Mashups Knowledge

r=.05

r=.74**

CLT 2009, Heerlen, 03.03.09

Results: Cognitive Load

Cognitive load

Interface:F (1, 26) =6.23, p = .02, η2 = .19

WM Knowledge:ns.

Interface x Mashups Knowledge:ns.

0

20

40

60

80

100

futurearchitecture globalwarming mashups

cogn

itive

load

baseline exploratory

* *p=.13

CLT 2009, Heerlen, 03.03.09

Results: Interactive Feature Use

Average interactive feature use per task:

• M = 3.81 times (SD = 2.24)• Page Collection tasks: M = 5.54 times (SD = 3.03)• Summarization tasks: M = 2.07 times (SD = 1.71)

Positive correlation between feature use and cognitive load• r = .59 (p = .02)

The more the exploratory features were used,the higher was the cognitive load.

CLT 2009, Heerlen, 03.03.09

Relationship between Cognitive load, Performance, and Interactive Feature Use for Summarization Tasks

InteractiveFeature use

Cognitiveload

Summaryperformance

r=.38(p=.16)

r=-.32(p=.24)

For Exploratory interface users (n=15)

r=-.37(p=.17)

rpart=-.28(p=.33)

CLT 2009, Heerlen, 03.03.09

Summary and Conclusion

Performance:Exploratory search interface > Baseline interfaceExploratory search system supports searchers’ performance

• Particular support for domain novices

Cognitive load: Exploratory search interface > Baseline interface

• Interaction with exploratory search interface increases searchers’cognitive load

However, interpretation as germane cognitive load not warranted

CLT 2009, Heerlen, 03.03.09

Thanks for your attention!

http://mrtaggy.com

CLT 2009, Heerlen, 03.03.09

Backup

CLT 2009, Heerlen, 03.03.09

Exploratory Search Systems

Exploratory Search Features:Tools to support navigation and decision-making

• System provides options, e.g. keywords, to choose from when searching

• Relevance feedback for query refinements(from the user to the system)

CLT 2009, Heerlen, 03.03.09

Benefits and Costs of Exploratory Search Systems

Presentation of additional information and support of rich interactionimproves the relevance of the documents found

High complexity of interface increases cognitive load and search time

Bruza, McArthur, & Dennis (2000)

CLT 2009, Heerlen, 03.03.09

Summarization Tasks

Summarization tasks: writing more general summaries about the domain of search

Domain Summarization task

Future Architecture

Describe three different topics what "Future Architecture" could be about

Global Warming

Arguments or evidence in favor and against human-caused Global Warming

Web Mashups Benefits of the use of Web Mashups

CLT 2009, Heerlen, 03.03.09

Subscales 2-5 of NASA TLX

CLT 2009, Heerlen, 03.03.09

Results: Performance Data – Page Collection tasks

0

1

2

3

4

5

6

7

8

9

10

futurearchitecture globalwarming mashups

num

ber o

f pag

es c

olle

cted

baseline exploratory

Pages Collection task performance

Interface:ns.

Mashups Knowledge:ns.

Interface x Mashups Knowledge:ns.

CLT 2009, Heerlen, 03.03.09

Results: Time on Task for Summary Tasks

0

2

4

6

8

10

12

14

futurearchitecture globalwarming mashups

time

take

n fo

r sum

mar

ies

(in s

ec)

baseline exploratory

**

**p=.18

Time on task

Interface:F (1, 26) = 13.04, p = .001, η2 = .33

Mashups Knowledge:ns.

Interface x Mashups Knowledge:ns.

CLT 2009, Heerlen, 03.03.09

Results: Performance Data – Summarization tasks

Summarization task performance

With time on task as covariate

Interface:F (1, 26) = 6.13, p = .02, η2 = .20

Summary time:ns.

Interface x Summary time:ns.

Mashups Knowledge:ns.

Interface x Mashups Knowledge:F (1, 26) = 3.68, p = .07, η2 = .12

0

1

2

3

4

5

futurearchitecture globalwarming mashups

num

ber o

f sum

mar

y as

pect

s

baseline exploratory

p=.12

*

(*)

CLT 2009, Heerlen, 03.03.09

Results: Cognitive load (MANCOVA with 5 subscales)

Interface:(5, 22) = 2.24, p = .09, η2 = .34

Univariate main effects of Interface:• Mental demand: p = .21

• Time pressure: p = .01

• Perceived success: p = .08

• Effort: p = .11

• Stress level: p = .06

CLT 2009, Heerlen, 03.03.09

Relationship between Cognitive load, Performance, and Interactive Feature Use for Page Collection Tasks

InteractiveFeature use

Cognitiveload

Page Coll.performance

r=.61(p=.02)

r=.13(p=.65)

For Exploratory interface users (n=15)

r=-.07(p=.81)

rpart=-.19(p=.52)