MUTUALLY LINKED STUDIESBalancing Threats to Internal and

Ecological Validity in InfoVis Evaluation

NIKLAS ELMQVISTSchool of Electrical & Computer Engineering

Purdue University

2

The Problem of Validity

• InfoVis evaluation often comes in two flavors:– Rigorous + unrealistic• Scientific rigor• Toy dataset• => Lacking in ecological validity

– Realistic + ad hoc• Real dataset• Imperfect experimental design• => Lacking in internal validity

3

Example: The ColorLens[McDonnel & Elmqvist 2009]

4

Example: Evaluating the ColorLens

Realistic photographs:• Ecologically valid (+)• Learning effects (-)• Cannot control images (-)• Does not “feel” canonical! (-)

Perlin Noise:• Full control over dataset (+)• Eliminate learning (+)• Canonical and abstract (+)• Not realistic! (-)

?

5

Mutually Linked Studies

• Why choose one type of study? Do both! • Idea: Mutually linked studies– 1 x toy study – canonical and exhaustive– 1 x realistic study – prove it works– Complement each other– Recycle much of the mechanics for both

• Experimental design• Tasks• Analysis• Participants (!)

Reduces time and paper space requirements for conducting two studies (they are not separate)!

6

Questions?

• Acknowledgments: Color Lens co-authors(coming soon to a journal near you)Pierre DragicevicHM Jean-Daniel Fekete

7

Validity Primer

• Several different types of validity…• I will discuss the following here:– Internal validity• Degree to which the outcome is a function of the

controlled parameters of experiment

– Ecological validity• Degree to which the results of the experiment can be

applied to realistic situations