evaluation in experience-focused hci: exploring the virtual intimate object future applications lab...

Evaluation in Experience-Focused HCI: Exploring the

Virtual Intimate Object

Future Applications LabViktoria Institute

Gothenburg, Sweden13 January 2006

Joseph ‘Jofish’ KayeCulturally Embedded Computing

Cornell Information Sciencejofish@cornell.edu

Evaluation in Experience-Focused HCI

Evaluation in experience-focused HCI is the poor cousin of design. As a field (if we are one!) we have excellent theoretical foundations (Dourish, Wright & McCarthy), a wide repertoire of design techniques (Equator Project, IDEO) and extensive tools for building (CS, EE, PD, ID). But traditional HCI evaluation techniques (GOMS, etc) are inadequate for ready-at-hand technologies used in ways unreplicable in the lab.

Virtual Intimate Object (VIO)• A small red circle in

Windows’ taskbar, or in a Mac window.

• When circle is clicked, partner’s circle glows bright red, then fades over time.

http://io.infosci.cornell.edu• Kaye, Levitt, Nevins, Golden & Schmidt. Communicating Intimacy One Bit at a Time. Ext. Abs. CHI 2005, ACM Press.

• Kaye. I just clicked to say I love you. alt.chi, Ext. Abs. CHI 2006, ACM Press.

Intimacy in HCI

History of elegantly designed but ad-hoc interfaces for communicating intimacy in HCI; hard to evaluate

• Fields & Thresholds (Dunne & Raby DoP2 '94)• Feather, Scent & Shaker (Strong & Gaver CSCW'96)

• The Bed (Dodge CHI'97)• UbiComp'03 Workshop on Intimacy

www.intimateornot.org

• Digital Family Portrait (Mynatt et. al. CHI'01)• inTouch (Brave, Ishii, Dahley CSCW'98) • inStink, Honey I’m Home (Kaye '02, interactions '04)• ……

How much does Ariel love Eric? Count the hearts!

Disney (1997) The Special Edition Little

Mermaid Coloring Book. Golden Books Publishing.

Answer: 19

Two VIO Studies

• Study I– 10 couples in existing long-distance relationships (n=20)

• 5 couples assigned to VIO (n=10)• 5 couples assigned to MinIO (n=10)

– Paper logbooks; pre, post-tests, daily questions• Study II

– 80+ subjects recruited through long distance relationship communities on LiveJournal

– All subjects took pre-survey; all were asked to take post-survey

– Four groups: VIO, VIO + daily survey, daily survey alone, nothing

– All surveys online - Currently being analyzed

Experience surveys: starting points

•Gaver et. al: Cultural Probes•Geertz: Thick description•Dourish, Wright & McCarthy,

etc: experience-based HCI•Iterative design: Design,

Build, Evaluate..

Thick Description

• Winking isn’t just something your eye does• It’s culturally embedded• It’s only the thick description of the

context and culture that lets us understand the role of the wink in sharing a conspiracy, or even parodying another sharing conspiracy.

• Geertz, C. (1973) Interpretation of Cultures Ch. 1

Cultural Probes

• Gives context around a situation• Originally for inspiring the design part of the

design build evaluate iterative design cycle

• Repurposed here for inspiring the evaluation part of the cycle

• <provocative statement>We’re changing a key, generally unspoken assumption by proposing that evaluation is creative, arbitrary, and complex.</provocative>

Gaver et. al. Cultural Probes, interactions 6(1) 1999

Three topics

• Questions/tasks about the technology being evaluated.

• Questions/tasks about the situation.• Questions/tasks about instrument

being used to gather information

And three kinds of questions(for organization, not

canonical)• Context: Where? When? Take a

photo of… How long? How often?• Metaphor: What TV show? What

song? What colour? What metaphor?• Valence: What’s the best? What’s

the worse? What’s your favorite? What should we change? What shouldn’t we change? Draw the best, the worst.

Question Design chart  Context Metaphor Valence

Technology(VIO)

Situation(Longdistancerelationship)      

Instrument(Probe / Survey)

Getting beyond usability

• Evaluating in context requires working in context

• VIO becomes ready-at-hand, not present-at-hand, and so evaluation must be in context.

• Users must have significant, continuous use, and you must have solved the usability problems otherwise most of what you get is usability issues – i.e. Mobile Probes paper

Hulkko et. al. Mobile Probes Proc. NordCHI ‘04

Discussion

• Unpacking meanings of ‘evaluate’– …tell me if people like it– …tell me if a variable that I can track has

changed– …make a comparison between my stated

(unstated?) goals/values/principles and the built technology

– …to tell me what to do next

• When? (Formative, iterative, final, discrete, continuous)

Thanks to Maria Håkansson, Lars Erik Holmquist, Phoebe Sengers, the Culturally Embedded Computing Group, my subjects

and my coauthors.

Joseph ‘Jofish’ KayeCulturally Embedded Computing

Cornell Information Sciencejofish@cornell.edu

This talk at jofish.com/talks. VIO at io.infosci.cornell.edu