evolving evaluation: from engineers to experience
DESCRIPTION
Evolving Evaluation: from Engineers to Experience. GIST, Glasgow 16 November 2006. Joseph ‘Jofish’ Kaye Microsoft Research, Cambridge Cornell University, Ithaca, NY jofish @ cornell.edu. What is evaluation?. Something you do at the end of a project to show it works… - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/1.jpg)
Evolving Evaluation:from Engineers to Experience
Joseph ‘Jofish’ KayeMicrosoft Research, CambridgeCornell University, Ithaca, NYjofish @ cornell.edu
GIST, Glasgow16 November 2006
![Page 2: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/2.jpg)
What is evaluation?• Something you do at the end
of a project to show it works…• … so you can publish it.• Part of the design-build-
evaluate iterative design cycle• A way of defining a field• A way a discipline validates
the knowledge it creates.• A reason papers get rejected
![Page 3: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/3.jpg)
HCI Evaluation: Validity“Methods for establishing
validity vary depending on the nature of the contribution. They may involve empirical work in the laboratory or the field, the description of rationales for design decisions and approaches, applications of analytical techniques, or ‘proof of concept’ system implementations”
CHI 2007 Website
![Page 4: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/4.jpg)
So…• How did we get to where we
are today?• Why did we end up with the
system(s) we use today?• How can our current
approaches to evaluation deal with novel concepts of HCI, such as experience-focused (rather than task focused) HCI?
• And in particular…
![Page 5: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/5.jpg)
Evaluation of the VIO• A device for couples in long
distance relationships to communicate intimacy
• It’s about the experience; it’s not about the task
www.intimateobjects.orgKaye, Levitt, Nevins, Golden & Schmidt.
Communicating Intimacy One Bit at a Time. Ext. Abs. CHI 2005.
Kaye. I just clicked to say I love you. alt.chi, Ext. Abs. CHI 2006.
![Page 6: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/6.jpg)
A Brief History and plan for the talk
1. Evaluation by Engineers2. Evaluation by Computer
Scientists3. Evaluation by Experimental
Psychologists & Cognitive Scientists
4. Evaluation by HCI Professionals
5. Evaluation in CSCW6. Evaluation for Experience
![Page 7: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/7.jpg)
A Brief History and plan for the talk1. Evaluation by Engineers2. Evaluation by Computer
Scientists3. Evaluation by Experimental
Psychologists & Cognitive Scientists
a. Case study: Evaluation of Text Editors
4. Evaluation by HCI Professionals
a) Case Study: The Damaged Merchandise Debate
5. Evaluation in CSCW6. Evaluation for Experience
![Page 8: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/8.jpg)
3 Questions to ask about an era
• Who are the users?• Who are the evaluators?• What are the limiting
factors?
![Page 9: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/9.jpg)
Evaluation by Engineers• Users are engineers &
mathematicians• Evaluators are engineers• The limiting factor is
reliability
![Page 10: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/10.jpg)
Evaluation by Computer Scientists
• Users are programmers• Evaluators are
programmers• The speed of the machine
is the limiting factor
![Page 11: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/11.jpg)
Evaluation by Experimental Psychologists& Cognitive Scientists• Users are users: the
computer is a tool, not an end result
• Evaluators are cognitive scientists and experimental psychologists: they’re used to measuring things through experiment
• The limiting factor is what the human can do
![Page 12: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/12.jpg)
Perceptual issues such as print legibility and motor issues arose in designing displays, keyboards and other input devices… [new interface developments] created opportunities for cognitive psychologists to contribute in such areas as motor learning, concept formation, semantic memory and action. In a sense, this marks the emergence of the distinct discipline of human-computer interaction. (Grudin 2006)
Evaluation by Experimental Psychologists& Cognitive Scientists
![Page 13: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/13.jpg)
Case Study of Evaluation: Text EditorsRoberts & Moran, 1982,
1983.Their methodology for
evaluating text editors had three criteria:objectivitythoroughnessease-of-use
![Page 14: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/14.jpg)
Case Study: Text Editorsobjectivity “implies that the methodology not
be biased in favor of any particular editor’s conceptual structure”
thoroughness “implies that multiple aspects of editor use be considered”
ease-of-use (of the method, not the editor itself)“the methodology should be usable by editor designers, managers of word processing centers, or other nonpsychologists who need this kind of evaluative information but who have limited time and equipment resources”
![Page 15: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/15.jpg)
Case Study: Text Editorsobjectivity “implies that the methodology not
be biased in favor of any particular editor’s conceptual structure”
thoroughness “implies that multiple aspects of editor use be considered”.
ease-of-use (of the method (not the editor itself),“the methodology should be usable by editor designers, managers of word processing centers, or other nonpsychologists who need this kind of evaluative information but who have limited time and equipment resources.”
![Page 16: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/16.jpg)
Case Study: Text Editors
Text editors are the white rats of HCI
Thomas Green, 1984,in Grudin, 1990.
![Page 17: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/17.jpg)
Evaluation by HCI Professionals• Usability professionals• They believe in expertise
(e.g. Neilsen 1984)• They’ve made a decision to
decide to focus on better results, regardless of whether they were experimentally provable or not.
![Page 18: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/18.jpg)
Case Study: The Damaged Merchandise Debate
![Page 19: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/19.jpg)
Damaged Merchandise Setup
Early eighties:usability evaluation methods (UEMs)- heuristics (Neilsen)- cognitive walkthrough- GOMS- …
![Page 20: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/20.jpg)
Damaged Merchandise Comparison Studies
Jefferies, Miller, Wharton and Uyeda (1991)
Karat, Campbell and Fiegel (1992)
Neilsen (1992)Desuirve, Kondziela, and
Atwood (1992)Neilsen and Phillips (1993)
![Page 21: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/21.jpg)
Damaged Merchandise Panel
Wayne D. Gray, Panel at CHI’95
Discount or Disservice? Discount Usability Analysis at a Bargain Price or Simply Damaged Merchandise
![Page 22: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/22.jpg)
Damaged Merchandise Paper
Wayne D. Gray & Marilyn Salzman
Special issue of HCI:Experimental Comparisons of
Usability Evaluation Methods
![Page 23: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/23.jpg)
Damaged Merchandise ResponseCommentary on Damaged
MerchandiseKarat: experiment in contextJefferies & Miller: real-worldLund & McClelland: practicalJohn: case studiesMonk: broad questionsOviatt: field-wide scienceMacKay: triangulateNewman: simulation & modelling
![Page 24: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/24.jpg)
Damaged Merchandise What’s going on?
Gray & Salzman, p19There is a tradition in the human factors literature of providing advice to practitioners on issues related to, but not investigated in, an experiment. This tradition includes the clear and explicit separation of experiment-based claims from experience-based advice. Our complaint is not against experimenters who attempt to offer good advice… the advice may be understood as research findings rather than the researcher’s opinion.
![Page 25: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/25.jpg)
Damaged Merchandise What’s going on?
Gray & Salzman, p19There is a tradition in the human factors literature of providing advice to practitioners on issues related to, but not investigated in, an experiment. This tradition includes the clear and explicit separation of experiment-based claims from experience-based advice. Our complaint is not against experimenters who attempt to offer good advice… the advice may be understood as research findings rather than the researcher’s opinion.
![Page 26: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/26.jpg)
Damaged Merchandise Clash of Paradigms
Experimental Psychologists & Cognitive Scientists
(who believe in experimentation) vs.
HCI Professionals (who believe in experience and expertise, even if ‘unprovable’) (and who were trying to present
their work in the terms of the dominant paradigm of the
field.)
![Page 27: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/27.jpg)
CSCWBriefly…• CSCW vs. HCI• Not just groups instead of
users, but philosophy & approach (ideology?)
• Posits that work is member-created, dynamic, and explictly not cognitive, modelable
• Follows failure of ‘workplace studies’ to characterize work
![Page 28: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/28.jpg)
Evaluation in CSCW• Ramage, The Learning Way
(Ph.D, Lancaster 1999)– No single ‘right’ or wrong– Identify why evaluate here– Determine stakeholders– Observe & analyze– Learn
• Note the differences between this kind of approach and more traditional HCI user testing.
• Fundamentally different from HCI: so much so they became a new field.
![Page 29: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/29.jpg)
Experience Focused HCI
• A possibly emerging sub-field, drawing from traditions and disciplines outside the field
• Emphasis on the experience, not [just] the task
• But how to evaluate?
![Page 30: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/30.jpg)
Experience focused HCI
Isbister et. al.: open-ended affective evaluations that leverage realtime individual interpretations.
Isbister, Höök, Sharp, Laaksolahti. The Sensual Evaluation Instrument: Developing an Affective Evaluation Tool. Proc. CHI’06
![Page 31: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/31.jpg)
Experience focused HCI
Gaver et. al.: cultural commentators with expertise in their own fields provide multi-layered assessment.
Gaver, W. Cultural Commentators for Polyphonic Assessment. To appear in IJHCI.
![Page 32: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/32.jpg)
Experience focused HCI
Kaye et. al. Cultural probes to provide user-interpreted thick descriptions of use experience
Kaye, Levitt, Nevins, Golden & Schmidt. Communicating Intimacy One Bit at a Time. Ext. Abs. CHI 2005.
![Page 33: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/33.jpg)
Epistemology
• How does a field know what it knows?
• How does a field know that it knows it?
• Science: experiment…• But literature? Anthropology?
Sociology? Therapy? Art? Theatre? Design?
• These disciplines have ways to talk about experience lacking in an experimental paradigm.
![Page 34: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/34.jpg)
Formally…
The aim of this work is to recognize the ways in which multiple epistemologies, not just the experimental paradigm of science, must inform the hybrid discipline of human-computer interaction if we wish to build systems that support users’ increasingly rich interactions with technology.
![Page 35: Evolving Evaluation: from Engineers to Experience](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813c5f550346895da5e42e/html5/thumbnails/35.jpg)
An evolving discussionThanks to• Louise Barkhaus & Barry
Brown• Alex Taylor & MS Research• Phoebe Sengers & CEmCom• Cornell S&TS Department• Maria Håkansson & the IT
University Göteborg• Andy Warr & The Oxford E-
Research Center