cse 595, winter 2000 1 january 20 topics (design and evaluation part 2) zadministrivia ynext...
TRANSCRIPT
CSE 595, Winter 2000 1
January 20 Topics (Design and Evaluation Part 2)
Administrivia next week’s readings on web (focus: technology);
name tag usability; reading report feedback
Usability Study Guidelines Contextual Design discussion Guest speaker: Domenick Dellino, “What Usability
Can Borrow from Anthropological Methos” More paper discussion:
Ethnography Heuristic Evaluation; Discount Usability Engineering McGrath - Methodology
Project Pitches or Group meetings
CSE 595, Winter 2000 2
Quote of the Week
“You can observe a lot just by watching.” -- Yogi Berra
CSE 595, Winter 2000 3
Another Quote of the Week
“Le mieux est l’ennemi du bien”-- Voltaire
CSE 595, Winter 2000 4
Guidelines for User Testing with Thinking Aloud
Practical study designWritten materialsCarrying out the studyImproving the study
(this material is also linked from the class web page)
CSE 595, Winter 2000 5
Practical Study Design
Reflect on the participants' backgrounds and how they might affect the study
Be aware of problems that arise when experimenters know the users personally
Prepare for the study carefully (avoid last minute panic) Select the tasks carefully to be representative and to fit the
allotted time In general, start with an easier (but not frivolous) task Write down features of the system that are not being
tested as well as those that are!
CSE 595, Winter 2000 6
Practical Study Design (2)
Define the start-up state for the study precisely Define precise rules for when and how users can be helped
during the study Plan the timing and cut-off procedure (if subject gets stuck)
for each part of the study Include reasonable provisions for data collection (e.g.,
notes, tape or video recorder, keystroke capture where appropriate)
Plan data analysis techniques in advance Carry out a pilot study (important but often overlooked)
CSE 595, Winter 2000 7
Written materials
Participant release form (if needed) Questionnaire covering prior experience etc. (if relevant) Introduction to the study for users, including scenario of
use Checklist for experimenters Evaluation survey (if relevant)
CSE 595, Winter 2000 8
Carrying out the study
Let users know that complete anonymity will be preserved Let them know that they may quit at any time Stress that the system is being tested, not the participant Indicate that you are only interested in their thoughts
relevant to the system Demonstrate the thinking-aloud method by acting it out for a
simple task, such as figuring out how to load a stapler, and a computer-related task
Hand out instructions for each part of the study individually, not all at once
Maintain a relaxed environment free of interruptions Encourage users to keep talking using unobtrusive comments
that don't point the user in a particular direction. Debrief each user after the experiment
CSE 595, Winter 2000 9
Improving the study
The pilot study should "debug" the study. This minimizes changes during the study, allowing quantitative data analysis. But improvements may be warranted.
Experimenters' role can be improved Tasks given to participant can be improved Written materials can be improved
CSE 595, Winter 2000 10
Notes on “Contextual Design”
Widely praised in industryThe class rated it highly (average 5.1)An eclectic customer-centered design
methodology — includes ideas from ethnography, participatory design, much empirical work, etc.
Data gathered from customers is the basis for making design decisions
Team and organizational considerations explicitly dealt with
CSE 595, Winter 2000 11
Principles of Contextual Inquiry
The initial data gathering stagePrinciples:
Context: summary vs ongoing experience; abstract vs concrete data
Partnership: master/apprentice model (avoid interviewer/interviewee, expert/novice, guest/host)
Interpretation : check interpretations on the spot (“but won’t it bias the data?”)
Focus: interviewer’s point of view while studying the work
CSE 595, Winter 2000 12
Stages in Contextual Design Methodology
contextual inquiryinterpretation sessionswork modelsaffinity diagramwork model consolidationvisionstoryboardsuser environment designpaper prototyping
CSE 595, Winter 2000 13
Notes on Ethnography
Dictionary definition: “a branch of anthropology dealing with the scientific description of individual cultures”
Many ethnographers reside in the field a year or more, learning the local language, and participating in everyday life, while maintaining some objective detachment (participant observation).
CSE 595, Winter 2000 14
Ethnography — Issues
identification with culture being studiedinformantscultural change resulting from
ethnographer’s presencetechnology: field notes, video, audio
recordings
CSE 595, Winter 2000 15
Relevance for Design
Designers create artifacts for work settings need to understand those settings.
Technology shapes practice designer's world view should not be imposed inappropriately on the users.
Allows us to gain broader understanding of technology in use (see traditional methods)
Joint exploration of technology and work allows both users and designers to participate in new designs.
How Usability Borrows from Anthropological Methods
Usability Research: Anthropology of the Workplace
Domenick J. Dellino
CSE 595, Winter 2000 17
Overview
Traditional anthropological methods Verbal methods Non-verbal methods
Anthropological methods in usability Examples and comparisons
Confounds of each method
CSE 595, Winter 2000 18
Traditional Anthropological Methods — Verbal
Participant observationKey-informant interviewingCollection of life historiesStructured interviews and surveysQuestionnairesRankings and Ratings
CSE 595, Winter 2000 19
Traditional Anthropological Methods—Non-verbal
ProxemicsKinesicsVideotape Research*Content Analysis*
Myths Folktales
Erosion and Accretion
CSE 595, Winter 2000 20
Participant Observation
Trad. Anthropology Anthropologist lives
with the tribe as a member of the community until barely noticed by the natives
Observes and records the culture while participating as a native
Usability Research Contextual Inquiry:
Usability Specialist infiltrates school computer lab as a computer nerd needing remedial training
Observes student’s questions/behavior
CSE 595, Winter 2000 21
Key Informant Interviewing
Trad. Anthropology Anthropologist finds one
native who is willing to tell everything
Informant enjoys special treatment (e.g.: Hershey bars)
Usability Research Participants volunteer
to be recruited for usability studies
Participants are rewarded (e.g.:software)
CSE 595, Winter 2000 22
Collection of Life Histories
Trad. Anthropology Anthropologist develops
rapport with a few individuals
Collects extensive materials about these individuals
Usability Research Heuristic Evaluation: Usability specialist
finds other specialists who understand interface design
Asks them to report all “usability issues” they can find
CSE 595, Winter 2000 23
Structured Interviews & Surveys
Trad. Anthropology Fieldworker
administers formal interview about lifestyle, earnings, expenditures, etc. by going from house to house.
Usability Research Specialist asks
specific questions in the workplace (or lab) about users’ work processes, policies, and practices
Contextual Interview
CSE 595, Winter 2000 24
Questionnaires
Trad. Anthropology Researcher hands out
questionnaires to an informant and returns to collect the answers
E.g.: Informants are asked to record what they purchase during the week
Usability Research Specialist distributes
questionnaires to a group in a lab setting or e-mails survey
Recipients are asked attitudinal and behavioral questions.
CSE 595, Winter 2000 25
Ranking and Ratings
Trad. Anthropology Commonly used to
determine the hierarchical structure of caste, class, or kinship systems
Usability Research Cluster Analysis Participants are
asked to sort cards of menu commands into stacks
CSE 595, Winter 2000 26
Proxemics
Trad. Anthropology Unobtrusive
observations of how close people stand to each other
Usability Research Eye tracker software Tells us where the
user is looking on the screen
CSE 595, Winter 2000 27
Kinesics
Trad. Anthropology Lurking Watching what people
do Observing the people
one touches, grooms, hits, holds
Usability Research Counts of clicks and
keystrokes Counts of “mistakes” Recording of success
rate
CSE 595, Winter 2000 28
Erosion & Accretion
Trad. Anthropology Archaeology Study of paths Studies of middens,
fireplaces, burial grounds, & tool making sites
Usability Research Use of “instrumented
versions” Investigation of query
& error logs, Web cache, trash
The “Lame” button F1 key wear
CSE 595, Winter 2000 29
Confounds of Each Method-1
Participant Observation Heisenberg Uncertainty Principle
Key Informant Interviewing/Life Histories Sampling—Informants may not be
“representative”
Structured Interviews & Surveys “Performance anxiety” Memory may be flawed, suspicion
CSE 595, Winter 2000 30
Confounds of Each Method-2
Questionnaires Literacy, focus, desire to show favor
Ranking and Ratings Task requires sophisticated thought Data analysis is aggregate (non-parametric) Validity of interpretation (are we measuring
what we think we’re measuring?)
CSE 595, Winter 2000 31
Confounds of Each Method-3
Proxemics Ethnocentric/design-centric interpretation
Kinesics Question of learning style: “I like to try
everything first.”
Erosion & Accretion “Whose data is it, anyway?”
CSE 595, Winter 2000 32
…for Reflection
What value does ethnography add?When should it be conducted?Is it cost effective?
CSE 595, Winter 2000 33
Ethnography — Ethics
anthropologists must be open about the purpose, potential impacts, and source of support
primary responsibility to people and animals with whom researchers work
responsibility to scholarship and scienceresponsibility to the public
American Anthropological Association link:http://www.ameranthassn.org/committees/ethics/ethics.htm
CSE 595, Winter 2000 34
Notes on J. Blomberg et al, “Ethnographic Field Methods”
Rating: 4.3 Why Observe? Why not just ask?
ideal vs. manifest behavior "What people say and do are not the same thing." People have a great deal of tacit knowledge that they cannot
verbalize.
Observational role: Unobtrusive Observer (aka Observer Participant).
This is difficult or inappropriate in many settings. Participant Observer
+access, +first-hand experience, +-point-of-view issues, -logistical problems of recording
CSE 595, Winter 2000 35
Blomberg (2)
In reality, the ethnographer doesn't assume either of the above extremes, but moves back and forth along the continuum between pure observer and pure participant.
CSE 595, Winter 2000 36
Blomberg (3)
What do we focus on? Event focus - meetings, seminars, ceremonies. Person focus - "a day/week/year in the life
of..." Place focus - receptionist's desk, printer room,
etc. Object focus - life history of a document,
transaction, etc.
How do you know you are finished? When you're no longer suprised by what you're
seeing!
CSE 595, Winter 2000 37
Blomberg (4) — Techniques
Notes: + inexpensive, interpretive, flexible (sketches,
etc), holistic - interpretive, can be hard to reconstruct, "low
bandwidth," lots of work for observer
Audiotape: + inexpensive, "medium bandwidth," less work
for observer - limited slice of activity (audible/verbal only),
requires some audible actions to be useful, difficult to reconstruct
CSE 595, Winter 2000 38
Blomberg — More Techniques
Videotape: + "high bandwidth," can make detailed reconstructions
(content log), advantages of audio, less work for observer, unobstrusive, somewhat holistic
- limited slice of activity, expensive, building content logs can be exhausting, coordinating multiple videos difficult, activity space/lighting might not be suitable for filming
Event Logs: (computer capture of input events) + "high bandwidth," can make detailed reconstructions, can
get good statistics about failures/difficulties, time sensitive - computer only, emphasizes human-machine dyad view –
human is viewed as mere source of input events anti-holistic
CSE 595, Winter 2000 39
Linking ethnography and design
Ethnographer reports findings to designers
Ethnographic study is undertaken by team of ethnographers and designers
Full participation in designing by ethnographers, designers, and users
Issues: Whose "side" is the ethnographer on? Who "owns" the result (knowledge) of
ethnographic research?
CSE 595, Winter 2000 40
Notes on Jacob Nielsen, “Guerilla HCI”
average rating: 5.1
Techniques: scenarios simplified thinking aloud heuristic evaluation
CSE 595, Winter 2000 41
Notes on Jacob Nielsen, “Heuristic Evaluation”
average rating: 4.3
Heuristic evaluation: have a small set of evaluators examine an interface and judge its compliance with recognized usability principles (the “heuristics”).
Evaluators work independently. (Can compare results afterwards.)
Output: a list of usability problems with reference to the heuristics.
Comment: doesn’t require working with eventual real users.
CSE 595, Winter 2000 42
# of Evaluators vs Problems
CSE 595, Winter 2000 43
Cost-Benefit Ratio
CSE 595, Winter 2000 44
Ten Usability Heuristics (from Nielsen)
Visibility of system status Match between system and the real world User control and freedom Consistency and standards Error prevention Recognition rather than recall Flexibility and efficiency of use Aesthetic and minimalist design Help users recognize, diagnose, and recover from
errors Help and documentation
CSE 595, Winter 2000 45
Usability Heuristics Applied to the Web
Highly recommended:
Keith Instone’s discussion of applying these to the web
(http://webreview.com/97/10/10/usability/sidebar.html)
CSE 595, Winter 2000 46
Notes on Joseph McGrath, “Methodology Matters”
Ratings: binary (five 1’s, three 6’s). Average 3.5
I’ve received very positive comments on this paper from social scientists
science vs. design and engineering; discount usability engineering
CSE 595, Winter 2000 47
McGrath: 3 domains
content that is of interest (e.g. grad students using Powerpoint to prepare a job talk). This is the substantive domain.
ideas that give meaning to the content (e.g. "able to prepare slides, frustrated" — i.e. task-related descriptions, affect-related descriptions, etc). This is the conceptual domain.
techniques or procedures for studying the ideas and content (ethnography, laboratory experiment, field study, heuristic evaluation). This is the methodological domain.
CSE 595, Winter 2000 48
McGrath: techniques
Techniques for manipulating features of the research situation: giving instruction Imposing constraints Selecting materials Giving feedback Using experimental confederates
CSE 595, Winter 2000 49
McGrath: desirable features of a research strategy
generalizibilityprecisionrealism
We can't maximize all of these at the same time!
We can have multiple studies, and ask "does study A support study B?"
CSE 595, Winter 2000 50
McGrath: Some key concepts
BaseratesCorrelationsCausality