1

User-Centered Design and Development

Instructor: Franz J. KurfessComputer Science Dept.

Cal Poly San Luis Obispo

FJK 2005

5

Chapter 12 Observing Users

FJK 2005

6

Chapter Overview

• Goals, Questions, and Paradigms• Observation• Data Collection• Tracking Activities• Analysis, Interpretation, and Presentation of

Data

FJK 2005

7FJK 2005

Motivation

• the actual observation of users as they perform tasks with the system may reveal information that is difficult to obtain otherwise

• technologies like video cameras and audio recorders make it fairly easy to capture many aspects of the observation

• the activities of users can also be tracked indirectly

• the analysis, interpretation and presentation is essential after the actual observation

8FJK 2005

Objectives

• be familiar with the main observation methods• be aware of the different perspectives of an

on-looker, a participant, and an ethnographer• appreciate the importance of analyzing,

interpreting, and presenting data obtained from observations

• gain experience in performing and evaluating observations

12

What and when to observe

• Goals & questions determine the paradigms and techniques used.

• Observation is valuable any time during design.• Quick & dirty observations early in design• Observation can be done in the field (i.e., field

studies) and in controlled environments (i.e., usability studies)

13

Roles of Observers

• outsiders looking on– sometimes through a window or one-way mirror– should the participant be aware of it?

• participating observers– the observer works with other participants– emphasis of the observer should be on observation,

not on activities to perform the task

• ethnographers– intentional immersion into the “world” of the

participants

FJK 2005

14

Observation Frameworks

• The person. Who? • The place. Where?• The thing. What?

Framework Examples– Goetz and LeCompte (1984)– Robinson (1993)

15

The Goetz and LeCompte (1984) Framework

• Who is present? • What is their role? • What is happening? • When does the activity occur?• Where is it happening? • Why is it happening? • How is the activity organized?

15

16

The Robinson (1993) Framework

• Space – What is the physical space like?

• Actors– Who is involved?

• Activities – What are they doing?

• Objects– What objects are present?

• Acts– What are individuals doing?

• Events – What kind of event is it?

• Goals– What do they to accomplish?

• Feelings– What is the mood of the group and of individuals?

16

17

Aspects to Consider

• goals and questions• selection of framework, techniques• data collection• equipment to use• acceptance and trust by participants• handling of sensitive issues• involvement of informants• data analysis• need for triangulation

– different perspectives

18

Observing as an Outsider

• same technique as in usability testing• more objective than participant observation• often done in usability labs

– layout of the facilities– equipment is in place

• recording is continuous• analysis & observation almost simultaneous• danger of drowning in data• analysis

– can be coarse or fine grained

• presentation– Video clips can be powerful for telling astory

19

Participant Observation and Ethnography

• debate about differences• participant observation is key component of

ethnography• co-operation of people observed• informants can be useful

– but also sensitive

• data analysis is continuous• interpretivist technique• questions get refined as understanding grows• reports usually contain examples

20

Activity: Observation in RPGs

• is it ethical to join a community for the main purpose of observation– with or without the knowledge of other participants– human observer or computer agent

• do you know of any incidents in games you’re participating in

FJK 2005

21

Data Collection Techniques

• notes – often selective, and include some interpretation

• visual data– still camera– video

• audio data– tape recorder– usually included in video– computer

• tracking users:– diaries– interaction logging

22

Observation Activity: Review

• data collection techniques used– appropriate– worth the effort

• constraints imposed by the selected techniques– missed data– aspects not captured– observer bias– differences between observers

• analysis and interpretation– bias, differences, jumping to conclusions

FJK 2005

23

Data Analysis

• qualitative data – interpreted and used to tell the ‘story’ about what

was observed – categorized using techniques such as content

analysis

• quantitative data – collected from interaction and video logs– presented as values, tables, charts, graphs and

treated statistically.

24

Interpretive Data Analysis

• key events – drive the group’s activity

• patterns of behavior • triangulation

– testing of data sources against each other

• presentation– report findings in a convincing and honest way

• ‘rich’ or ‘thick descriptions’ – include quotes, pictures, and anecdotes

• software tools can be useful – NUDIST, nVivo (see http://www.qsr.com.au/)– Ethnograph (see http://www.qualisresearch.com/)

augmented FJK 2005

25

Looking for Patterns

• critical incident analysis– coarse analysis of the overall data set to identify

important events– only those events are analyzed in more detail

• content analysis– systematic categorization of events– requires orthogonal categories (no overlap)

• discourse analysis– interpretation of dialog– also considers meaning, intention, context

• quantitative analysis (statistics)– means, standard deviation, etc.

augmented FJK 2005

26

Key Points

• observation from outside or as a participant• analyzing video and data logs can be time-

consuming• participant observation includes collections of

comments, incidents, and artifacts • ethnography is a philosophy with a set of

techniques that include participant observation and interviews

• ethnographers immerse themselves in the culture that they study