Chapter 12 Observing Users

Li, JiaLi, Wei

Outline

What and when to observeApproaches to observationHow to observeHow to collect dataIndirect observationAnalyzing, interpreting and presenting data

Goals and Questions

Provide a focus for observation Example: watch a group of people

First watch: what are they doing? … Second watch: more focused questions

Necessary for all evaluation studies Balance between being guided by

goals and being open-minded

What and When to Observe

During early design of a product Help understand users’ needs

During later refinement of a product Find out whether it meets users’

needs

Type of Observation

Participant observation

(e.g., in ethnography)(Not applicable)Insider

“Quick and dirty”

In field studies

“Quick and dirty”

In usability testing

Outsider looking on

Field environment

(i.e., natural)

Controlled environment

(i.e., lab-like)

Observation

Approaches to Observation

“ Quick and dirty” observationfind out what is happening quickly and with little formality

Observation in usability testingcontrolled environment:

Video, interaction logs Watch through one-way mirror or via remote TV

screen

Observation in field studies

Observation in Field Studies

Levels of participation (By Colin Robson, 1993) Complete participants More marginal participants Observers who also participate Outside observer

Observation in Field Studies - Outside Observer

When an observer is interested only in the presence of certain types of behavior

Example: Observe when a computer is

used by boys and when by girls, and how long

Observation in Field Studies - Inside Observer

Participant observers Participate in the lives of the people under

study Maintain a professional distance that allows

adequate observation and recording of data

EthnographersControversy Participant observation is virtually synonymous

with ethnography Participant observation is a technique used in

ethnography

How to Observe - In Controlled Environments

Issues before observation Decide laboratory to do the tests where

appropriate equipment is set up Test equipment to make sure it works Provide an informed consent form for users to

read and sign at the beginning of the study

ProblemsObservers do not know what users are thinking Think-aloud technique

How to Observe - In the Field

Issues before observation State initial study goal and questions clearly Select a framework to guide activity in the field Decide how to record events Go through notes and other records as soon as

possible Highlight and separate personal opinion from

what really happens

How to Observe - In the Field (Cont’d)

Issues before observation Refine goals and questions while

observation Gain the acceptance and trust of those to

be observed Think about ways to handle sensitive issues Consider working as a team Check notes with an informant or members

of the group Plan to look at situation from different

perspectives

Frameworks

Why use frameworks Provide focus Organize/structure observation

What framework to select A simple framework:

The person The place The thing

Frameworks (Cont’d)

What framework to select Goetz and LeCompte framework

Who is present? What is happening? When does the activity occur? Where is it happening? How is the activity organized?

Frameworks (Cont’d)

What framework to select Colin Robson framework

Space: appearance and lay-outActors: names and relevant detailsActivities: what are they doing and whyObjects: present physical objectsActs: what are specific individuals doingGoals: what to accomplish for actorsFeelings: mood of the group and

individuals

How to Observe – Participant Observation and Ethnography

Issues before observation Identify goal and questions with the

help of observation framework Get immersed in the group to observe Collect a variety of data Get balance between broad picture

and specific questions Analyze data using a holistic approach

DilemmasWhen should I stop observing When see similar patterns of behavior being

repeated When finish listening to all the main

stakeholder groups and understand their perspectives

How can I adapt ethnography to fit the development process Preparation Field study Analysis Reporting

Data collection

Notes plus still cameraAudio recording plus still cameraVideo

Notes plus still camera

Advantages Less technical. Flexible. Equipment easy to

get and not expensive.

Unobtrusive. Easy to transcribe.

Disadvantages Difficult and tiring

to write and observe at the same time.

Writing speed limited.

Less reliable, rely on the note-takers.

Audio plus still camera

Advantages Inexpensive Flexible Relatively

unobtrusive Low disturbance

Disadvantages Lack of visual

record Difficult to

transcribing the data.

Have to change tapes and position microphone .

Video

Advantages Capture both

visual and audio High reliability Permanent

original record that can be revisited.

Disadvantages Equipment more

expensive Intrusive Time-consuming

to analyze the video data

Need positioning and focusing camera lens.

Indirect observation: tracking users’s activities

DiariesInteraction logging

Diaries (1)

What users didWhen they did itWhat they thought about their interactions with the technology

Useful when users are scattered and unreachable

such as: Internet and web evaluations

Diaries (2)

Advantages Inexpensive No special

equipment or expertise

Suitable for long-term studies

Disadvantages Rely on participants

being reliable and remembering to complete the diaries.

Participants often remember events as being better or worse than they really were, or taking more or less time than they actually did.

Interaction logging

Advantages Unobtrusive Large volumes of

data can be logged automatically

Disadvantages Ethical concerns. Powerful tools are

needed.Webtrends

Dilemma --- observing without being seen

They do not know we are watching. Shall we tell them? Tell: users may react or change

their behavior. Depends on the context:

How much personal information is collected?

How the information will be used?

Analyzing, interpreting, and presenting the data

Qualitative data that is interpreted and used to tell “the story” about what was observed.

Qualitative data that is categorized using techniques such as content analysis.

Quantitative data that is collected from interaction and video logs and presented as values, tables, charts and graphs and is treated statistically.

Qualitative analysis to tell a story

Review the data Record the themes Record the date and time Check your understanding Iterate this processReport your findings

Qualitative analysis for categorization

Looking for incidents or patternsAnalyzing data into categoriesAnalyzing discourse

Looking for incidents or patterns

Look for critical incidents, such as times when users were obviously stuck.Theory may also be used to guide the study.A variety of tools are available to record, manipulate and search the data. NIDIST Observer Video-Pro tool

Analyzing data into categories

Content analysis provides another fine grain way of analyzing video data.It is a systematic, reliable way of coding content into a meaningful set of mutually exclusive categories.

Analyzing discourse

Another approach to video and audio analysis is to focus on the dialog.Analyzing discourse on the Internet has started to influence designers’ understanding about users’ needs in these environments.

Quantitative data analysis

Typically this data is further analyzed using simple statistics such as means, standard deviations, etc.

Feeding the findings back into design

Clearly written reports with Overview at the beginning Detailed content list

Verbal presentation