HumanComputerInteractionLaboratory

CMSC434Introduction to Human-Computer Interaction

Week 02 | Lecture 03 | Feb 4, 2016

Formative Inquiry & User Research Methods

Jon Froehlich

@jonfroehlich

COMPUTER SCIENCEUNIVERSITY OF MARYLAND

Title: Is this Large Enough? (36 pt)

• Content Level 1: What about this? (28 pt)

– Content Level 2: This is a 2nd level bullet (24 pt)

• Content Level 3: This is a 3rd level bullet (20 pt)

This is a sub-title to the title (24 pt)

Title: Is this Large Enough? (40 pt)

• Content Level 1: What about this? (36 pt)

–Content Level 2: This is a 2nd level bullet (32 pt)

• Content Level 3: This is a 3rd level bullet (28 pt)

This is a sub-title (24 pts)

Assignments

• IA03 Project Pitch

–Matt and I read, analyzed, and graded ~96 proposals

• IA04 Project Pitch Vote

–Assigned Yesterday, Due Friday 2/5 (11:59PM)

• IA05 Observation

–Assigned Today, Due Tues 2/16 (11:59PM)

Use the Calendar feature in Canvas to see all upcoming assignments and deadlines

IA05 Observation: Due Tues 2/16 (11:59PM)

1. Review Study Ethics. Read UMD IRB documentation. Determine whether your observational study of Metro users would require an exempt review, an expedited review, or a full board review.

2. Perform User Observation. Go to Metro station. Watch people interact with Metro machines (e.g., ticket machine, turnstiles). Take notes, pictures, video.

3. Analyze Data. Identify problems, patterns, and insights.

4. Write Report and Sketch Alternative Designs. Write a report synthesizing your findings and include sketches of alternative designs.

This assignment has four parts

Key Learning Outcomes Today

• Reflect on how to learn about people

• Understand importance and use of IRB

• Begin building a designer’s toolbox of user research

methods

Hall of Fame Hall of Shame

Let’s begin with a Hall of Fame/Shame!

Hall of Shame: Quitting SkypeConsistency is one of Jakob Nielsen’s 10 Usability Heuristics for User Interface Design

Jakob Nielsen

One of the most well-known “UI Experts”We’ll learn more about him throughout semester

Source: http://www.nngroup.com/articles/ten-usability-heuristics/

Hall of Shame: Quitting SkypeConsistency is one of Jakob Nielsen’s 10 Usability Heuristics for User Interface Design

Users should not have to wonder whether

different words, situations, or actions mean

the same thing. Follow platform conventions.

Jakob Nielsen

One of the most well-known “UI Experts”We’ll learn more about him throughout semester

Source: http://www.nngroup.com/articles/ten-usability-heuristics/

Hall of Shame: Quitting Skype

Normally when you think of “bad design”, you think of

the creator as being sloppy or lazy but with no ill intent.

This type of bad design is known as a “UI anti-pattern”.

Harry Brignull

Independent User Experience Designer

Curates darkpatterns.org

Hall of Shame: Quitting Skype

Normally when you think of “bad design”, you think of

the creator as being sloppy or lazy but with no ill intent.

This type of bad design is known as a “UI anti-pattern”.

Harry Brignull

Independent User Experience Designer

Curates darkpatterns.org

We call this “Hall of Shame”

Hall of Shame: Quitting Skype

Normally when you think of “bad design”, you think of

the creator as being sloppy or lazy but with no ill intent.

This type of bad design is known as a “UI anti-pattern”.

Dark Patterns are different – they are not mistakes,

they are carefully crafted with a solid understanding of

human psychology, and they do not have the user’s

interests in mind.

Harry Brignull

Independent User Experience Designer

Curates darkpatterns.org

The Beginning: Figure Out What to Design

Dix et al. Interaction Design Process

Stanford d.school Design Process Rogers et al. Interaction Design Process

Waterfall Model with Step Iteration

Though design processes differ, they share a first step, which is…?

The Beginning: Figure Out What to Design

Dix et al. Interaction Design Process

Stanford d.school Design Process Rogers et al. Interaction Design Process

Waterfall Model with Step Iteration

Despite differences, all design processes start with data gathering & identifying needs

Before You

Design

While You

Design

After You’ve

“Finished”

Your Design

User Research MethodsYou can think about applying user research methods during 3 stages of a design

Before You

Design

While You

Design

After You’ve

“Finished”

Your Design

User Research MethodsYou can think about applying user research methods during 3 stages of a design

Repeat!

Before You

Design

While You

Design

After You’ve

“Finished”

Your Design

Formative Inquiry/Evaluation

Summative Evaluation

User Research MethodsYou can think about applying user research methods during 3 stages of a design

Before You Design

While You Design

After You’ve “Finished” Your

Design

Class Activity: Brainstorm User Research MethodsLet’s list as many methods as we can under each category

• Idea 1 • Idea 1 • Idea 1

Some Questions To Help Provoke Discussion

• …what people think, believe, their ideals, cultural values?

• …how people act and what they do?

• …how people use competing products?

• …what people think of competing products?

• …what people think of your idea?

• …how your product compares to other products on market?

How can we find out…

Before You Design

While You Design

After You’ve “Finished” Your

Design

Class Activity: Brainstorm User Research MethodsLet’s list as many methods as we can under each category

• Surveys• Direct Observation• Look at demographic data• Interviews

• Interviewing users• Interviewing sales people

• Rely on prior experience• Study how people user other products (similar to

yours). Cost analysis• Study psychological models of humans• Read existing literature• Focus groups• Use device of similar product• Evaluate resources/personnel• All of the above

• Make (& evaluate) lo-fi prototype• Make a higher fidelity prototype• Make sketches & evaluate• Watch focus group use prototype• Get feedback from users/clients of

prototype• Observe• Testing/Evaluation• Eye-tracking• Questionnaire after using system

• Get feedback from users/clients• EMA or ESM (think Skype dialog box)• Automated Error Reports• Filling out bug reports• Observe• Testing/Evaluation• Eye-tracking• A/B Testing• Logging• Questionnaire after using system• Do people use it

Intentionally Left Blank

Intentionally Left Blank

Intentionally Left Blank

Intentionally Left Blank

Intentionally Left Blank

Before You Design

While You Design

After You’ve “Finished” Your

Design

• Surveys• 1:1 Interviews• Focus Groups• Observation

• Rapid ethnography• Contextual Inquiry• Online data sources• In-the-lab

• Experience Sampling Method (ESM) aka Ecological Momentary Assessment

• Diary Study• Cultural Probes• Empathy Tools

• Applying human performance models such as KLM or GOMS• Design Probes• Participatory Design• Paper Prototype Evaluation• Mid-Fi Prototype Evaluation• Wizard-of-Oz Evaluation• Lab-Based Evaluations• Alpha/Beta Deployments

• With ESM (think Skype)• With App Logging

• Heuristic Evaluation• Empathy Tools

• Application Logging• ESM (think Skype after-call

prompts)• Heuristic Evaluation• Remote Usability Testing

(screenshare)• A/B Testing

Lots of Methods: How to Choose?

Each method:• Has it’s own benefits/drawbacks• Is good for answering certain kinds of questions• Collects different types of data (& “measures” different things)• Requires different kinds of analyses• Thus, most UX/HCI researchers triangulate (use more than one)

Lots of Methods: How to Choose?

Each method:• Has it’s own benefits/drawbacks• Is good for answering certain kinds of questions• Collects different types of data (& “measures” different things)• Requires different kinds of analyses• Thus, most UX/HCI researchers triangulate (use more than one)

Note: There are literally entire classes dedicated to user research methods. We will only scratch the surface in this class!

Before we go any further, it’s important to

talk about study ethics and informed consent.

Before we go any further, it’s important to

talk about study ethics and informed consent.

This is always extremely important but particularly so in social justice topics where dealing with potentially vulnerable populations

What constitutes human subjects research?

Human Subjects Research: DefinitionCode of Federal Regulations: 45CFR46.102(f )

Human Subject means a living individual about whom an

investigator conducting research obtains:

1. data through interaction or intervention with the

individual, or

2. identifiable private information

Code of Federal Regulations

45CFR46.102(f )

Source: Code of Federal Regulations, http://www.hhs.gov/ohrp/humansubjects/guidance/45cfr46.html; UMD IRB Office, http://www.umresearch.umd.edu/RCO/New/IRBProcess.html#Definitions

Human Subjects Research: DefinitionCode of Federal Regulations: 45CFR46.102(f )

Human Subject means a living individual about whom an

investigator conducting research obtains:

1. data through interaction or intervention with the

individual, or

2. identifiable private information

Research means a systematic investigation, including research

development, testing and evaluation, designed to develop or

contribute to generalizable knowledge.

Code of Federal Regulations

45CFR46.102(f )

Source: Code of Federal Regulations, http://www.hhs.gov/ohrp/humansubjects/guidance/45cfr46.html; UMD IRB Office, http://www.umresearch.umd.edu/RCO/New/IRBProcess.html#Definitions

Institutional Review Board (IRB)

• Most professional societies, universities, government, and other research offices require researchers to provide detailed study plan information for any research activities that involves humans

• This study plan is evaluated by an Institutional Review Board (IRB) that assesses the value of the research compared to the risk/stress on participants

• A research team must receive full study approval by the IRB before conducting human subjects research

• IRBs exist to ensure that humans are not endangered physically or emotionally and that their right to privacy is protected– Yes, IRBs also exist to protect animals!

Performing human subjects research means you need IRB approval

Source: UMD IRB Office, http://www.umresearch.umd.edu/RCO/New/IRBProcess.html#Definitions; Rogers et al., Interaction Design, Chapter 15, 2011

My Experience with IRBs

A bit of history…

Tuskegee Syphilis Study

Nazi Experimentation

Aforementioned are extreme examples, but..

ABC Primetime, Basic Instincts: The Science of Evil, 2007, http://youtu.be/GHuI2JIPylk

Stanley MilgramObedience experiments, circa 1961

Milgram Obedience ExperimentWhat are some potential human subjects ethical issues?

Milgram Obedience ExperimentWhat are some potential human subjects ethical issues?

Milgram’s participants were placed in an emotionally

excruciating situation in which an experimenter

instructed them to continue administering electric

shocks to another individual despite hearing that

person’s agonizing screams of protest. The research

became, as I often told my students, the study that can

never be replicated.

Jerry Bruger, PhD

Professor of Pyschology

Santa Clara University

Source: Bruger, J., Replicating Milgram, Association for Psychological Science’s Observer, 2007

Milgram Obedience ExperimentOften touted as why IRB is necessary and important

Milgram’s work was conducted in the early 1960s

before the current system of professional guidelines

and IRBs was in place. It is often held up as the

prototypic example of why we need policies to

protect the welfare of research participants.

Jerry Bruger, PhD

Professor of Pyschology

Santa Clara University

Source: Bruger, J., Replicating Milgram, Association for Psychological Science’s Observer, 2007

Brainstorm: Human Subjects Study Principles

• Principle 1

How should we interact with participants? What should we guarantee?

IDEO Principles for Studies with Human Participants

• At the beginning of the study, identify yourself, your intent, and what you are looking for

• Describe what data you are gathering and why it’s valuable

• Get permission to use information and photos/videos you take

• Keep all information confidential

• Let people know they can decline to answer questions or stop participating at any time

• Provide compensation

• At all times, have consideration for your participants’ health, privacy, and dignity

This is what IDEO recommends for its human subjects studies

Source: IDEO Method Cards

IDEO’s principles map nicely to the “informed

consent” process, which is required of most

human subjects research

Informed Consent

• A consent form is used to provide

information to prospective subjects

to enable persons to voluntarily

decide whether or not to

participate as research subjects.

• A consent form usually contains

the basic elements of informed

consent as identified in and

required by the Federal regulations

for the protection of human

subjects (45 CFR 46).

A written form approved by IRB and signed by participants before research begins

Source: UMD IRB Office, http://www.umresearch.umd.edu/RCO/New/IRBProcess.html#Definitions

UMD IRB Template Consent Form

Source: UMD IRB Office, http://www.umresearch.umd.edu/RCO/New/IRBForms.html#consentform

Companies Have Different Standard

• Commercial entities not funded by the federal

government, who don’t plan to publish, and who do

not plan to apply for, say, FDA approval, need not

follow human subjects research regulations

• Often times, study plans are reviewed but by lawyers

rather than an IRB and concerns include legal liability

and potential for public blowback

Example: Facebook Emotion Contagion Study

• Researchers from Facebook and Cornell University manipulated the news feed of nearly 700,000 Facebook users for a week in 2012 to gauge whether emotions spread on social media

• They found that users who saw more positive posts tended to write more positive posts themselves, and vice versa.

• The study was published in the Proceedings of the National Academy of Sciences earlier in June,

Source: http://www.wsj.com/articles/facebook-study-sparks-ethical-questions-1404172292

Example: Facebook Emotion Contagion Study

Sources: http://www.wsj.com/articles/facebook-study-sparks-ethical-questions-1404172292, http://www.cnn.com/2014/07/02/opinion/klitzman-facebook-experiment/, https://www.washingtonpost.com/news/morning-mix/wp/2014/07/01/facebooks-emotional-manipulation-study-was-even-worse-than-you-thought/

Leading Study Ethics Experts Among Us

Whew, OK, back to methods…

Before You Design

While You Design

After You’ve “Finished” Your

Design

• Surveys• 1:1 Interviews• Focus Groups• Observation

• Rapid ethnography• Contextual Inquiry• Online data sources• In-the-lab

• Experience Sampling Method (ESM) aka Ecological Momentary Assessment

• Diary Study• Cultural Probes• Empathy Tools

• Applying human performance models such as KLM or GOMS• Design Probes• Participatory Design• Paper Prototype Evaluation• Mid-Fi Prototype Evaluation• Wizard-of-Oz Evaluation• Lab-Based Evaluations• Alpha/Beta Deployments

• With ESM (think Skype)• With App Logging

• Heuristic Evaluation• Empathy Tools

• Application Logging• ESM (think Skype after-call

prompts)• Heuristic Evaluation• Remote Usability Testing

(screenshare)• A/B Testing

Before You Design

• Surveys• 1:1 Interviews• Focus Groups• Observation

• Rapid ethnography• Contextual Inquiry• Online data sources• In-the-lab

• Experience Sampling Method (ESM) aka Ecological Momentary Assessment

• Diary Study• Cultural Probes• Empathy Tools

Before You Design

• Surveys• 1:1 Interviews• Focus Groups• Observation

• Rapid ethnography• Contextual Inquiry• Online data sources• In-the-lab

• Experience Sampling Method (ESM) aka Ecological Momentary Assessment

• Diary Study• Cultural Probes• Empathy Tools

ObservationSource: http://goo.gl/SPwdPE

Why Not Ask Users?

If I had asked my customers what they

wanted…

Henry Ford

Inventor / Car Designer

Industrial Revolutionary

Why Not Ask Users?

If I had asked my customers what they

wanted, they would have said a faster

horse.

Henry Ford

Inventor / Car Designer

Industrial Revolutionary

Why Not Ask Users… According to Steve Jobs

It’s not the consumers’ job to know what

they want.

Steve Jobs

Legend

Quick Design Activity

• Who are your users?

• What tasks are you trying to support?

• What are some of the problems with the state-of-the-art measuring cup (shown)?

• Both of you will sketch your new design in individually in your notebooks

You and a partner: design a better measuring cup

State-of-the-art Measuring Cup

Dark Palette

Light Palette

Light Palette

Light Palette