defining user research methodologies: a pragmatic approach

Monday, April 30, 2012

DEFINING USER RESEARCH METHODOLOGIESA PRAGMATIC APPROACH

M I N N E W E B C O N 2 0 1 2

DEFINING USER RESEARCH METHODOLOGIESA PRAGMATIC APPROACH

M I N N E W E B C O N 2 0 1 2

#mwcresearch

ZACKSENIOR USER EXPERIENCE DESIGNERUSER RESEARCH PRACTICE LEAD

@zacknaylor

DAVEUSER EXPERIENCE DESIGNER

PHD CANDIDATE Tech Comm & User Experience

@Dave_L_Jones

WHAT ABOUT YOU?

WHAT WE’LL COVER

Common user research methods

WHAT WE’LL COVER

Common user research methods

Identifying the appropriate use and combination of methods

WHAT WE’LL COVER

Identifying the appropriate use and combination of methods

WHAT WE’LL COVER

Goals in research planning

WHAT WE’LL COVER

Research data analysis

WHAT WE’LL COVER

USER RESEARCH

IS NOT

MARKETING RESEARCH

USER RESEARCH

IS NOT

MARKETING RESEARCH

...but there is some overlap

USER RESEARCH

IS NOT

MARKETING RESEARCH

USER RESEARCH

IS NOT

MARKETING RESEARCH

We do this

USER RESEARCH

IS NOT

MARKETING RESEARCH

We do this

USER RESEARCH

IS NOT

MARKETING RESEARCH

We do this

We DON’T do this

MARKETING RESEARCHUSER RESEARCH

MARKETING RESEARCH

Preferences

Opinions

Desires

USER RESEARCH

MARKETING RESEARCH

Preferences

Opinions

Desires

USER RESEARCH

Behavior

Mental & Physical Context

RESEARCH SPECTRUM

http://www.flickr.com/photos/rosenfeldmedia/2159500714/in/set-72157603511616271/ Indi Young - Mental Models: Rosenfeld Media

RESEARCH SPECTRUM

overlap

WHY DO USER RESEARCH?

Remember when we said that design solves problems?

Remember when we said that a UX process starts with defining the problem?

The systematic investigation into and study of materials and sources in order to establish facts and reach new conclusions.

Research tells us what the problem is.

Research tells us why it’s a problem.

Research shows us how to fix it.

Research tells us why it’s a problem.

BENEFITS OF USER RESEARCH

Throughly defines the problem.

Informs design decisions.

Provides direction & priority.

SHOWS US THE “WHAT”

TEACHES US THE “WHY”

GUIDES US TO THE “HOW”

EXAMPLE PROJECT NEEDS

“WHAT”

“WHY”

“HOW”

Stated: What Research Tells Us:

“WHAT”

“WHY”

“HOW”

“We need a website redesign”

“WHAT”

“WHY”

“HOW”

“We need a website redesign” The website doesn’t meet users’ expectations

“WHAT”

“WHY”

“HOW”

“The current site isn’t working for us”

The website doesn’t meet users’ expectations

“WHAT”

“WHY”

“HOW”

There are usability flaws in the design

“WHAT”

“WHY”

“HOW”

“We should update the look & feel”

“WHAT”

“WHY”

“HOW”

We need to design clearercall to action buttons.

“WHAT”

“WHY”

“HOW”

“WHAT”

“WHY”

“HOW”

“WHAT”

“WHY”

“HOW”

Vague & Undefined

“WHAT”

“WHY”

“HOW”

Vague & Undefined

“WHAT”

“WHY”

“HOW”

Vague & UndefinedClear & Actionable

vs. QUANTITATIVEQUALITATIVE

BEWARE OF THE DATA TYPE

vs. X QUANTITATIVEQUALITATIVE

Just remember

X QUANTITATIVEQUALITATIVE

Just remember

We need both!

X QUANTITATIVEQUALITATIVE

vs. X QUALITATIVEQUANTITATIVESubjective

“Non-measurable”NOT Numerical

Concepts

Objective“Measurable”

Numerical DataStatistics

QUALITATIVEQUANTITATIVE

QUALITATIVE

QUANTITATIVE

BEWARE OF THE DATA TYPEQUALITATIVEQUANTITATIVE

Page Views

Bounce Rate

Time On Site

Yes/No

True/False

Page Views

Bounce Rate

Time On Site

Yes/No

True/False

Expectations

Reactions

Confusion

Comprehension

Behavior

BEWARE OF THE DATA TYPEQUANTITATIVE QUALITATIVE

QUANTITATIVE

QUALITATIVE

QUANTITATIVE QUALITATIVE

informs QUANTITATIVE QUALITATIVE

informs

“WHAT”

informs

“WHAT”

informs

“WHAT” “WHY”

informs “WHAT” “WHY”

informs i.e. Sign-up page bounce rate it high.

informs i.e. Sign-up page bounce rate it high. i.e. Call to action text is confusing.

WHEN SHOULD YOU DO USER RESEARCH?

Timeframe Beginning of the project During Design Production Project Completion

- inform direction & scope- throughly defines the problem- provides insights for next steps

- re-enforce direction- validate design decisions- acquire design feedback at significantly lower cost

- gauge progress/success- guides product direction- discover areas for improvement

- Contextual Inquiry- Field Study/Ethnography- Stakeholder Interviews- Surveys

- Usability Testing- Card Sorting- Personas- Mental Models

- Usability Testing- Site Search Analytics- Web Analytics- A/B Testing/Multivariate Testing

Benefit

Example Method/Use

BONUS ROUND

ONGOING RESEARCH

BONUS ROUND

ONGOING RESEARCH

BONUS ROUND

ONGOING RESEARCH

How: - Create a solid and sustainable research plan for a continuous feedback loop from your customers/users

BONUS ROUND

ONGOING RESEARCH

BONUS ROUND

ONGOING RESEARCH

Why: - Your audience changes and evolves over time

BONUS ROUND

ONGOING RESEARCH

- Your product/service/website will attract new audience segments

BONUS ROUND

ONGOING RESEARCH

- Your product/service/website will attract new audience segments

-Other products/services/websites introduce new expectations for interacting with your information

COMMON RESEARCH SKILLS

INTERVIEWING

OBSERVATION

LISTENING

ANALYSIS (SENSE-MAKING)Monday, April 30, 2012

COMMON RESEARCH SKILLS

INTERVIEWING

OBSERVATION

LISTENING

COMMON RESEARCH SKILLSESSENTIAL

INTERVIEWING

OBSERVATION

LISTENING

NERDERY RESEARCH METHODS

Usability Testing

Surveys

Card Sorting

Stakeholder Interviews

User Interviews

Contextual Inquiry

Personas

Field Study/Ethnography

Site Search Analytics

Web Analytics

A/B Testing : Multivariate Testing

Mental Models

Usability Testing

Surveys

Card Sorting

User Interviews

Contextual Inquiry

Personas

Web Analytics

Mental Models

“WHAT”

Usability Testing

Surveys

Card Sorting

User Interviews

Contextual Inquiry

Personas

Web Analytics

Mental Models

“WHY”

Usability Testing

Surveys

Card Sorting

User Interviews

Contextual Inquiry

Personas

Web Analytics

Mental Models

“HOW”

STAKEHOLDER INTERVIEWS

WHAT ARE STAKEHOLDER INTERVIEWS?

Definition:

WHAT ARE STAKEHOLDER INTERVIEWS?

one on one conversations with client champions of a project aimed at gaining understanding of three overarching themes of information:

1. Project Context

2. Target Audience

3. Project Success

Definition:

THE “HOW-TO” OFSTAKEHOLDER INTERVIEWS

4. Analyze

5. Insights Report

1. Recruiting

3. Conduct Interviews

2. Research Plan

Review what you learned. Did it match your hypothesis? What patterns emerged?

Create appropriate documentation to communicate what you found to the team.

Identify the stakeholders on the project, schedule ONE HOUR with each stakeholder separately.

Watch users completing relevant goals & tasks in their own context.

Establish a clear focus for what you expect to find & what you hope to learn.

WEB ANALYTICS

WHAT ARE WEB ANALYTICS?

Definition:

WHAT ARE WEB ANALYTICS?

Web analytics is the measurement, collection, analysis and reporting of internet data for purposes of understanding and optimizing web usage. (Wikipedia)

Definition:

THE “HOW-TO” OFWEB ANALYTICS

4. Insights Report

5. Pinpoint knowledge gap(s)

1. Identify data needs

3. Analyze

2. Gather data

Target unknowns and craft a qualitative research plan to fill known gaps in understanding. Outline how you plan to uncover the “why” of the problem.

What information and data are useful for defining the problem?

Segment where relevant and cross reference the raw data to find patterns and/or correlations.

Collect that data and information within a determined time period for analysis.

CONTEXTUAL INQUIRY

WHAT IS CONTEXTUAL INQUIRY?

Fancy Definition:

a field data-gathering technique that studies a few carefully selected individuals in depth to arrive at a fuller understanding of the work practice across all customers.

Fancy Definition:

- Hugh Beyer and Karen Holtzblatt - Contextual Design: Defining Customer-Centered Systems

Fancy Definition:

Non-Nerd Version:

Fancy Definition:

go watch people work in their own context.Non-Nerd Version:

Fancy Definition:

go watch people work in their own context.Non-Nerd Version:

THE “HOW-TO” OF CONTEXTUAL INQUIRY

4. Analyze

5. Report

1. Recruiting

3. Observe

2. Research Plan

Create appropriate documentation to communicate what you found to the team.

Contact & schedule the people you will observe. THIS IS THE MOST IMPORTANT STEP.

Watch users completing relevant goals & tasks in their own context.

Establish a clear focus for what you expect to find & what you hope to learn.

WHAT’S THE BENEFIT OF CONTEXTUAL INQUIRY?

Environment

Applications

Issues

What is their physical location like? How does it impact the design or how they use it?

What other hardware or software are they using to do their work? Can (or should) they be integrated? Can the new design eliminate the need of these factors?

Are there current frustrations or problems with the existing design? What about with their physical environment or other systems and processes? Can the new design support those?

What are the steps people are taking to accomplish those goals?

What are the high level priorities of the people using the current design? What are they trying to accomplish?

WHAT’S THE BENEFIT OF CONTEXTUAL INQUIRY?

Who does the person work with to accomplish a goal or task?

Do people have “cheat-sheets” or other materials to help them accomplish goals and tasks?

Are people creating ways to work around a poor design now?

Are there several ways in which people are accomplishing the same goal or task? Should the design support one? Both?

What causes someone to begin down a path of completing a goal?

Work-Arounds

Triggers

Variation(s)

Partners

“Crutches”

RECOMMENDATIONS FOR CONTEXTUAL INQUIRY

Recruiting

Timeline

Be sure to observe an appropriate, representative sample of your target audience.Aim for 3-5 participants separately.

Allow 1-2 weeks for recruiting effort. (Varies depending on the project & participant availability)Allow 1-2 weeks for conducting the research. (Assuming 5 participants)Allow AT LEAST 1 week for analysisAllow 1 week to create a report.

Approximately 4-8 weeks total

CARD SORTING

WHAT IS CARD SORTING?

Definition:

WHAT IS CARD SORTING?

method of gathering data to inform the information architecture, navigation, taxonomy and labeling of a design

Definition:

THE “HOW-TO” OFCARD SORTING

4. Analyze

5. Report

1. Recruiting

3. Conduct Sorts

2. Research Plan

Collect your findings and perform exploratory analysis or statistical analysis (or both).

Create appropriate documentation that conveys the findings from the research.

Determine who you will conduct card sorts with, write a screener to ensure you recruit your target audience from the responses & schedule the participants.

Conduct the card sort(s) with the target audience and selected data/content.

Where will the sorts take place? (remote or in person? with how many?) As before, Establish a clear focus for what you expect to find & what you hope to learn.

2.1 Logistics Choose your data, expected analysis method and sort method (open/closed sort). Single or group sorts?

WHAT’S THE BENEFIT OFCARD SORTING?

Labeling

Categorization

Navigation

Grouping

Hierarchy

WHAT’S THE BENEFIT OFCARD SORTING?

Informed Information Architecture

USABILITY TESTING

WHAT IS USABILITY TESTING?

Definition:

WHAT IS USABILITY TESTING?

a form of gathering feedback from actual users of a design by having them attempt to complete intended goals and tasks with said design.

Definition:

THE “HOW-TO” OFUSABILITY TESTING

4. Analyze

5. Report

1. Recruiting

3. Conduct Tests

2. Research Plan

Discuss test results with any participating team members & review recordings if available.

Create appropriate documentation that conveys the findings from the research.

Determine who you will conduct usability testing with, write a screener to ensure you recruit your target audience from the responses & schedule the participants. Again, a CRITICAL step.

Conduct the test (ideally with an experience moderator) & observe, while taking notes.

Where will the tests take place? What will the research cover? As before, Establish a clear focus for what you expect to find & what you hope to learn.

WHAT’S THE BENEFIT OF USABILITY TESTING?

Path Taken

Impression

Expectations

Level of difficulty

Task Completion

What were the steps involved in completing a given task?

Did the user(s) understand the overall message and intent that the design meant to convey?

Is the solution designed meeting expectations of those intended to use it?

If so, how difficult was it for them to complete the task(s)? Why?

Can the users complete the available tasks of the design?

VARIATIONS OFUSABILITY TESTING

Guerilla

In-Person, Moderated

Remote, Un-moderated

Remote, Moderated

Usability testing done with minimal recruiting effort and logistical planning. Common locations are coffee shops, bars, offices, etc.

Sessions are conducted at a physical location with a live, in-person moderator leading the session(s) with participants.

Sessions are conducted using an online service that allows users to participate at their convenience, without a moderator.

Research is conducted via “live recruiting” from an existing website. Participants are immediately connected with a moderator from a remote location.

Guerilla

Remote, Moderated

Guerilla

Remote, Moderated

Better

Guerilla

Remote, Moderated

RECOMMENDATIONS FORUSABILITY TESTING

Recruiting

Timeline

Do your own recruiting if possible. If this is not possible, work closely with a recruiting agency to ensure the participants being recruited match your target audience.(especially for the particular research you’re doing)

Aim for 5-10 participants. (dependent on the study)

Allow 1-2 weeks for recruiting effort. (Varies depending on the recruiting method)Allow 1 week for conducting the tests. (Assuming 5 participants)Allow 3 days to 1 week for analysis.Allow 3 days to 1 week to create a report.

Approximately 3-6 weeks total

WHERE TO BEGIN

STEP 1:UNDERSTANDING BUSINESS GOALS

What are the stated problems?

What impact (and implications will our design have on those goals?

How do we arrive at the core problem or root cause?

STEP 1:UNDERSTANDING BUSINESS GOALS

STEP 2:DEFINING YOUR INFORMATION NEEDS

What info is necessary for you to meet those business goals?

What information do you need to successfully design?

What information do you have available?

Where are you knowledge “blind spots”?

STEP 2:DEFINING YOUR INFORMATION NEEDS

STEP 3:CRAFT YOUR APPROACH

Which methods are applicable to reach your information needs?

How will you gather information about the target audience?

How do you plan to use that information?

What are the impacts of that data in design?

STEP 3:CRAFT YOUR APPROACH

ACTIVITY:USER RESEARCH PLAN

30-40 minutes

PUTTING IT ALL TOGETHER

FICTIONAL PROJECT: WEB APP REDESIGN

“We need to redesign our web application to increase customer engagement”

Let’s get started:

THE “WHAT”

Targeted SurveysTHE “WHAT”

Targeted Surveys Web AnalyticsTHE “WHAT”

THE “WHY”

User InterviewsTHE “WHY”

User Interviews Contextual InquiryTHE “WHY”

THE “HOW”

Usability TestingTHE “HOW”

Usability Testing

“HOW”

Project Overview:

Targeted Surveys

Web Analytics

User Interviews

Contextual Inquiry

Prototype[design & development}

“WHY”“WHAT”

iteration(s)

DATA ANALYSIS

STEP 1:REVISIT YOUR RESEARCH GOALS

Did you collect the information necessary to design?

Were the goals of your research met?

Have you gained an understanding of the problem space?

STEP 1:REVISIT YOUR RESEARCH GOALS

STEP 2:BUILD A MODEL - DATA INTO KNOWLEDGE

Craft a visual representation of all the raw data you collected.

Pull out patterns, problems and context.

Prioritize the issues and patterns.

Brainstorm an approach to design from your new found knowledge.

STEP 2:BUILD A MODEL - DATA INTO KNOWLEDGE

ACTIVITY:DATA ANALYSIS

30-40 minutes

QUESTIONS?

@zacknaylor

@Dave_L_Jones

Fire away...

Keep the party goin’...

THANKS

defining user research methodologies: a pragmatic approach

user research methodologies

notmarketing research

thismarketing research

research planningmonday

user researchis

guides us

website doesnt

whatinforms design decisions

Design

methodologies for defining and assessing ecosystem services

pragmatic group9

chapter 7: project planning: task determination and...

pragmatic markers

pragmatic connectives

pragmatic agility

pragmatic failure

management methodologies inmanagement methodologies in...

pragmatic odoo openerp amazon aws platform - pragmatic

pragmatic ischaemic stroke thrombectomy evaluation:...

pragmatic smalltalk

pragmatic competence and pragmatic instruction in...

pragmatic microservices

defining entrepreneurial · pdf file ·...

pragmatic thermal

tapping the potential -...

pragmatic consciousness.pdf

pragmatic view

defining methodologies and data sources for project-level

pragmatic linguistics»