practical user research: a crash course

PRACTICAL USER RESEARCH:A CRASH COURSE

#MWC_PRACTICALUX

1

INTROS

2

h2p://www.flickr.com/photos/mytudut/5188623575/sizes/z/in/photostream/

http://www.flickr.com/photos/mytudut/5188623575/sizes/z/in/photostream/

http://www.flickr.com/photos/mytudut/5188623575/sizes/z/in/photostream/

ABOUT ME

3

Ma3hew Doty @ma$hewjdoty

User Experience LeaderWhole-‐brained Thinkificator All-‐around Swell Guy

4

http://marketingpie.risdall.com/wp-content/uploads/2013/03/Tron-fight-for-the-user.jpg



WHO I’VE DONE WORK FOR TargetBest BuyLife Time FitnessOfficeMaxSearsMarrio$

KraK FoodsHarland ClarkeArrow ElectronicsHalliburtonPennwellAshley Furniture

Goldman SachsPNCAT&TCTU OnlineAIU OnlineJC Penny

© 2013 EVANTAGE CONSULTING

ABOUT EVANTAGE

6

-‐ BouSque customer experience consultancy based in Minneapolis.

-‐ Founded in 1999.-‐ Consistently ranked one of the best places to work.

-‐ Offers... -‐ Customer Experience Strategy -‐ Digital Strategy, Experience Design, and

ImplementaSon -‐ Performance OpSmizaSon


ABOUT YOU...

7

• Who are you?• What’s your problem?• What do you want from me?

http://www.flickr.com/photos/86979666@N00/7623744452/sizes/z/in/photostream/




OBJECTIVES 1.Demonstrate how quality user insights are actually within the reach of any organizaSon, project, budget, or schedule… even yours!

2.Empower you with the basic tools and skills you need to start doing pracScal research of your own.

3.Send you off with at some kind of real insight for your parScular project/problem.

8

OUR AGENDA

9

1:00 Learn about user research and select your technique1:30 Learn your technique2:00 Design & conduct your study (take breaks as needed)3:15 Document & Present your findings 3:45 Wrap up

WARNING

10

We will be moving fast I will require your parScipaSonThis will be awesome

LEARN ABOUT USER RESEARCH & SELECT YOUR TECHNIQUE

11

WHAT IS USER RESEARCH?

12

USER RESEARCH

13

An set of acSviSes that help us... 1. Understand the people who will be

actually using what we’re designing. 2. Validate our decisions/assumpSons

throughout the project.

PRACTICAL USER RESEARCH

14

A flexible approach to user research which delivers many of the benefits of user insight while remaining sensiSve to Sme and budget constraints.

WHY SHOULD WE DO USER RESEARCH?

15

© 2013 EVANTAGE CONSULTING 16h$ps://www.youtube.com/watch?v=RnZyc5LrVEs

https://www.youtube.com/watch?v=RnZyc5LrVEs

https://www.youtube.com/watch?v=RnZyc5LrVEs

SO, WHAT WENT WRONG?

17

NONE OF THE DECISIONS WERE BASED ON DATA!

WHY SHOULD WE DO USER RESEARCH?

18

TO GET DATA THAT HELPS US...


OVERCOME PERSONAL BIAS

19


COUNTER IVORY TOWER DESIGN

20

http://www.flickr.com/photos/dplanet/94441587/

http://www.flickr.com/photos/justinhoch/4998133254/



REPLACE THE ELASTIC USER

21


22

16© 2006 Steve Krug. Used with permission of the author.

16

AVOID RELIGIOUS DEBATES



MYTHS OF USER RESEARCH

23

1. WE DON’T NEED TO DO USER RESEARCH

“We’re smart enough to get it right!”“Users don’t know what they want!”“We’ll just fix anything aKer we go live.”

24

h2p://www.flickr.com/photos/evertwh/2723737126/sizes/l/in/photostream/

http://www.flickr.com/photos/evertwh/2723737126/sizes/l/in/photostream/

http://www.flickr.com/photos/evertwh/2723737126/sizes/l/in/photostream/

2. WE DON’T HAVE THE BUDGET

Too oKen, we think “research” automaScally means expensive labs, recruiSng fees and expensive research partners.

25

h2p://www.flickr.com/photos/danmoyle/5634567317/sizes/l/in/photostream/

http://www.flickr.com/photos/danmoyle/5634567317/sizes/l/in/photostream/

http://www.flickr.com/photos/danmoyle/5634567317/sizes/l/in/photostream/

3. WE DON’T HAVE THE TIME

“We can only test our exact target audience.”“I can’t have my development team sirng on their hands!”

26

http://www.flickr.com/photos/alancleaver/2661425133/sizes/o/in/photostream/




THE REALITY• User research yields cost & Sme savings as well as happy, loyal, producSve users.

• Insights can be gained without all the bells and whistles. • Parallel paths & informal methods can reduce or eliminate extra Sme needed.

27

WHAT KIND OF RESEARCH SHOULD WE DO?

28


Access to UsersTi

me

& B

udge

t

FORMALWHAT KIND OF RESEARCH SHOULD WE DO?

29

FORMAL

INFORMAL


Access to UsersTi

me

& B

udge

t

FORMALWHAT KIND OF RESEARCH SHOULD WE DO?

30

SECONDARY PRIMARY


USER RESEARCH DECISION MODEL

31

Access to UsersTi

me

& B

udge

t

FORMALPRIMARY

FORMALSECONDARY

INFORMALSECONDARY

INFORMALPRIMARY

Adapted from http://matthewjdoty.com/img/research.png Used with Permission

http://matthewjdoty.com/img/research.png

http://matthewjdoty.com/img/research.png

WHEN SHOULD WE DO USER RESEARCH?

32

THE SHORT ANSWER...

33

EARLY & OFTEN

THE LONGER ANSWER...

34

IT DEPENDS...


WATERFALL

35

DISCOVER

DEFINE

DESIGN

DEVELOP

DEPLOY

EthnographyUser InterviewsFocus Groups SurveysBaseline User Tes\ng Web Metrics AnalysisSecondary Research

Card Sor\ngTree-‐tes\ngConcept Tes\ng Usability Tes\ng

Concept Tes\ng Usability Tes\ng

Usability Tes\ng Surveys

Baseline User Tes\ng Web Metrics Analysis

A/B Tes\ngMVT


AGILE

36

UX SPRINT 1 UX SPRINT 2 UX SPRINT 3

DEV SPRINT 0 DEV SPRINT 1 DEV SPRINT 2 DEV SPRINT 3ENVISIONING

EthnographyUser InterviewsFocus Groups SurveysBaseline User Tes\ng Web Metrics AnalysisSecondary Research

Card Sor\ngTree-‐tes\ngConcept Tes\ng

Usability Tes\ng



Usability Tes\ng Usability Tes\ng

WHO SHOULD DO THE RESEARCH?

37


YOU!

38

http://www.flickr.com/photos/subharnab/2721507697/sizes/z/in/photostream/



WHAT TECHNIQUES SHOULD I USE?

39

WHAT QUESTIONS ARE YOU TRYING TO ANSWER?How do they want to interact?Card SorSng (open)Design Games/WorkshopsFocus GroupsSurveysBaseline Usability TesSngTask Analysis

40

Who is our audience & what do they need?Contextual Inquiry (Ethnography)InterviewsFocus GroupsSurveysDiaries

What do they think of what we’ve done?Concept TesSngUsability TesSngCard sorSng (closed/reverse)Tree TesSngContextual InquiryAnalysis of Web MetricsA/B, MVT tesSng

WHAT QUESTIONS ARE YOU TRYING TO ANSWER?How do they want to interact?Card SorVng (open)Design Games/WorkshopsFocus GroupsSurveysBaseline Usability TesVngTask Analysis

41

Who is our audience & what do they need?Contextual Inquiry (Ethnography)InterviewsFocus GroupsSurveysDiaries

What do they think of what we’ve done?Concept TesSngUsability TesVngCard sorVng (closed/reverse)Tree TesVngContextual InquiryAnalysis of Web MetricsA/B, MVT tesSng

WHAT IS CARD SORTING?A research technique that aims to understand how people organize and group informaSon. Results can be used to structure the site/app in more intuiSve way.

42

WHAT IS TREE TESTING?A research technique that aims to uncover issues with labeling & findability. Results are used to inform the design/organizaSon of your menu/navigaSon.

43

WHAT IS USABILITY TESTING?A research technique that aims to validate decisions and catch usability issues. Results can be used to...• Fix issues before producSon• Inform the next release• Inform a redesign

44

WHICH TECHNIQUE SHOULD YOU USE?

45


NOW WHAT? • Break into groups based on your technique.

• Prepare to be amazed at what you can do.

46

http://www.flickr.com/photos/21496790@N06/5065834411/sizes/l/in/photostream/



OUR AGENDA

47


LEARN YOUR SELECTED TECHNIQUE

48


THE CORE STAGES OF USER RESEARCH 1. ArSculate your objecSves2. Design the study 3. Recruit parScipants4. Conduct the study5. Analyze the findings6. Document the findings & recommendaSons

49

USABILITY TESTING 101

50

HOW DOES USABILITY TESTING WORK?1. Show the user interface (a sketch, mockup, prototype or the actual site/app)

2. Ask the parScipant to perform a task

3. Observe interacSons & take notes

51

HOW MANY PARTICIPANTS ARE NEEDED?

52

AT LEAST

5http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

HOW DO I DESIGN A USABILITY STUDY?1. IdenSfy the pages/screens/elements where you’d like to have user feedback

2. Clearly arSculate and document your goals with each page/screen/element and what successful interacSon looks like

3. Create realisSc scenarios & tasks that will elicit credible feedback.

53


IDENTIFY PAGES/SCREENS/ELEMENTS• Which ones are required for the criScal tasks?• Which ones perform poorly in your web analyScs?• Which ones are causing the most internal disagreement (e.g. religious debates)?

• Which ones are falling vicSm to ivory tower design?• Which ones represent a risk you are taking?

54

EXAMPLE: ARTICULATE YOUR GOALSThe goal of this study is to observe user interacSons with the site to see if parScipants...• Are able to successfully use the navigaSon (main and secondary) to complete

tasks.• Are successfully able to idenSfy links (they know what a link is and what is not

link).• Understand terminology and labels.

55

EXAMPLES: SCENARIOS & TASKSLet’s say that you entered [SEARCH TERM} into Google. and clicked on the link for [WEBSITE]. This is the home page of the new site. Without clicking on anything, take a few moments to look it over and take it in. (COUNT TO 10, THEN ASK THE FOLLOWING QUESTIONS)• What are your first impressions? • What can you do?• What would you do first?• What’s missing?

56

EXAMPLES: SCENARIOS & TASKSFrom what we understand you are in the market for a [PRODUCT]. I’d like you show me what you would do find and add [PRODUCT] to your cart.

Tell me about a \me that you were looking for [PRODUCT]. I’d like you now to put yourself back in that situa\on and show me what you would do find and add [PRODUCT] to your cart.

57

CARD SORTING 101

59

HOW DOES CARD SORTING WORK?Open SorSng 1. People are given a set of “cards”

that represent features/content of the site/app.

2. They organize informaSon into groups which make sense to them.

3. They name each of the groups.

60

Closed SorSng1. People are given a set of “cards”

that represent features/content of the site/app.

2. They organize informaSon into several pre-‐defined groups in a way that makes sense to them.

HOW DOES CARD SORTING WORK?

61


IDEAS FOR CREATING YOUR “CARDS”

62

• Use your current site and create a card for items in your navigaSon & important features.

• Create a conceptual feature list for the new site or app you are thinking of building and use the items in this list.

• Look to your web metrics to see what areas of the site may be experiencing issues related to site organizaSon.


HOW MANY CARDS SHOULD I MAKE?

63

20 -‐ 50


64

In card sorSng, “...there's only a 0.75 correlaSon between the results from five users and the ulSmate results.”

“You must test fiKeen users to reach a correlaSon of 0.90, which is a more comfortable place to stop.”

http://www.nngroup.com/articles/card-sorting-how-many-users-to-test/

AT LEAST

15




HOW DO I DESIGN A CARD SORT?

65

h2p://uxpunk.com/websort/

http://uxpunk.com/websort/


TREE TESTING 101

67

HOW DOES TREE TESTING WORK?1. Present a basic menu of opSons.

2. Ask people to find key features and content.

3. Analyze how oKen tasks are successfully completed.

68


IDEAS FOR CREATING YOUR TEST• Which navigaSon elements are needed for criScal tasks?• Which ones perform poorly in your web analyScs?• Which ones are most likely to lead to revenue generaSon?• Which ones are suspected of poor labeling?

69


70

• Since it’s similar to usability tesSng, some say at least 5.

• Since it’s Similar to card sorSng, some say at least 15.

• Tree tesSng vendors will recommend a minimum of 40-‐50.

IT DEPENDS...

QUICKER TURNAROUND

MORE RELIABLE


HOW DO I DESIGN A TREE TEST?

71

h2p://uxpunk.com/plainframe/

http://uxpunk.com/plainframe/


OUR AGENDA

73


DESIGN & CONDUCT YOUR STUDIES

74

DOS AND DON’TS OF IN-‐PERSON RESEARCHDo...• Dress appropriately• Bring business cards• Ask open-‐ended quesSons• Use words other than those used in

what you are researching• Ask the parScipant to “think out

loud”

Don’t...• Sigh or groan if the parScipant

makes an incorrect choice or misstates something

• Ask leading quesSons • Ask closed-‐ended/manipulaSve

quesSons• “Help” or correct the parScipant

75

DESIGN & CONDUCT YOUR STUDIESDesign1. Build your study based on real

problems you face 2. Work as a group to solve issues &

quesSons but use me as a resource if you get stuck

Conduct 1. Use your group as study subjects2. Decide within your group whose

studies you will run3. Run your studies4. Work as a group to solve issues &

quesSons but use me as a resource if you get stuck

76

HOW DID THAT GO?

77

OUR AGENDA

78


DOCUMENT & PRESENT YOUR FINDINGS

79


HOW DO I ANALYZE A CARD SORT?

80

h2p://uxpunk.com/websort/




IT’S ALL ABOUT RELATIONSHIPS

81

VerScal lines closer to the leK indicate stronger relaSonships between items

VerScal lines further to the right indicate weaker relaSonships between items


HOW DO I ANALYZE A TREE TEST?

82

h2p://uxpunk.com/plainframe/




SUCCESSES VS. FAILS

83


PATHS TAKEN

84

Imagine you are invesSgaSng universiSes and wish to obtain a Master's degree. Where would you expect to find out if this university offered the parScular Master's you were looking for?

ParVcipant 1:1. Departments & Academics > Graduate ProgramsParVcipant 21. Future Students > Graduate Students 2. Departments & Academics > Graduate ProgramsParVcipant 31. Future Students > Graduate Students 2. Departments & Academics > Graduate ProgramsParVcipant 41. Future Students > Graduate Students 2. Departments & Academics > Graduate ProgramsParVcipant 51. Departments & Academics > Graduate Programs

HOW DO I FIND THE “NUGGETS”?

85


NUGGETS

86


FINDING THE NUGGETS

87

• ParScularly impressive/inspiring moments/results

• “Eureka!” moments (yours or the parScipant’s)

• “What the..?!?!” moments/results• Surprising moments/results

http://www.flickr.com/photos/flyingsaab/4732416270/




EXAMPLES OF NUGGETS • ParScipants struggled to find the login.• Unexpectedly, parScipants engaged in horizontal scrolling behavior and expressed that they preferred it.

• Racquetball and kids camps showed a very strong relaSonship.

• Everyone found yoga on the first try.

88

HOW DO I DELIVER THE FACTS IN COMPELLING WAYS?

89

TELL A COMPELLING STORY“The next Sme you struggle with gerng people on board with your projects and ideas, simply tell them a story... ...storytelling is the only way to plant ideas into other people's minds.”

Leo WidrichThe Science of Storytelling: Why Telling a Story is the Most Powerful Way to AcSvate Our Brains(h2p://lifehacker.com/5965703/the-‐science-‐of-‐storytelling-‐why-‐telling-‐a-‐story-‐is-‐the-‐most-‐powerful-‐way-‐to-‐ac\vate-‐our-‐brains)

90

http://blog.bufferapp.com/science-of-storytelling-why-telling-a-story-is-the-most-powerful-way-to-activate-our-brains

http://blog.bufferapp.com/science-of-storytelling-why-telling-a-story-is-the-most-powerful-way-to-activate-our-brains

http://lifehacker.com/5965703/the-science-of-storytelling-why-telling-a-story-is-the-most-powerful-way-to-activate-our-brains

http://lifehacker.com/5965703/the-science-of-storytelling-why-telling-a-story-is-the-most-powerful-way-to-activate-our-brains

CONNECT THE DOTS... IN THE STUDY

91

ParScipants do not feel

confident about the purchaseParScipants spent a lot of Sme

going back & forth between the PLP and the PDP.

ParScipants commented size and fit are the most important consideraSons when buying something like this online.

ParScipants were reluctant to add items to their cart.

CONNECT THE DOTS... OUTSIDE THE STUDY

92

User Research

Stakeholder Interviews

CompeVVve Analysis

Web AnalyVcs

Findings in Context


TAKE 10 MINUTES...• Look for the nuggets in your study.

• Start connecSng dots and forming a story.

93

http://www.flickr.com/photos/33909700@N02/3159761620/sizes/o/in/photostream/





HOW DID THAT GO?WHO WANTS TO SHARE?

94

http://www.flickr.com/photos/jakecaptive/3205277810/sizes/l/in/photostream/

THOUGHTFUL PAUSEUser research does not uncover everything.

We should manage our client’s/team’s expectaSons.

95

OUR AGENDA

96


WRAP UP

97

WHEN SHOULD I HIRE EXPERTS?

98



99

Access to UsersTi

me

& B

udge

t

FORMALPRIMARY

FORMALSECONDARY

INFORMALSECONDARY

INFORMALPRIMARY



100

When you need...• Dedicated resources• Specialized experSse• Help to avoid mistakes• External perspecSves• A fresh set of eyes• An objecSve point of view• Help negoSaSng differences of opinion• Training

Adapted from: h2p://www.fresh-‐sprouts.com/2011/04/when-‐to-‐hire-‐consultants/

h2p://www.planning.org/consultants/choosing/part1.htmh2p://managementconsulted.com/consul\ng-‐skills/6-‐reasons-‐why-‐companies-‐hire-‐management-‐consultants-‐

that-‐charge-‐2-‐million-‐for-‐3-‐months-‐of-‐work/h2p://www.nonprofi\nclusiveness.org/why-‐hire-‐consultant-‐0

http://www.fresh-sprouts.com/2011/04/when-to-hire-consultants/

http://www.fresh-sprouts.com/2011/04/when-to-hire-consultants/

http://www.planning.org/consultants/choosing/part1.htm

http://www.planning.org/consultants/choosing/part1.htm

http://managementconsulted.com/consulting-skills/6-reasons-why-companies-hire-management-consultants-that-charge-2-million-for-3-months-of-work/




http://www.nonprofitinclusiveness.org/why-hire-consultant-0

http://www.nonprofitinclusiveness.org/why-hire-consultant-0

GUIDING PRINCIPLES

101

h2p://www.flickr.com/photos/ekilby/2484798276/sizes/o/in/photostream/

GET USER INSIGHT EARLY & OFTEN

102

http://www.flickr.com/photos/ekilby/2484798276/sizes/o/in/photostream/

http://www.flickr.com/photos/ekilby/2484798276/sizes/o/in/photostream/

h2p://www.flickr.com/photos/ericmcgregor/346990046/sizes/m/in/photostream/

FOCUS ON THEHEAVY HITTERS

103

http://www.flickr.com/photos/ericmcgregor/346990046/sizes/m/in/photostream/

http://www.flickr.com/photos/ericmcgregor/346990046/sizes/m/in/photostream/

h2p://www.flickr.com/photos/supersonicphotos/4483487579/sizes/l/in/photostream/

SOMETHING IS BETTER THANNOTHING

104

http://www.flickr.com/photos/supersonicphotos/4483487579/sizes/l/in/photostream/

http://www.flickr.com/photos/supersonicphotos/4483487579/sizes/l/in/photostream/

practical user research: a crash course

Technology