learning from users in their natural habitat -

96
Learning from Users in their Natural Habitat Sara Cambridge [email protected] @maybanks 2015 UXPA

Upload: uxpa-international

Post on 15-Aug-2015

132 views

Category:

Design


1 download

TRANSCRIPT

Page 1: Learning From Users in their Natural Habitat -

Learning from Users in their Natural Habitat

Sara Cambridge [email protected]@maybanks2015 UXPA

Page 2: Learning From Users in their Natural Habitat -

What is the best way to learn about the experience that people are having with your product?

Page 3: Learning From Users in their Natural Habitat -

3

Interviews don’t capture the range of ways people do things or the rich, in-the-moment details that people quickly forget.

Page 4: Learning From Users in their Natural Habitat -

Digital diary studies allow you to: • Get instant feedback as people naturally use a

product or service • Ask them follow-up questions throughout the study • Access everything through a digital platform

Page 5: Learning From Users in their Natural Habitat -

5

Are You Wearing One?

Page 6: Learning From Users in their Natural Habitat -

Fitness Tracker Case StudyWe recruited 40 people who use fitness trackers to participate in several digital diary studies over 8 months.

The first diary study had them report on 5 things they do with their tracker. With every entry, they submitted a short video and answered a few questions. Over 170 entries were collected.

We did 4 other diary studies with them over the course of 8 months to better understand how their fitness tracker usage evolved and the role it plays in their overall health goals.

6

Page 7: Learning From Users in their Natural Habitat -

7

Erica, 33

“It’s 10:53pm, and I’m only at 7388 steps….we’re about to go take the dog for a walk. Got to make sure we get to our 10,000 steps before the end of the night.”

Page 8: Learning From Users in their Natural Habitat -

What This Presentation Will CoverWhat a Diary Study is

The Benefits of Using Diary Studies

How Diary Studies Compare to Other Research Methods

Traditional Diary Studies vs. Digital Diary Studies

What the Participant Experience Is Like

How to Create a Digital Diary Study

Findings From Our Fitness Tracker Diary Studies

8

Page 9: Learning From Users in their Natural Habitat -

What is a Diary Study?

Page 10: Learning From Users in their Natural Habitat -

A Diary Study is a Cultural ProbeCultural probes use a collection of tools, artifacts and tasks intended to provoke the user to think about their environment in new ways….With minimal intrusion, researchers can glean insights into participants’ environments that can help to identify problem statements and uncover new opportunities.

10

Adam Starkman & Jennifer Chow http://designresearchtechniques.com/casestudies/cultural-probes/

Page 11: Learning From Users in their Natural Habitat -

Diary Studies Have 4 Key ElementsTheme-based: the subject and questions are suggested by the researcher.

Self-reported: the actual reporting is done by the participant, using whatever platform/prompts were provided.

In-the-moment: the participant is asked to do the reporting immediately after engaging in the activity, in order to capture as much detail as possible.

Longitudinal: the goal is to capture a range of uses, rather than a singular experience. This can include how the user’s experience evolves over time.

11

Page 12: Learning From Users in their Natural Habitat -

Traditional Diary StudyMozilla used paper diary studies in 2012 to learn how participants were using bookmarks in Firefox.

They asked 10 people to record their experiences with bookmarks over 3 days. Each day had a different set of questions.

12

Brian Groudan, Mozilla research, 2012 / https://blog.mozilla.org/ux/page/24/

Page 13: Learning From Users in their Natural Habitat -

Traditional Diary StudyPeople came in for an in-person interview afterwards, where they discussed their entries and sorted them by theme.

Guiding design principles were culled from the analysis that were the basis for design better bookmarking tools.

13

Page 14: Learning From Users in their Natural Habitat -

Why Use Diary Studies?

Page 15: Learning From Users in their Natural Habitat -

Looking for the DetailsThe goal of research is to understand not just what people do (and want and believe), but why.

This is complex and requires a deep understanding of people’s behavior.

Knowing details about their behavior is necessary to get to that understanding.

15

Page 16: Learning From Users in their Natural Habitat -

However people often have a hard time remembering those details when not engaged in the activity.

This is especially true for habitual behaviors.

Habits are formed when our brains:

Recognize that we repeatedly do the same sequence of actions (like checking email) after being triggered by a specific cue (such as picking up our phones).

The brain “chunks” that sequence into one automatic routine.

This allows our brain to save mental energy for more complex tasks.

Barriers to Collecting Details

16The Power of Habit, Duhigg

Page 17: Learning From Users in their Natural Habitat -

A consequence of having a sequence of behaviors “chunked” together is that details of the habit become accessible only after the cue has been activated.

So outside the actual ‘doing’ of the task we have difficulty remembering the details.

This is why gathering information in context collects such rich details.

Barriers to Collecting Details

17The Power of Habit, Duhigg

Page 18: Learning From Users in their Natural Habitat -

18

The best way to collect these rich details are either observation (contextual interviews) or in-the-moment reporting (diary studies).

Page 19: Learning From Users in their Natural Habitat -

How Diary Studies Compare to Other Methods

Page 20: Learning From Users in their Natural Habitat -

20

Quantitative methods only explains what people do while qualitative methods offer insight into why people do them.

Focus Groups

Diary Studies

Usability Testing

Surveys

Big Data

Contextual Interviews

why people do it

what people do

Quantitative (conscious & logical)

Qualitative(subconscious &

emotional)

Page 21: Learning From Users in their Natural Habitat -

21

Once you understand why, your insights can move from evaluative (improvement) to generative (new opportunities).

Focus Groups

Diary Studies

Usability Testing

Surveys

Big Data

Contextual Interviews

evaluative

Data

Stories

generative

Page 22: Learning From Users in their Natural Habitat -

22

Yesterday’s session about Storytelling by Mary Wharmby discussed this.

Page 23: Learning From Users in their Natural Habitat -

23

Karen, 26,grad student

“I got drunk with my friend and convinced her to download Snapchat on the contingency that I send her a [Snapchat] everyday. I use my [Jawbone Up] to remind me at 7pm to send her a picture….it vibrates on my wrist.”

Page 24: Learning From Users in their Natural Habitat -

How to Use Diary Studies with Other Research Methods

Page 25: Learning From Users in their Natural Habitat -

Insight Study ProcessAn Insight Study is a series of 2-3 focused research cycles built around a specific question. Multiple research methods are used in order to capture different types of insights.

Each cycle concludes with an Insight Review that shares the findings with the client team.

The benefit for the client is that they:

• Deliver real-time findings that can be acted upon right away

• Offer an opportunity to refine the next phase of research based on the learnings.

25

Page 26: Learning From Users in their Natural Habitat -

26

Insight Study Methods: Contextual Interviews allow us the deepest insights by observing people’s usage firsthand.

Diary Studies collect in-the-moment behaviors around a variety of issues throughout the project.

Remote Interviews fill in the gaps by probing deeper into the diary study entries and related areas.

Page 27: Learning From Users in their Natural Habitat -

The Benefits of Using Mixed MethodsUsing mixed methods allow us to validate our findings and take advantage of the unique strengths that the various research methods offer.

• Diary studies let us collect actual behaviors around a range of issues or track ongoing usage patterns.

• Contextual interviews give us the deepest insights by letting us observe their natural environment directly.

• Remote interviews give us access to people who are geographically distributed and allow more frequent opportunities to engage with them.

27

Page 28: Learning From Users in their Natural Habitat -

What is a Digital Diary Study?

Page 29: Learning From Users in their Natural Habitat -

29

Getting people to actually do the reporting is a major obstacle for traditional diary studies.

http://kotaku.com/hong-kong-residents-supposedly-cant-live-without-their-476758731

Page 30: Learning From Users in their Natural Habitat -

30

Smartphones are ideal for digital diary studies since people use them in short spurts throughout the day.

http://kotaku.com/hong-kong-residents-supposedly-cant-live-without-their-476758731

Page 31: Learning From Users in their Natural Habitat -

31

Digital diary studies can be as simple as text or email prompts that people reply to.

Page 32: Learning From Users in their Natural Habitat -

Digital diary study platforms

32

Name Platform Cost

24Tru Web/app diary study high

CrowdTap Web/app diary study high

UX 360 Web diary study high

Dscout App diary study medium

Dedoose Web diary study low

Tumblr private blog low

Typeform online survey low

Page 33: Learning From Users in their Natural Habitat -

33

$595 for recruiting (or you can bring in your own participants)

$295 for each study

Page 34: Learning From Users in their Natural Habitat -

34

Dscout has great guides and online support to help get you started.

Page 35: Learning From Users in their Natural Habitat -

The Participant Experience

Page 36: Learning From Users in their Natural Habitat -

36

This view is only if recruiting through dscout.

Participant reviews opportunities on app

Apply for a study Fill out screener

Page 37: Learning From Users in their Natural Habitat -

37

Once invited to a study, this is what all participants see.

Email invite to mission Q1: Begin with media prompt (photo, video or text)

Inside the app

Page 38: Learning From Users in their Natural Habitat -

38

Studies can have up to 10 questions each, and can be any mix of multiple choice, rating, or open-ended questions.

Page 39: Learning From Users in their Natural Habitat -

How to Conduct a Digital Diary Study

Page 40: Learning From Users in their Natural Habitat -

The 3 Phases of a Diary Studies

40

The goal: a concise, focused study with the right participants.

The goal: high participation and high quality data.

The goal: to turn data into relevant insights.

Page 41: Learning From Users in their Natural Habitat -

• Clarify your research question

• Refine and pretest your study

• Minimize the work for your participants

• Recruit the right participants

Page 42: Learning From Users in their Natural Habitat -
Page 43: Learning From Users in their Natural Habitat -

Clarify Your Research Question Our Initial Questions:

How are activity trackers used by people attempting to make concrete improvements to their fitness habits?

Can we discern patterns around when activity trackers are more or less effective at helping people make those changes?

43

Page 44: Learning From Users in their Natural Habitat -

44

Make a list of possible questions. You will have too many. Edit and sort them by theme.

Page 45: Learning From Users in their Natural Habitat -

45

Define the goals for each study. This will help maintain focus as you refine the questions.

Page 46: Learning From Users in their Natural Habitat -

46

Diary Studies are basically surveys with a twist.

Page 47: Learning From Users in their Natural Habitat -

47

That means answer options, which require a lot of refining. The more time spent building the study, the better the data will be.

Page 48: Learning From Users in their Natural Habitat -

Make it Easy for your Participant to SucceedA guiding principle as you build your study should be to minimize the work your participant has to do.

48

Page 49: Learning From Users in their Natural Habitat -

49

Consider how often you ask them to report. It should be enough to collect useful data, but not enough to cause fatigue.

Page 50: Learning From Users in their Natural Habitat -

50

Pairing themed multiple-choice questions with open-ended ones creates a logical flow for people and answers both “what” and “why”.

Page 51: Learning From Users in their Natural Habitat -

51

Another benefit of multiple-choice is you can quickly see trends as the data comes in.

Page 52: Learning From Users in their Natural Habitat -

52

Open-ended questions provide the richest data but are more work for both for you and the participant. Use them sparingly.

I think it should send reminders on when to be more active when I'm lagging in my goal such as vibrating.

I do wish that other activities could be tracked with the Fitbit without having to input it myself.

Maybe if I could get some sort of notification on my phone if a specific friend or group of friends had passed me.

Page 53: Learning From Users in their Natural Habitat -

53

Pretest by sitting down with someone as they go through every step. Look for awkward or unclear phrasing.

Page 54: Learning From Users in their Natural Habitat -

54

Media options: video adds a lot, but also adds more work for both you and the participants.

Take a picture that symbolizes the role your tracker plays in your health/fitness goals

Tell us what ideal health would look like for you.

Page 55: Learning From Users in their Natural Habitat -

55

Some questions are better suited for a photo, which makes it easier to compare (and share) responses.

A long road ahead. Before last year I never thought I could be THAT mom who was fit & healthy.

My tracker is a set of tools I'm using to build a healthier life for myself.

Page 56: Learning From Users in their Natural Habitat -

56

When capturing real experiences or complex ideas, video is better.

Karen, 26,grad student

“Being able to walk upstairs without being out of breath. Also, be at the pick of my thinking and not feeling tired and drained because of ill health. And I think that my tracker helps me with that especially with sleep.”

Page 57: Learning From Users in their Natural Habitat -
Page 58: Learning From Users in their Natural Habitat -

58

Diary studies generate a lot of data. It is good to start small.

10 x 10 x 10 =questions participants entries

data points

1000

Page 59: Learning From Users in their Natural Habitat -

59

Recruiting options include customers, friends, Craigslist, social media and dscout.

Page 60: Learning From Users in their Natural Habitat -

60

If recruiting through dscout, you’ll only pay incentives for participants that finish. dscout recommends paying $5-10 per entry.

Page 61: Learning From Users in their Natural Habitat -

61

We looked for participants who have a fitness tracker and were regularly using it as part of a fitness or weight loss goal.

Page 62: Learning From Users in their Natural Habitat -

62

Typical dropout rate is 15-25%, so over-recruit.

40

34

Page 63: Learning From Users in their Natural Habitat -

63

You can view each applicant’s responses inside the screener survey and rate them based on how good they are.

Page 64: Learning From Users in their Natural Habitat -

64

It’s easy to add participants if you do your own recruiting. They get an email link for downloading the app.

Page 65: Learning From Users in their Natural Habitat -

• Review & make notes from your data as it’s coming in

• Be persistent with participants

• Ask them questions for clarity

• Do post-study interviews

Page 66: Learning From Users in their Natural Habitat -
Page 67: Learning From Users in their Natural Habitat -

67

Watch the data as it comes in. This is the fun part.

What is your favorite thing about your fitness tracker?

Kayla, 27,teacher

“My favorite thing is that by a single click of a button, the display shows me the time, date, exact steps I've taken, my heart rate, steps climbed and calories burned as opposed to having to find my phone, unlock, open the app...it's all right there, easy to read. I like having it all right there on the display. .

Page 68: Learning From Users in their Natural Habitat -

68

Entries are reviewed individually.

Page 69: Learning From Users in their Natural Habitat -

69

Take notes of interesting patterns and questions to ask participants as you go through it.

Page 70: Learning From Users in their Natural Habitat -

70

Moment of truth: are your questions holding up? If not, can you fix it? Either way, learn for next time.

Page 71: Learning From Users in their Natural Habitat -
Page 72: Learning From Users in their Natural Habitat -

72

Stay in touch with your participants. Send them reminders if they haven’t submitted an entry. Be friendly and persistant.

Page 73: Learning From Users in their Natural Habitat -

73

Ask them questions if you want more clarity about their answers. Be appreciative and interested in what they have to say.

Page 74: Learning From Users in their Natural Habitat -

74

Do follow-up interviews with the most engaged participants. You already have rapport and can dig deeper into their actual behaviors.

Page 75: Learning From Users in their Natural Habitat -

• Start by answering your initial questions

• Use the built-in analysis tools

• Identify patterns and/or exceptions

• Share your findings with real examples

Page 76: Learning From Users in their Natural Habitat -

76

The good news is that by simply reviewing and making notes on the data, you have already started on the analysis.

Page 77: Learning From Users in their Natural Habitat -

What Further Analysis Should be Done?That depends on many variables, such as:

• What your initial question/s were

• How surprised you are by the data that has come in

• How it fits into other research methods

• Your final deliverables

77

Page 78: Learning From Users in their Natural Habitat -

78

The internal tools dscout offers are helpful. Tagging can quickly reveal patterns.

Page 79: Learning From Users in their Natural Habitat -

79

Multiple choice questions can be filtered. Here we wanted to review the entries in which people were not as satisfied with their tracker.

Page 80: Learning From Users in their Natural Habitat -

For Those Who Rated an Experience with Their Tracker as “OK” or Lower: • 30% were unhappy with some aspect of the band

(too bulky, not water resistant or something had broken)

• 25% were complaining about the charger or battery

• 15% had issues logging their food (either through a secondary app or via their main device)

• The remaining 30% were a random mix of issues

80

Once the responses were filtered, it was quick to manually categorize what remained.

Page 81: Learning From Users in their Natural Habitat -

81

Transcribing video is time-consuming, but makes the data much more accessible. Tagging is an ideal way to remember interesting video.

Page 82: Learning From Users in their Natural Habitat -

82

Downloading the data into a spreadsheet that can be coded, refined, sorted and easily reviewed is our preferred method if we intend to dive deep.

Page 83: Learning From Users in their Natural Habitat -

83

If the original answer options were not accurate enough, you’ll have to throw data away or recode it yourself; here we recoded it.

Page 84: Learning From Users in their Natural Habitat -

Fitness Tracker High-Level Findings

Page 85: Learning From Users in their Natural Habitat -

85

Why did you get your first fitness tracker?

Larry, 31

“I got a fitness tracker because was about 300 lbs, on a diet, exercising, wasn't losing weight. I knew there was something missing. I wanted to learn how many calories I was burning so could know how little or much I should be eating. That was the missing link. I have lost 150 lbs, and the only way did it was because I started using Jawbone Up.”

Page 86: Learning From Users in their Natural Habitat -

People Love Trackers Because of What They Learn About ThemselvesFitness trackers teach people things about their eating and exercise patterns they’ve never known before.

They had made guesses, but they had guessed wrong.

People usually overestimated the exercise they were getting and underestimated how much they were eating.

Knowing the actual numbers gives people control so they can get to the outcome they want.

86

Page 87: Learning From Users in their Natural Habitat -

87

What do you like least about your tracker?

Amy, 38

“Now for least favorite. Even though I loved a lot about all the trackers: Omron and Fitbit, the one thing I didn't like was that after a while I got bored. There's only so much competing you can do, so many steps you can take, but they didn't do anything with the data. It was great when I reached my 10,000 steps for the day, but then what? It doesn't help me do anything more with it.”

Page 88: Learning From Users in their Natural Habitat -

Once People Learn their Patterns, they get Bored (and/or Frustrated)Depending on how motivated people are to make changes to their health behavior, they usually go through a slump after they learn their patterns.

Changing exercise and eating pattens require significant changes to one’s lifestyle, and knowing one’s patterns is only the first step.

People want more help getting to their goals.

88

Page 89: Learning From Users in their Natural Habitat -

89

What does ideal health look like to you?

Mary, 42

“My ideal health would be to feel better, not drag, and eating better foods. It would help me feel motivated to follow my exercises I need a tracker that would also [suggest what] types of foods to eat. It knows that I am running and this is the food you should eat. That would be so wonderful! That would help me a lot even more with my runs, my exercise, with getting in shape.

Page 90: Learning From Users in their Natural Habitat -

Trackers Only Address a Fraction of People’s Health GoalsWhen asked what their ideal health looks like, people talked about both complex ideas (such as mood, energy level, endurance and eating habits), and hard numbers (like weight and body fat).

Trackers only help with some of the aspects of ideal health.

People want their trackers to know their goals, learn their patterns and offer suggestions to help them achieve their goals.

90

Page 91: Learning From Users in their Natural Habitat -

Digital Diary Study Key Takeaways

Page 92: Learning From Users in their Natural Habitat -

Main Benefits of Using Diary Studies• To collect rich stories about how and why people do things

• Get intimate glimpses of how a product fits into people’s lives

• Allows for tracking ongoing use of a product

• Cost-effective way to collect contextual insights

92

Page 93: Learning From Users in their Natural Habitat -

Tips for Building Your Own Diary StudyBefore you can use it, you may have to sell it. Here are a few things to keep in mind:

• It’s an extremely flexible tool; make it work for your needs

• Start small, let it prove it’s value to your organization and yourself

• Evangelize the insights and stories you collect to build more support.

• You become an expert by doing. So dive in!

93

Page 94: Learning From Users in their Natural Habitat -

What Will You Do with Diary Studies?

Page 95: Learning From Users in their Natural Habitat -

For a 4-min video about diary studies: www.gotoresearch.com/speaking/sara/

Questions about diary studies? Ask!

[email protected]

@ maybanks

Page 96: Learning From Users in their Natural Habitat -

Thank you!

[email protected]

@ maybanks