designing to learn: creating effective mvp experiments - lean ux nyc 2014

Designing to LearnCreating Effective MVP Experiments

Melissa Perri @lissijean

Consultant Lean Product Manager UX Designer Teacher Twitter addict (@lissijean) Blogger (melissaperri.com)

Hi! I’m Melissa.

@lissijean

• What is an MVP (and why should we care)?

• Setting Up MVP Experiments

• Types of MVPs and How to Choose

• Iterating on MVPs

• Agile + MVP: How does it work together?

Agenda

@lissijean

Why MVP?

@lissijean

“Match your taste with the exceptional finds of industry insiders.”

@lissijean

@lissijean

My way is faster.

This isn’t what I needed.

I don’t know how to use it.

The information I want to see isn’t on here.

@lissijean

@lissijean

@lissijean

@lissijean

Validate Before Building

@lissijean

Build

Measure

Learn

@lissijean

Iterate your way to Product/Market Fit

your users

@lissijean

MVPMinimum Viable Product

@lissijean

MVP Smallest amount of work you can do

to test your idea.

@lissijean

@lissijean

@lissijean

@lissijean

@lissijean

LEARN

@lissijean

MVP = Minimum Feature Set

Agile Methodology MVPs

@lissijean

Focus on problem not features.

@lissijean

MYTH

@lissijean

Crappy products

Crappy products everywhere

@lissijean

Good MVPs offer a great experience to the customer.

@lissijean

@lissijean

MVP =

Experiment

@lissijean

Experiment Process Set Up !

Design !

Run !

Feedback !

Iterate

@lissijean

@lissijean

Setting up MVP Experiments

PART I

@lissijean

1. Define Customer & Problem 2. Investigate assumptions 3. Design a test 4. Measure customer behavior 5. Evaluate success 6. Pivot or persevere

Process

@lissijean

@lissijean

DefineDefine

@lissijean

Customers want twitter-like messages in their feed.

@lissijean

Customers want twitter-like messages in their feed.

Not a problem.

@lissijean

Real ProblemCustomers don’t think the

curators are authentic.

@lissijean

Investigate

@lissijean

Assumptions

Shoppers care that curators are authentic.

Shoppers will engage more with real messages on site.

Shoppers think tweets are authentic.

@lissijean

Riskiest Assumption

Shoppers care that curators are authentic.

Shoppers will engage more with curator messages on site.

Shoppers think tweets are authentic.

@lissijean

Design a Test

@lissijean

Setting up a test1. What do we want to learn? 2. How do we measure that? 3. What’s the least we can build to learn?

@lissijean

Will shoppers engage more with curator messages?

What do we want to learn?

@lissijean

Engagement Clicks, purchases, added to cart

How do we measure that?

@lissijean

What is the least we can build to learn?

Wizard of OzLooks real to the customer,

completely manual on the backend.

@lissijean

@lissijean

@lissijean

Measure

@lissijean

How do we measure engagement?

@lissijean

Clicks on product.

Purchases. Add to cart.

@lissijean

What do we consider successful?

@lissijean

Minimum Success CriteriaWhat is the minimum number we need to

hit to consider this validated?

@lissijean

Time Salary# Peoplex x

Cost to build completely

@lissijean

Minimum Success Criteria

Improvement needed to justify cost to build.

@lissijean

25% increase in engagement in A/B test.

@lissijean

@lissijean

Evaluate

@lissijean

@lissijean

0% change over other products.

@lissijean

INVALIDATED

@lissijean

But why?

@lissijean

Why didn’t users engage?

What do we want to learn?

@lissijean

User Testing Sessions watch them use it.

How do we find that out?

@lissijean

Emails.

@lissijean

Pivot

@lissijean

strategic change in direction based on learning.

Pivot

@lissijean

Riskiest Assumption

Shoppers will engage more with curator messages.

@lissijean

Will shoppers engage more with curator messages

through email?

What do we want to learn?

@lissijean

Engagement Opens, Click throughs, purchases.

How do we measure that?

@lissijean

What is the least we can build to learn?

@lissijean

How do we determine success?

30% increase in engagement over other emails.

@lissijean

Success!2x Open Rate 3x Purchases

@lissijean

Now, we automate. Now, we invest.

@lissijean

workshop one:

Setting up MVP Experiments

@lissijean

@lissijean

We’ve recently received a flurry of complaints from Macy’s shoppers.

@lissijean

“Every time I come into Macy’s, I spend HOURS I don’t have and can never seem to find what I’m looking for. I know exactly what type of item I want, but options are all over the place. It’s daunting.”

@lissijean

“I needed a blue tie to go with my suit for a friend’s wedding, so I went to find it but there were too many options. Where do I even start? I don’t even know how to choose.”

@lissijean

Who is the customer?

@lissijean

“OpenSky shoppers who visit the site at least once a month, but rarely buy.”

@lissijean

What is their problem?

@lissijean

“Do not think the curators are authentic.”

@lissijean

10 minutes

In your teams, write down your customer and their problem for Macy’s.

@lissijean

Shoppers who are purchasing something specific out of necessity, not browsing.

Customer Hypothesis

@lissijean

Cannot find what they need efficiently.

Problem Hypothesis

@lissijean

What are we assuming about the customer and the problem?

@lissijean

Riskiest AssumptionWhat are we the most uncertain about?

@lissijean

10 minutes

In your teams, write down your assumptions and pick the riskiest one.

@lissijean

Assumptions

Shoppers are overwhelmed by the amount of options.

Shoppers do not want to browse.

Shoppers want help finding items.

@lissijean

Assumptions

Shoppers are overwhelmed by the amount of options.

Shoppers do not want to browse.

Shoppers want help finding items.

@lissijean

Now, we design a test for that riskiest assumption.

@lissijean

Types of MVPs

PART II

@lissijean

WHAT SHOULD YOU ALWAYS DO FIRST?

@lissijean

InterviewsGet out of the building and talk to people.

@lissijean

• First thing you should do. • Talk to at least 10 people. • Verify the problem and the customer hypothesis. • Verify as many assumptions as you can.

When to do Interviews

@lissijean

ConciergeDelivering a service manually.

Delivering a service manually.Concierge

@lissijean

@lissijean

@lissijean

The Good The Bad• Very close to the

customer, easy to learn. • Can adjust easily based on

feedback. • Very easy and cheap to

get started.

• Not scalable. • Time consuming to run.

Concierge

@lissijean

• If you are testing a service. • Trying to determine what customers will respond to. • Testing and refining with a small number of people.

When to use Concierge.

@lissijean

Wizard of OzLooks real, but manual on the back end.

@lissijean

@lissijean

Subscription

@lissijean

@lissijean

@lissijean

The Good The Bad

• Looks exactly real to customers, so very accurate response to if you would build it.

• Need to code to get started. • Can’t tell you why customers

don’t use it.

Wizard of Oz

@lissijean

• Running a test with a large audience. • Trying to sell a feature or service before it is built,

but pretend it is already there. • If you are concerned about sacrificing brand.

When to use Wizard of Oz.

@lissijean

Landing PageA pitch of your product, to gauge idea reception.

@lissijean

@lissijean

@lissijean

The Good The Bad

• Just a pitch, don't need to refine the whole idea

• Can launch in a matter of minutes

• MOST PEOPLE FORGET THEY NEED TO DIRECT PEOPLE HERE THROUGH MARKETING.

Landing Page

@lissijean

• Looking for initial customers to test. • Trying to gauge how much interest there is

in the idea. • Great way to pitch to a large audience.

When to use Landing Page.

@lissijean

VideosDemo what your product can do through hacking

it together for a video.

@lissijean

@lissijean

The Good The Bad

• Can demonstrate complicated things.

• Show a prototype before producing them all.

• More complicated to put together.

• Needs to be a certain amount done before you can make the video.

Videos

@lissijean

• If your product does something very easily. • When your product sounds complicated. • When people need to see it to believe it.

When to use Videos.

@lissijean

What if I’m not solving a dire need?

@lissijean

workshop two:Design your test.

@lissijean

You have already interviewed your customer,

they have that problem.

@lissijean

Now how are you going to solve that problem?

@lissijean

Choose an MVP strategy and plan out a test.

30 minutes

• What you are trying to learn? • Explain every little detail of the test. • What are you measuring? • Minimum success criteria.

@lissijean

Resources

Access to 2 Macy’s stores Access to Macy’s online page !

3 Store Interns $30,000 salary each 10 Sales Associates In Store $40,000 salary each 5 Developers $100,000 salary each 1 Designer $100,000 salary Yourself $100,000 salary

@lissijean

Every time you start with a minimum feature set,

a kitten dies.

@lissijean

Responding to Constraints &

Feedback

PART III

@lissijean

The goal of an MVP is to learn.

@lissijean

Customer Feedback

@lissijean

@lissijean

@lissijean

Constraints

@lissijean

Restricts medium. Restricts resources.

@lissijean

workshop three:

Alter your test based on constraints.

@lissijean

You learned that people only have this problem online.

How do you make an online MVP?

@lissijean

You learned that people only have this problem in store.

How do you make an in person MVP?

@lissijean

How would you change your MVP experiment based on these new

constraints?

5 minutes

@lissijean

MVPs in Agile Teams

PART IV

@lissijean

Gather Requirements

Spec features

Design Interfaces

Development

QA Process

Release

Waterfall

@lissijean

Gather Requirements

Spec features

Design Interfaces

Development

QA Process

Release

Product Manager UX Designer Developer

@lissijean

Agile

@lissijean

Individuals and Interactions over

Processes and Tools

@lissijean

Agile SprintsShort, goal oriented sprints formed around a problem running Build, Measure, and Learn.

@lissijean

Build

Measure

Learn

@lissijean

Product Manager

UX Designer Developer

Small, cross-functional teams

@lissijean

@lissijean

monday tuesday wednesday thursday friday

Reflect & Define Specify Build & Refine Build & Refine Customer Feedback

Sprint Structure

@lissijean

Problem Exploration

MVP Experiments

Feature Development

@lissijean

MVPs are only a test.

@lissijean

“My solution is too obvious, if I just build it people will use it.”

@lissijean

Customers will only use your features if it fills a need.

Customers will only use your features if it fills a need.

@lissijean

“Lean is just for startups, my company is too grown up.”

@lissijean

Lean applies to all companies.

@lissijean

Melissa Perri !

Want an MVP Guide? melissa@produxlabs.com !

produxlabs.com melissaperri.com !

!

@lissijean