hypothesis testing workshop (cezary.co)
TRANSCRIPT
HYPOTHESIS TESTING WORKSHOP
CEZARY PIETRZAK
Intel EdTech Accelerator Redwood City, California
August 4, 2015
@ckp
MY BACKGROUND
2000 - 2004
2005 - 2008
CEZARY & CO
2008 - 2012
2012
2013
2014
AGENDA WORKSHOP (1:30PM - 4:30PM)
Developing hypotheses (25min) Exercise: Create your own (60min)
Break (10min)
Testing hypotheses (15min)Exercise: Deploy a test (70min)
HYPOTHESIS STATEMENTS ARE DECLARATIVE
Definition States a fact or an argument
Ends with a full stop
Example I believe that…(X is true)
THEY BEGIN WITH THE CUSTOMER-PROBLEM
Customer-Problem Customer X has Problem X
(that is worth solving)
I believe that… High school students struggle with
basic data literacy
*Always phrase the problem from the customer’s perspective *Add emotional language whenever possible
AND THEN ADDRESS THE PROBLEM-SOLUTION
Problem-Solution Problem X can be solved
by Solution X
I believe that… Bringing real-life data into the classroom
will make math and science more engaging
*Only relevant after you’ve validated that a problem exists *Solution should be unique to alternatives
STRONG HYPOTHESES ARE DERIVED FROM DATA & INSIGHTS
Customer-Problem High school students struggle with
basic data literacy
Rationale 25M high school students are
falling behind in quantitative subjects
Problem-Solution Bringing real-life data into the classroom
will make math and science more engaging
Rationale The explosion of data today makes it
more accessible than ever before
AND ARE ALWAYS BASED ON ASSUMPTIONS
Customer-Problem High school students struggle with
basic data literacy
Assumption Parents and school districts consider
math and science a high priority
Problem-Solution Bringing real-life data into the classroom
will make math and science more engaging
Assumption Most students are genuinely interested
in math and science
*Focus on the riskiest assumption
MANY EARLY-STAGE STARTUPS TEST CORE HYPOTHESES WITH THE EXPERIMENT BOARD
BEFORE MOVING TO MORE GRANULAR TESTING
Customer Overarching persona
Specific segment Stage in lifecycle
Acquisition source Geography/location
Users vs. influencers vs. decision-makers
Problem / Challenge Master problem
Sub-problem Distribution/marketing channels
Buying process/sales funnel Willingness to pay/pricing
Messaging Loyalty/retention
Evangelism/virality
Solution Existing product/service
New product/service Specific feature/flow
Marketing tactic
*Keep one variable constant
YOU CAN SOURCE HYPOTHESES FROM MANY PLACES
Behavioral insights Background research Industry benchmarks
Company stage Growth goals
Investor requirements Previous hypotheses
Highest risk Biggest potential impact
Start high-level, then get specific
Questions to ask Is the hypothesis core to your value proposition?
Will validating the hypothesis significantly de-risk your business? For each hypothesis, have you validated the higher-level one above it?
BUT MUST PRIORITIZE THEM TO CREATE VALUE
LAUNCH EXPERIMENTS IF THEY MEET 3 CRITERIA
Speed “one-afternoon” rule simple over elaborate
Costcheap tools sweat equity shared resources
Accuracy right audience right approach
SET GOALS AND SUCCESS METRICS
Goals should be quantitative when possible Qualitative results can be measured
Examples A minimum of X respondents will agree
The majority of respondents will exhibit positive language User will convert at a 5% significance level
Overall retention will rise by X% 1 in 5 people will recommend a friend
FOLLOW THESE BEST PRACTICES
Baseline Better
Keep it simple Customers understand the detailed technology behind my product
Customers understand why my product is faster than competition
Break big hypotheses into smaller batches I can acquire users via paid spend
I can acquire users via social ads I can acquire users via paid content I can acquire users via trade shows
Add details where possible
I can acquire new users from Facebook efficiently
I can acquire new users from Facebook mobile ads at $1/click
Test one thing at a time (avoid: and/either/or)
Customers prefer product X over product Y or Z
Customers prefer product X over Y Customers prefer product X over Z
BREAKOUTS(60 minutes)
Define core customer-problem & problem-solution Compile & prioritize list of key hypotheses
Share work with group
bitly.com/hypothtesting
PART 2(15 minutes)
Testing hypotheses
START WITH QUALITATIVE TESTING TO GATHER FEEDBACK
In-person conversations / live demos Focus groups
Video conversations User testing / video recording
Public forums / blogs Surveys ($ or no $)
FOCUS THE CONVERSATION FOR BETTER RESULTS
Don’ts Do’sDon’t rely on friends and family Do speak with outsiders who lack a filterDon’t tell respondents you’re working on an idea
Do tell respondents that you’re seeking an opinion
Don’t ask leading questions Isn’t it terrible when…?
Do let them describe their environment and motivations
Don’t provide options unless necessaryIs your preference A, B or C? Do let people decide on the answer
Don’t ask hypotheticalsWhat would you do if…?
Do ask about past behaviorHow have you solved this problem in the past?
Don’t ask about intentWould you tells others about this product?
Do test their willingness to take action Can you share this product with a friend?
USE QUANTITATIVE TESTING TO CAPTURE KEY DATA AND IDENTIFY ACTION
Product analytics Heat maps
Social media ads Paid search
Landing pages Placeholders / waitlists
A/B testing
ENSURE THAT YOUR DATA IS RELIABLE
Questions to ask Who is your audience?
How did they come to your app? Are the results real or sampled?
Is the sample representative of your audience? How is significance determined?
EVALUATE RESULTS
Significance of sample Minimum 10 responses for qual, 30 for quant
Tangible outcome Signed letter of intent, verbal agreement
Strong body language Overall enthusiasm, leaning in
Correct metric hierarchy Later funnel metrics trumps early funnel
If data is ambivalent, revisit hypothesis
BREAKOUTS(70 minutes)
Define your experiment Choose tools
Start implementing
LEARN ABOUT MARKETING + GROWTH + MOBILE AT
CEZARY.CO@ckp