class 2: how to create solid hypotheses and set up powerful a/b tests
TRANSCRIPT
Class 2How to create solid hypotheses and set up powerful A/B tests
#OptimizelySchoolwww.summerschool2015.eu
Improve annual profit by being a reliable go to store for outdoor
Company Goal
Reduce shipping & handling costs from returns
Business Unit
Increase UX of the order process
Optimization Goals
Relation size calculator and UX
Experiment Goals
#OptimizelySchool
Success metrics differ per industry
Ecommerce/ Travel Media Lead Gen/ B2B
Product page views Pages per visit Key content page views
Add to cart rate Ad impressions Content engagement
Checkout completion rate Ad viewability AD unit CTR Funnel entrance rate
Average order value CPM CPC Entry field completion
Conversion rate Subscription rate Conversion rate
#OptimizelySchoolwww.summerschool2015.eu
There is not enough motivating imagery
Page is overly focused on price
Call to Action is too far down the page
Problem Solution Result
How to find testing ideas
Add product visuals
Balance price with discount
Duplicate Call to Action at top of page
Increased conversion rate
Increased conversion rate
Increased Pageviews on Checkout Page
The goal of the Hypothesis
Gathering insights on your users
A good hypothesis enables you to:1. Focus on the right goals & KPIs
2. Focus on the right elements3. Collect data to make data-driven decisions
#OptimizelySchool
If______ Then______ Due to______change result
(quantifiable)rationale
IF we add a size calculator to every product page
If______ Then______ Due to______change result
(quantifiable)rationale
IF we add a size calculator to every product page THEN the conversion rate will increase
If______ Then______ Due to______change result
(quantifiable)rationale
IF we add a size calculator to every product page THEN the conversion rate will increase DUE TO the increased trust of buying the right size product.
Hypothesis: removing elements will allow for more product copy and less clutter on the page.
Good hypothesis: if we divide the homepage in a 80-20 ratio between product related and non product related items
then we will see an increase in article page views,
due to the reduction of distracting elements on the page customers can better focus on article content.
Hypothesis:
If we show text suggestive of the range of products we offer
then we can increase the product page views and conversions.
Due to users arriving at their preferred products easier.
Hypothesis:
If we integrate weather information from Skymosity into the predictive text in the search bar
then we can increase the product page views and conversions.
Due to users arriving at their preferred products easier.
Hypothesis:
If we hide the coupon code field behind a link
then we can increase the conversions and reduce the drop off rate
due to users not leaving site to go find a coupon code
Hypothesis:
If we add a tab to search for vacation packages
then we can increase the average order value
due to the fact that users will engage with a feature they could not previously considered.
Hypothesis:
If we mention all shows on the homepage
then we will increase the overall engagement.
Due to the better visibility of all shows.
• is easy to find if you follow the Problem, Solution, Result pattern
• Consists of 3 fundamental building block: IF… THEN…DUE TO
• Shows a change in a quantifiable metric
• Gives actionable insights (no matter the outcome)
So a good hypothesis…