optimization summer games - test ideation
TRANSCRIPT
Generating High-Impact ExperimentsHazjier Pourkhalkhali
Senior Strategy Consultant
@Hazjier
Optimizely is the world’s largest digital laboratory
We researched the world’s top companies
Interviewed experts
§ Spoke with thought leaders in the industry
§ Met with top customers
Built maturity model
§ Developed comprehensive maturity model
§ Assessed 100 customers
§ Surveyed 50+ customers
Held 100+ Business Reviews
§ Met with executives at over 100 Optimizely clients
Analyzed Top Revenue Experiments
§ Analyzed the 100 best experiments run on Revenue Per Visitor
§ Asked customers to describe test and process
User acquisition:
1.5m
User retention
33%
Add-to-Cart:4.00%
Cart Checkout:
50.0%
Average quantity:
2.00
Average per unit price:€25.00
Average Order Value:€50.00
Conversion Rate:2.00%
Revenue Per Visitor:€1.00
Visitors:2.0m
Revenue:€2.0m
Understand the revenue drivers
Goals
Strategies
Tactics
Cart Checkout
Rate
Wording / messaging
Location / size
Design / color /
iconography
Establish a visual
hierarchy
Remove non-
critical content
Limit number of
choices
Select/target the right USP’s
Refine the messaging
Crystallize visually
Clear and consistent
UI
Structure, naming,
and order of sections
Describe visually
Test ideal product visuals
Use the right
review systems
DescriptionsReviewsImages
Emphasize the Primary
Call to ActionMinimize
DistractionsCommunicate
Unique Selling Points
Design Intuitive
Navigation
Market Products
EffectivelyBuild a Sense
of Urgency
Show quantity
left
Push temporary
offers
Promote buying
before set times
Testing requires freedom
LOW FREEDOM HIGH FREEDOM
“Every change we make to ourwebsite has to be approvedby our senior staff”
-Financial Services company
Testing requires freedom
LOW FREEDOM HIGH FREEDOM
“Our Brand Styleguide is key.We cannot make changes that do not fit our design book”
-Financial Services company
Testing requires freedom
LOW FREEDOM HIGH FREEDOM
“We would change our company logo if itconverted better”
-Travel Company
NO BEARD CAVEMAN
.30% .29% .28% .32% .25% .74%
.11% .28% .24% .32% .09% .59%
.74%
.59%
Minimize Distractions:Remove Non-Critical Content +14%
Revenue PerVisitor
BASED ON KEYWORDCURATED
+24%Click to Sale
+30%Add-to-Cart
+210%Conversions
§ The user is reminded repeatedly of price
§ The user must remember the product benefits
§ The user must scroll down to the CTA when their basket is 3 or more
Problem Solution Result
§ Reduce size of price
§ Highlight discounts
§ Add product visuals
§ Duplicate CTA at top of page
§ Increase Conversion Rate
§ Increase Conversion Rate
§ Increase CTR to checkout
Build powerful hypotheses
§ Analyze one section of your application / website
§ Document 10 different problems for your users
§ Group them by common root causes
§ Prioritize by urgency
Problem Solution Result
Optimizely Hypothesis Workshop
§ Analyze one section of your application / website
§ Document 10 different problems for your users
§ Group them by common root causes
§ Prioritize by urgency
Problem Solution Result
Optimizely Hypothesis Workshop
§ Select 1 concrete problem
§ Design 10 solutions per theme
§ Choose 4 preferred solutions
The problem with “A/B Testing”
2
A B
4
A B C D
Most people test only 2 variations
0
10,000
20,000
30,000
40,000
50,000
60,000
2 3 4 5 6 7 >8
Tests 65% 17% 17%
# of Variations
Most people test only 2 variations
0
10,000
20,000
30,000
40,000
50,000
60,000
2 3 4 5 6 7 >8
Tests
# of Variations
Test 5 variations and nearly double your win rate!Win Rate
# of Variations
14% 18% 20% 27% 20% 22% 23%0%
10%20%30%40%50%60%70%80%90%
100%
2 3 4 5 6 7 >8
+90%
Running more variations will significantly improve your testing!
§ Analyze one section of your application / website
§ Document 10 different problems for your users
§ Group them by common root causes
§ Prioritize by urgency
Problem Solution Result
Optimizely Hypothesis Workshop
§ Select 1 concrete problem
§ Design 10 solutions per theme
§ Choose 4 preferred solutions
§ Primary Goals:What do you seek to impact immediately
§ Secondary Goals:What would you like to impact, but do not have to
§ Monitoring Goals:What are metrics you want to track
Optimizely funnel analysis
HomepageFlight
ResultsBookFlight
PassengerDetails
FlightOptions
Payment Page
Step
45%30%
65% 65%
90%
70%
40%
40%
15% 20%
5%
25%
15%30%
20% 15%5% 5%
Exit Rate
Divergence
Completion
Optimizely funnel analysis
HomepageFlight
ResultsBookFlight
PassengerDetails
FlightOptions
Payment Page
Step
Primary /SecondaryGoals
§ Searches Submitted
§ Category tabs§ Deals§ Frequent Flyer
§ Flight Selection§ Filter Usage§ Refine Search§ Sort Order
§ Continue button
§ Edit search§ Upsells
§ Field completion rate
§ Continue button
§ Continue button
§ Upsells§ Seat
selection
§ Conversion rate
§ Form field completion
§ Payment Type
MonitoringGoals
§ Checkboxes§ Flight tracker
§ Home page clicks
§ Menu interactions
§ Site search
§ Home page clicks
§ Menu interactions
§ Site search
§ Field error rate
§ Home page clicks
§ Menu interactions
§ Site search
§ Shopping cart pageviews
§ Home page clicks
§ Return to previous page
§ Edit address, information
§ Error rate
§ Analyze one section of your application / website
§ Document 10 different problems for your users
§ Group them by common root causes
§ Prioritize by urgency
Problem Solution Result
Optimizely Hypothesis Workshop
§ Select 1 concrete problem
§ Design 10 solutions per theme
§ Choose 4 preferred solutions
§ Primary Goals:What do you seek to impact immediately
§ Secondary Goals:What would you like to impact, but do not have to
§ Monitoring Goals:What are metrics you want to track
