conversion optimization breakfast workshop

Lessons from the World’sGreatest Testers

May 2016

@Hazjier

Hazjier PourkhalkhaliLead Strategy Consultant

@Hazjier

Optimizely is the world’s largest digital laboratoryAlexa 10k websites, Q1 2016

We researched the world’s top companies

Interviewed experts

§ Spoke with thought leaders in the industry

§ Met with top customers

Built maturity model

§ Developed comprehensive maturity model

§ Assessed 100 customers

§ Surveyed 50+ customers

Held 100+ Business Reviews

§ Met with executives at over 100 Optimizely clients

Analyzed Top Revenue Experiments

§ Analyzed the 100 best experiments run on Revenue Per Visitor

§ Asked customers to describe test and process

1 Understand yourrevenue levers

3 Solve realproblems

2 Test all changes

4 Test multiplevariations

User acquisition:

1.5m

User retention

33%

Add-to-Cart:4.00%

Cart Checkout:

50.0%

Average quantity:

2.00

Average per unit price:

£25.00

Average Order Value:

£50.00

Conversion Rate:2.00%

Revenue Per Visitor:£1.00

Visitors:2.0m

Revenue:£2.0m

Understand the revenue drivers

Goals

Strategies

Tactics

Cart Checkout

Rate

Wording / messaging

Location / size

Design / color /

iconography

Establish a visual

hierarchy

Remove non-

critical content

Limit number of

choices

Select/target the right USP’s

Refine the messaging

Crystallize visually

Clear and consistent

UI

Structure, naming,

and order of sections

Describe visually

Test ideal product visuals

Use the right

review systems

DescriptionsReviewsImages

Emphasize the Primary

Call to ActionMinimize

DistractionsCommunicate

Unique Selling Points

Design Intuitive

Navigation

Market Products

EffectivelyBuild a Sense

of Urgency

Show quantity

left

Push temporary

offers

Promote buying

before set times

2 Test all changes

“Every change we make to ourwebsite has to be approvedby our senior staff”

-Financial Services company

Testing requires freedom

LOW FREEDOM HIGH FREEDOM



“Our Brand Styleguide is key.We cannot make changes that do not fit our design book”

-Luxury goods company



“We would even change our logo if it converted better”

-Travel Company

NO BEARD CAVEMAN

.30% .29% .28% .32% .25% .74%

.11% .28% .24% .32% .09% .59%

.74%

.59%

Minimize Distractions:Remove Non-Critical Content +14%

Revenue PerVisitor

BASED ON KEYWORDCURATED

+24%Click to Sale

+30%Add-to-Cart

+210%Conversions

Problem Solution Result

All great hypotheses begin with the problem, NOT the solution!

The problem with “A/B Testing”

2

A B

4

A B C D

Most people test only 2 variations

0

10,000

20,000

30,000

40,000

50,000

60,000

2 3 4 5 6 7 >8

Tests 65% 17% 17%

# of Variations

Most people test only 2 variations

0

10,000

20,000

30,000

40,000

50,000

60,000

2 3 4 5 6 7 >8

Tests

# of Variations

Test 5 variations and nearly double your win rate!Win Rate

# of Variations

14% 18% 20% 27% 20% 22% 23%0%

10%20%30%40%50%60%70%80%90%

100%

2 3 4 5 6 7 >8

+90%

Running more variations will significantly improve your testing!



2 Test all changes


Workshop

§ The <user persona>§ Cannot find…§ Does not know…§ Does not see…§ Wants to…§ Is distracted by…§ Is forced to…§ Needs…


§ By <…> the user will… <…>

§ If <solution> solves <problem>, then <metric> will rise

Build powerful hypotheses

§ Discover 5-10 problems for your users

§ Group problems by themes

§ Prioritize most important problems


§ Select 1 concrete problem

§ Design 10 solutions per theme

§ Choose 4 preferred solutions

§ Primary Goals:What do you seek to impact immediately

§ Secondary Goals:What would you like to impact, but do not have to

§ Monitoring Goals:What are metrics you want to track

Exercise: Build Powerful Hypotheses

Optimizely funnel analysisHomepage

FlightResults

BookFlight

PassengerDetails

FlightOptions

Payment Page

Step

45%30%

65% 65%

90%

70%

40%

40%

15% 20%

5%

25%

15%30%

20% 15%5% 5%


FlightResults

BookFlight

PassengerDetails

FlightOptions

Payment Page

45%30%

65% 65%

90%

70%

40%

40%

15% 20%

5%

25%

15%30%

20% 15%5% 5%

Step

Completion


FlightResults

BookFlight

PassengerDetails

FlightOptions

Payment Page

45%30%

65% 65%

90%

70%

40%

40%

15% 20%

5%

25%

15%30%

20% 15%5% 5%

Step

Divergence


FlightResults

BookFlight

PassengerDetails

FlightOptions

Payment Page

45%30%

65% 65%

90%

70%

40%

40%

15% 20%

5%

25%

15%30%

20% 15%5% 5%

Step

Exit Rate


FlightResults

BookFlight

PassengerDetails

FlightOptions

Payment Page

Step

45%30%

65% 65%

90%

70%

40%

40%

15% 20%

5%

25%

15%30%

20% 15%5% 5%

Exit Rate

Divergence

Completion


FlightResults

BookFlight

PassengerDetails

FlightOptions

Payment Page

Step

Primary /SecondaryGoals

§ Searches Submitted

§ Category tabs§ Deals§ Frequent Flyer

§ Flight Selection§ Filter Usage§ Refine Search§ Sort Order

§ Continue button

§ Edit search§ Upsells

§ Field completion rate

§ Continue button

§ Continue button

§ Upsells§ Seat

selection

§ Conversion rate

§ Form field completion

§ Payment Type

MonitoringGoals

§ Checkboxes§ Flight tracker

§ Home page clicks

§ Menu interactions

§ Site search

§ Home page clicks


§ Site search

§ Field error rate

§ Home page clicks


§ Site search

§ Shopping cart pageviews

§ Home page clicks

§ Return to previous page

§ Edit address, information

§ Error rate

§ Discover 5-10 problems for your users

§ Group problems by themes

§ Prioritize most important problems


§ Select 1 concrete problem

§ Design 10 solutions per theme

§ Choose 4 preferred solutions

§ Primary Goals:What do you seek to impact immediately

§ Secondary Goals:What would you like to impact, but do not have to

§ Monitoring Goals:What are metrics you want to track

Exercise: Build Powerful Hypotheses

conversion optimization breakfast workshop

Technology