the evolution of a/b testing - nmhc optech 2016

THE EVOLUTION

OF A/B TESTING

@Nicole_Mintiens

Data Analysis & Insights Manager @GetG5

41 Shades of Blue

The Evolution of A/B Testing

@Nicole_Mintiens

$200 MILLION


@Nicole_Mintiens


@Nicole_Mintiens

Advertising and Marketing• Improved ROI = Good


@Nicole_Mintiens

Advertising and Marketing• Improved ROI = Good• Wasted Resources = Bad


@Nicole_Mintiens

Advertising and Marketing• Improved ROI = Good• Wasted Resources = Bad• Marketing = Complex


@Nicole_Mintiens

Advertising and Marketing• Improved ROI = Good• Wasted Resources = Bad• Marketing = Complex• Facts > Opinion


@Nicole_Mintiens

WHY SPEND TIME TESTING?• $92 traffic /$1 optimization

Source: Econsultancy & Redeye Conversion Rate Optimization Report 2014


@Nicole_Mintiens

WHY SPEND TIME TESTING?• $92 traffic /$1 optimization• 87% run 1-5 tests/month



@Nicole_Mintiens

WHY SPEND TIME TESTING?• $92 traffic /$1 optimization• 87% run 1-5 tests/month

– Best brands run hundreds per month



@Nicole_Mintiens

A/B TESTING FACTS• Improves conversion rate


@Nicole_Mintiens

A/B TESTING FACTS• Improves conversion rate• Enhances effectiveness


@Nicole_Mintiens

A/B TESTING FACTS• Improves conversion rate• Enhances effectiveness• Optimizes ROI


@Nicole_Mintiens

A/B TESTING FACTS• Improves conversion rate• Enhances effectiveness• Optimizes ROI

So… YOU should test!


@Nicole_Mintiens


@Nicole_Mintiens

MISCONCEPTIONS• Hurt conversion rate


@Nicole_Mintiens

MISCONCEPTIONS• Hurt conversion rate• Reduce effectiveness


@Nicole_Mintiens

MISCONCEPTIONS• Hurt conversion rate• Reduce effectiveness• Waste resources


@Nicole_Mintiens

MISCONCEPTIONS• Hurt conversion rate• Reduce effectiveness• Waste resources• Damage ROI


@Nicole_Mintiens

TESTING IS NOT

OPTIMIZATION


@Nicole_Mintiens

TESTING IS NOT

OPTIMIZATION…but it plays a role.


@Nicole_Mintiens

DON’T JUST TEST, OPTIMIZE.• Avoid mistakes


@Nicole_Mintiens

DON’T JUST TEST, OPTIMIZE.• Avoid mistakes• Increase effectiveness


@Nicole_Mintiens

DON’T JUST TEST, OPTIMIZE.• Avoid mistakes• Increase effectiveness• Improve ROI


@Nicole_Mintiens

I’LL BE YOUR GUIDE• 7+ years optimizing multifamily

– PPC, SMM, SEO & CRO


@Nicole_Mintiens

A/B TESTING BASICS• Statistical hypothesis-based

experimentation between variations of a control.


@Nicole_Mintiens

A/B TESTING BASICS• Statistical hypothesis-based

experimentation between variations of a control.

• Goal: – Isolate differences that

maximize an outcome • i.e. conversion


@Nicole_Mintiens

CONVERSION DEFINITIONAction YOU want the user to take.


@Nicole_Mintiens

A/B/N TESTING


@Nicole_Mintiens

• 2000 – Google runs first test to find optimum number of SERP

results.

A/B TESTING HISTORY


@Nicole_Mintiens


results.

• 2008 – A/B tests raise additional $60 million for Obama campaign.

A/B TESTING HISTORY


@Nicole_Mintiens


results.


• 2010– Obama campaign’s Director of Analytics founds Optimizely.

A/B TESTING HISTORY


@Nicole_Mintiens


results.


• 2010– Obama campaign’s Director of Analytics founds Optimizely.

• 2011 – Google has run over 7,000 A/B tests

A/B TESTING HISTORY


@Nicole_Mintiens

MULTIFAMILY AVG CONVERSION RATE

Source: 2016 G5 Multifamily Google Analytics Data


@Nicole_Mintiens

MULTIFAMILY AVG CONVERSION RATE


~2%The Evolution of A/B Testing

@Nicole_Mintiens

ECOMMERCE AVG CONVERSION RATE

~3%The Evolution of A/B Testing

@Nicole_Mintiens

AMAZON PRIME’S CONVERSION RATE

Source: https://www.internetretailer.com/2015/06/25/amazon-prime-members-convert-74-time


@Nicole_Mintiens

https://www.internetretailer.com/2015/06/25/amazon-prime-members-convert-74-time

2366%Better than

average!

AMAZON PRIME’S CONVERSION RATE

Source: https://www.internetretailer.com/2015/06/25/amazon-prime-members-convert-74-time


@Nicole_Mintiens

https://www.internetretailer.com/2015/06/25/amazon-prime-members-convert-74-time

AMAZON’S TECHNOLOGY

Source: Builtwith.com


@Nicole_Mintiens

SOFTWARE ADVANCEMENTS• Marketing automation


@Nicole_Mintiens

SOFTWARE ADVANCEMENTS• Marketing automation • Easy to install


@Nicole_Mintiens

SOFTWARE ADVANCEMENTS• Marketing automation • Easy to install• WYSIWYG editors


@Nicole_Mintiens

WHAT A/B TESTING CAN DO

• Validate hypotheses


@Nicole_Mintiens


• Validate hypotheses• Inform choices


@Nicole_Mintiens


• Validate hypotheses• Inform choices • Insight for iteration


@Nicole_Mintiens

WHAT A/B TESTING CAN’T DO• Determine YOUR

–Problem

Thanks, Boss!


@Nicole_Mintiens


–Problem –User’s

needs/wantsThanks, Boss!


@Nicole_Mintiens



needs/wants

• Improve traffic–Quality

Thanks, Boss!


@Nicole_Mintiens



needs/wants

• Improve traffic–Quality–Volume

Thanks, Boss!


@Nicole_Mintiens

CAN DOES NOT EQUAL SHOULDTesting will waste your time without:• Enough traffic


@Nicole_Mintiens

CAN DOES NOT EQUAL SHOULDTesting will waste your time without:• Enough traffic• Enough research


@Nicole_Mintiens

CAN DOES NOT EQUAL SHOULDTesting will waste your time without:• Enough traffic• Enough research• Technical resources


@Nicole_Mintiens

THEN HOW DO YOU IMPROVE?!


@Nicole_Mintiens

A PROCESSto improve performance

informed by INSIGHTand defined by GOALS.

CONVERSION RATE OPTIMIZATION(CRO)


@Nicole_Mintiens

A PROCESSto improve performance

informed by INSIGHTand defined by GOALS.

GOALS

CONVERSION RATE OPTIMIZATION(CRO)


@Nicole_Mintiens

ALWAYS ROOM FOR IMPROVEMENT• Customer acquisition cost


@Nicole_Mintiens

ALWAYS ROOM FOR IMPROVEMENT• Customer acquisition cost• Return on investment


@Nicole_Mintiens

ALWAYS ROOM FOR IMPROVEMENT• Customer acquisition cost• Return on investment• Customer understanding


@Nicole_Mintiens

ALWAYS ROOM FOR IMPROVEMENT• Customer acquisition cost• Return on investment• Customer understanding• Google’s perception


@Nicole_Mintiens

GOALS & TRACKING

Source: http://borsen.dk/opinion/blogs/view/17/4159/den_svaereste_marketingdisciplin.html


@Nicole_Mintiens

YOUR IDEAL RESIDENTS

Prospects

Valu

e

I only rent the most affordable apartments.


@Nicole_Mintiens

YOUR IDEAL RESIDENTS

Prospects

Valu

e

I only rent the most affordable apartments.

I’ll rent near my tech coworkers regardless of price.


@Nicole_Mintiens

CUSTOMER RESEARCH

Source: https://blog.kissmetrics.com/wp-content/uploads/2015/02/NYU.lowres.ifiwere.jpg


@Nicole_Mintiens

“The EFFECTIVENESS of

any lead generation campaign

comes down to your HOOK.”Mike King @IPullRankFounder iPullRankConsultant to Fortune/INC500 brands.


@Nicole_Mintiens

VOICE OF CUSTOMER RESEARCH• What makes your

property unique?


@Nicole_Mintiens

VOICE OF CUSTOMER RESEARCH• What makes your

property unique?

• Why you and not the competition?


@Nicole_Mintiens

UNIQUE VALUE PROPOSITION (UVP):Above-the-fold EXPERIENCEthat communicates WHATyou do and WHYit’s UNIQUE.


@Nicole_Mintiens

Original (A)

UVP: FIVE SECOND TEST


@Nicole_Mintiens

Original (A) Variation (B)

UVP: FIVE SECOND TEST


@Nicole_Mintiens

CUSTOMER SEGMENTATION


@Nicole_Mintiens

CUSTOMER SEGMENTATION

User Persona Creator: https://xtensio.com/user-persona/


@Nicole_Mintiens

https://xtensio.com/user-persona/

Slide credit to Angie Schottmuller - Check out the full presentation:

http://www.slideshare.net/aschottmuller/content-marketing-onpage-optimization-holy-grail


@Nicole_Mintiens

http://www.slideshare.net/aschottmuller/content-marketing-onpage-optimization-holy-grai

LARGEST LEAKS

Image source: https://s-media-cache-ak0.pinimg.com/564x/7e/c9/f9/7ec9f92ab4e7819c76011e432e760ece.jpg


@Nicole_Mintiens

ANALYZE ENGAGEMENT PATTERNSHeatmaps


@Nicole_Mintiens

ANALYZE PERCEPTIONS


@Nicole_Mintiens

ANALYSIS FRAMEWORKG5 D-LITE

Deliver Relevance - Match then exceed user expectations.D

L

I

T

EThe Evolution of A/B Testing

@Nicole_Mintiens


Lower Friction - Reduce user anxiety. Simplify their process.


L

I

T


@Nicole_Mintiens


Improve Clarity - Ensure messaging is clear, concise & valued.



L

I

T


@Nicole_Mintiens




Deliver Relevance - Match then exceed user expectations.

Trigger Action - Motivate conversion. Remove distraction.

D

L

I

T


@Nicole_Mintiens




Deliver Relevance - Match then exceed user expectations.

Experiment - Learn, test and optimize for your users.

Trigger Action - Motivate conversion. Remove distraction.

D

L

I

T


@Nicole_Mintiens

“A solid test hypothesis is an

INFORMED SOLUTIONto a real problem - not an arbitrary guess. ”Michael Aagaard

@ContentVerveInternational Speaker and Senior CRO @Unbounce


@Nicole_Mintiens

HYPOTHESIS TEMPLATEIf [cause], then [effect], because [rationale].

Source: https://help.optimizely.com/Ideate_and_Hypothesize/Design_a_hypothesis_that_drives_your_business_goals


@Nicole_Mintiens

TESTING


@Nicole_Mintiens

TESTING Wait… do you even have enough traffic?


@Nicole_Mintiens

Gives you CONFIDENCE that the changes you make to

your website ACTUALLYhave a positive IMPACT.

https://www.optimizely.com/optimization-glossary/statistical-significance/

STATISTICAL SIGNIFICANCE

“”


@Nicole_Mintiens

MULTIFAMILY AVG SESSIONS/DAY



@Nicole_Mintiens

MULTIFAMILY AVG SESSIONS/DAY


66The Evolution of A/B Testing

@Nicole_Mintiens

TEST DURATION CALCULATOR

Source: https://www.convert.com/tools/ab-test-duration-calculator/

Image source: http://www.vintag.es/2016/09/the-good-old-days-12-crazy-vintage-ads.html


@Nicole_Mintiens

LOW TRAFFIC OPTIMIZATION

Source: http://unbounce.com/landing-pages/i-dont-have-enough-traffic-to-ab-test-my-landing-pages/


@Nicole_Mintiens

A/B TESTING TOOLS


@Nicole_Mintiens

BUILD VARIATIONSUse technical resources to ensure proper setup and tracking


@Nicole_Mintiens

LAUNCH TESTOriginal

(A)Variation

(B)


@Nicole_Mintiens

MEASURE RESULTS

Image source: https://www.optimizely.com/ab-testing/


@Nicole_Mintiens

DRAW INSIGHTS AND ITERATE


@Nicole_Mintiens

Your properties aren't 1-size fits all and your strategy shouldn't be either.

DRAW INSIGHTS AND ITERATE


@Nicole_Mintiens

MOST IMPORTANTLYResearch and

user feedback is

VITAL.


@Nicole_Mintiens

REPEAT

GOALS


@Nicole_Mintiens

THERE IS NO

BEST COLOR

41 Shades of Blue


@Nicole_Mintiens

INDUSTRY BENCHMARKS

• Contextless data


@Nicole_Mintiens

INDUSTRY BENCHMARKS

• Contextless data• Weak conclusions


@Nicole_Mintiens

INDUSTRY BENCHMARKS

• Contextless data• Weak conclusions• Blind duplication


@Nicole_Mintiens

INDUSTRY BENCHMARKS

• Contextless data• Weak conclusions• Blind duplication • Tactics over strategy


@Nicole_Mintiens

USE STRATEGY,NOT JUST TACTICS

• Avoid risks


@Nicole_Mintiens


• Avoid risks• Use data effectively


@Nicole_Mintiens


• Avoid risks• Use data effectively• Discover solutions to

YOUR problems


@Nicole_Mintiens

FREEMIUM (OR CLOSE) RESOURCES• Research:

– Personas

– Tracking

– Heatmapping

– Test Duration Calculator


@Nicole_Mintiens

https://www.convert.com/tools/ab-test-duration-calculator/




https://www.google.com/analytics/analytics/#?modal_active=none


www.hotjar.com

www.hotjar.com


– Personas

– Tracking

– Heatmapping


• Analysis Tools:

G5 D-LITE CRO Framework


@Nicole_Mintiens







www.hotjar.com

www.hotjar.com


– Personas

– Tracking

– Heatmapping


• Analysis Tools:


• Hypotheses:

If [cause], then [effect], because [rationale]


@Nicole_Mintiens







www.hotjar.com

www.hotjar.com


– Personas

– Tracking

– Heatmapping


• Analysis Tools:


• Hypotheses:

If [cause], then [effect], because [rationale]

• Testing:


@Nicole_Mintiens

www.usetesting.com

www.usetesting.com

https://usabilityhub.com/

https://usabilityhub.com/

https://adwords.google.com

https://adwords.google.com

www.optimizely.com

www.optimizely.com







www.hotjar.com

www.hotjar.com

https://www.google.com/analytics/optimize/

https://www.google.com/analytics/optimize/

“CRO IS LEARNING WHYYOUR USERS aren’t

converting and FIXING IT.”The Evolution of A/B Testing

@Nicole_Mintiens

GET THESE SLIDES: goo.gl/TPUOgA

THANKS!

DON’T JUST TEST, OPTIMIZE.


@Nicole_Mintiens