opticon 2015-scaling your testing program for maximum impact

#opticon2015

Scaling your Testing Program for Maximum ImpactAndy NelsonDirector of Growth Marketing, Moz

@theandynelsonandy.nelson@moz.com

Martijn ScheijbelerLead Growth, SEO & Analytics, The Next Web

@martijnschmartijn@thenextweb.com

Khattaab KhanStrategy Consultant, Optimizely

@khattaabkhankhattaab@optimizely.com

Increase Your Impact Without More Resources Andy NelsonDirector of Growth Marketing, Moz

@theandynelsonandy@moz.com

#opticon2015

~2M visits/month

#1 Go Big

Evolution vs. Revolution

Quality > Quantity

#2 Prioritize Intelligently

Elements of good prioritization

Core• Potential impact to success metric• Estimated runtime required• Estimated level of effort

Elements of good prioritization

Modifiers• Alignment with strategy• Support from VOC, analytics, prior

testing• Urgency

Core• Potential impact to success metric• Estimated runtime required• Estimated level of effort

http://bit.ly/TestPrioritization

#3 Swim Lanes

Homepage

Blog

Conversion Funnel

In Product

Visitors 1-25

Visitors 26-50

Visitors 51-75

Visitors 76-100

Homepage

#4 Multiple Streams of Work

1 week

1 day1 day

Checkout Optimization

Product Page Optimization

Test Analyze

Design/Build

Test 1

Test 2

Test 3

Test 4

Test Analyze

Design

Test

Design/Build

#5 Prevent Busted Tests (or at least catch them early)

Chrome

Safari

IE

Firefox

Chrome

Safari

IE

FirefoxBUSTED

Have a process

Pre-launch QA

Final Sign-Off

Post-Launch QA

Weekly Check-Up

Test across browsers/devices

0

22500

45000

67500

90000

5/3 5/10 5/17 5/24 5/31 6/7

Watch for irregular events

Test big, bold changesImprove your prioritizationRun multiple testsEliminate dead timeImprove your QA

Takeaways

http://bit.ly/TestPrioritization

#opticon2015

Thank you!

Andy NelsonDirector of Growth Marketing, Moz

@theandynelsonandy@moz.com

#opticon2015

How The Next Web’s Testing Program ScaledMartijn ScheijbelerLead Growth, SEO & Analytics - The Next Web

@martijnschmartijn@thenextweb.com

Yearly 5.000 attendees, 75.000 products sold. Monthly 6 million unique users, 12 million pageviews.

The Next Web: 2006 > 2015

What’s happened in the last 12 months?

Scaling our testing program

That clearly wasn’t good enough

1 test a month :(

Currently 5-6 tests a week, how did we get there?

15-20 tests a month :)

What did we learn along the way?

• Swim lanes, KPIs, Goals.

• Segments: traffic sources, etc.

• Idea creation: backlog + prioritisation.

• Scaling > Analysing results.

• Do it live!

Unique Users, Page Views, Emails & Revenue.

Goals: UU, PV, Emails & $

Related stories, pagination, sharing, email signups.

KPI: CTR, Engagement, Funnels

New v.s. Returning - Europe v.s. USA - Facebook v.s. SEO - Mobile v.s. Desktop

Type, Regions, Source, Device

Idea creation v.s. Planning

• Different templates + segments.

• How much time per test? Run tests for a fixed predefined time.

• The maximum of your program.

• Take away: create a backlog!

The part that we can’t scale yet

Analysing > Scaling!

Well, this is both the most boring and most exciting part!

1-2 main KPIs, 15 secondary KPIs

Rails, Google Analytics API Built with love by the TNW Marketing team.

TNW Analytics

TNW Analytics was born!

Adding a new idea

Schedule

Test

Saving 20% of time per test!

Take away: Be able to push code!

What’s next?

Scaling our testing program to the next level?• Even more templates, across more projects.

• 25+ tests a month with current growth of traffic.

• Bigger team? We can’t keep up with coding ;-)

• More automation? Push test variant to GTM.

• Mix it up with personalisation/ behaviour.

• Push our experiments to Slack for sharing.

• Create a better style guide + Bootstrap guidelines.

Summary & Take aways

• Know your KPI framework, expectations and limitations.

• Know your segments, CR% so you know how to prioritise.

• Follow your planning, priorities.

• Scale = 40% - 50% automation: analyse fast!

• Be able to push code live!

#opticon2015

Have fun & Thank you!

Martijn ScheijbelerLead Growth , SEO & Analytics - TNW

@martijnschmartijn@thenextweb.com

Strategies Essential to Achieve Testing Program Scale

#opticon2015

Khattaab KhanStrategy Consultant, Optimizely

@khattaabkhankhattaab@optimizely.com

Focus on Optimization Opportunities that Matter

44% 42%

8%

52%

87% 81%

41% 31%

44%

32%

11% 12%

15% 27%

48%

16% 7%

Homepage to Category

Category to Product Product to Shopping Cart

Shopping Cart to Checkout

Checkout to Payments

Payments to Conversion

Continue down funnel

Visit different page

Exit website

EXAMPLE

Homepage Category

Page Product

Page Shopping

Cart Checkout

Page Payment

Page

With a conversion funnel, we could analyze all steps to see how and where users drop off Evaluate User Path

44% 42%

8%

52%

87% 81%

41% 31%

44%

32%

11% 12%

15% 27%

48%

16% 7%

Homepage to Category

Category to Product Product to Shopping Cart

Shopping Cart to Checkout

Checkout to Payments

Payments to Conversion

Continue down funnel

Visit different page

Exit website

EXAMPLE

Homepage Category

Page Product

Page Shopping

Cart Checkout

Page Payment

Page

With a conversion funnel, we could analyze all steps to see how and where users drop off

Focus Testing on Problem Areas that are Drivers of Business Value

Intelligent Documentation for

Intelligent Prioritization

Roadmap

Roadmap + Resource Planning

Custom Prioritization Scoring

Pauline Marol Lead PM, Site Optimization

“I don’t want to be the one making decisions on what experiments I run and don’t run because I am biased.”

Results Analysis

Results Page Best Practices • Evaluate opportunity costs

• Analyze segment performance

• Shareable results link

Results Sharing

Communicating Results + Roadmap Iteration • Hold team accountable to strategy that inspired variation design

• Validate/invalidate hypothesis to evaluate goal effectiveness

• Define where results output will live (e.g. database, wiki, etc.)

Simultaneous Testing

Simultaneous Testing Use Cases

Experiments on same page

Experiments on same flow (e.g. checkout / registration flow)

Multivariate experiment

Beware of Interaction Effects

Elements on same page

Elements on subsequent pages in a funnel

When multiple points of user interaction influence the same goal, interaction effects can taint data

Evolution of Testing Org Structure

Core Testing Team

MEASURE RESULTS

Analytics

Dev + QADesign

Program Manager

CRO Analystoptional

Evolution intoOptimization Center of Excellence

MEASURE RESULTS

Analytics

Dev + QADesign

Program Manager

ProductManager

ProductManager

Dedicated optimization resources serving

multiple BUs

Leverage data to focus on goals that directly drive business valueHold your team accountable with effective planning documentsKnow when to run simultaneous tests

Scale your program structure to support multiple business units

Takeaways

#opticon2015

Thank you!

Khattaab KhanStrategy Consultant, Optimizely

@khattaabkhankhattaab@optimizely.com

#opticon2015

Andy NelsonDirector of Growth Marketing, Moz

@theandynelsonandy.nelson@moz.com

Martijn ScheijbelerLead Growth, SEO & Analytics, The Next Web

@martijnschmartijn@thenextweb.com

Khattaab KhanStrategy Consultant, Optimizely

@khattaabkhankhattaab@optimizely.com

Now for some Q&A