Whattotestnext?PrioritizingyourtestsPaulineMarol

SeniorProductManager

pmarol@hotwire.com

JosephineFoucher

OptimizationProductManager

jfoucher@hotwire.com

ConversionXLLive2016

Agenda • Wheretostart

• DefiningPrioritizationRules

• TheHotwireframework

• MonitoringtheHealthofyourOptimizationProgram

2

Wheretostart?

Whyyouneedaprioritizationprocess

• Managing stakeholders• Bringtransparencytooptimization• Buildtrustfromthecompany• Empoweremployees

4

Makeyourlifeeasier

Knowyourdestination

• Testingprofile• Productstrategy• Optimizationgoals

5

Knowwhatyouwant

Knowyourself

• Getyouranalyticsright• Listentoyourcustomers• Lookatyourcompetitors

6

Identifyyouropportunities

Structureyourideas

• Translateideasintohypotheses• Centralize• Organizeandtag

7

Buildyourbacklogofhypotheses

Definingprioritizationrules

BryanEisenbergrules MonetatemodelPIEframework Pointsmodel

Therearemanyprioritizationframeworks

ItallcomesdowntoValueandEffort

Ifyougetvalueandeffortright,therestiseasy

10

High

Low

High Low

Value

Effort

ValidateHypothesis

Doit!

Forgetit Ifyouhavetheresources

Reach Lift Strategicfit

IdentifythedriversofValueandEffort

11

Value Effort

Howmanypeoplewillbeimpacted?

Reasonstoexpectastrongreaction

Alignedwithstrategy

Alltheframeworksbreakdownvalueand/oreffortintosub-elements,herearethemostcommon.

Creative Development Coordination

InthePIEframework

12

Value

Reach Lift Strategicfit

Effort

Creative Development Coordination

Potential Importance Ease

BreakingdownValueintoPotentialandImportance

IntheMonetatemodel

13

Value

Reach Lift Strategicfit

Effort

Creative Development Coordination

Potential CreativeLOE TechnicalLOE

BreakingdownEffortintoCreativeandDevelopment

IntheBryanEisenbergrules

14

Value

Reach Lift Strategicfit

Effort

Creative Development Coordination

Impact Resources

Time

BreakingdownEffortintoResourcesandTime

Definebinaryvaluerulesthat:• Areobjective (yes/no)• Discriminate ideas• Leverageyourknowledge

Takingittothenextlevel:ThePointsmodel

15

Value

Reach Lift Strategicfit

Effort

Creative Development Coordination

Defineeffortrulesthat:• Representyourbottlenecks• Arealigned withexistingsizing

Breakdowneachsub-element intospecificrules,toremovepersonalbias

HotwireCarFramework

1. Primarymetricisconversion

2. Addingorremovinginfo

3. Research/analytics4. Benchmarking

InthePointsmodel

17

Value

1. Above thefold2. Allcustomers

willseeit

1. $$product2. Cancellation3. Choice4. Multiplethemes

Effort

Creative Development Coordination

UX/UIpointsCopywritingneeds

Engineeringpoints

Reach Lift Strategicfit

HotwireCarprioritization

If weaddaFreecancellationflagonthecheckoutpageThen ourconversionratewillincreaseBecause customerswillfeelmoreconfidentaboutreservingacarknowingthattheycanchangetheirmind

18

HotwireCarprioritization

19

Fold

Test 1

HotwireCarprioritizationFree cancellationflag

20

Fold

100%

Test 1 0

Fold

100%

Mainmetric

Test 1 0 1

Fold

100%

Mainmetric

Adding

or

remov

ing

Test 1 0 1 1

Fold

100%

Mainmetric

Adding

or

remov

ing

Analytics

or

research

Test 1 0 1 1 0

Fold

100%

Mainmetric

Adding

or

remov

ing

Analytics

or

research

Benchm

arkin

g

Test 1 0 1 1 0 1

Fold

100%

Mainmetric

Adding

or

remov

ing

Analytics

or

research

Benchm

arkin

g

$$produ

ct

Test 1 0 1 1 0 1 0

Fold

100%

Mainmetric

Adding

or

remov

ing

Analytics

or

research

Benchm

arkin

g

$$produ

ct

Cancellatio

n

Test 1 0 1 1 0 1 0 0

Fold

100%

Mainmetric

Adding

or

remov

ing

Analytics

or

research

Benchm

arkin

g

$$produ

ct

Cancellatio

n

Choice

Test 1 0 1 1 0 1 0 0 0

Fold

100%

Mainmetric

Adding

or

remov

ing

Analytics

or

research

Benchm

arkin

g

$$produ

ct

Cancellatio

n

Choice

2them

es

Test 1 0 1 1 0 1 0 0 0 1

Fold

100%

Mainmetric

Adding

or

remov

ing

Analytics

or

research

Benchm

arkin

g

$$produ

ct

Cancellatio

n

Choice

2them

es

Total

Test 1 0 1 1 0 1 0 0 0 1 5

Fold

100%

Mainmetric

Adding

or

remov

ing

Analytics

or

research

Benchm

arkin

g

$$produ

ct

Cancellatio

n

Choice

2them

es

Total UX Dev

Test 1 0 1 1 0 1 0 0 0 1 5 S S

HotwireCarprioritization

21

Fold

100%

Mainmetric

Adding

or

remov

ing

Analytics

or

research

Benchm

arkin

g

$$produ

ct

Cancellatio

n

Choice

2them

es

Total UX Dev

Test1 1 0 1 1 0 1 0 0 0 1 5 S S

Test2 1 1 1 1 1 0 1 0 0 0 6 M L

Test150 0 0 1 1 1 1 1 1 1 1 8 L S

MonitoringtheHealthofyourOptimizationProgram

UseKPIstomonitorandcommunicateperformance

Choosemetricsbasedonyourriskprofileandtestingstrategy.

Werecommend:• Velocity• WinRate• Conversionlift

IfyourKPIsgodown,revisityourrules.

Benchmarkyourprioritizationrulestostayrelevant

Validatethecorrelationbetweenscoreandreality• Valuescorevs.Winrate

Ifyourrulesefficiencyflattens,revisityourrules.

0%

25%

50%

75%

100%

0 2 4 6 8 10

WinRate

ValueScore

BenchmarkingValuePoints

ShareandEducate

• Shareyourprioritization rulesandbetransparent• Getfeedbacktomaketherulesbetter• Educateandscaleoutsideyouroptimizationteam

Whenyouaggregatefeedback,revisityourrules.

Buildknowledge

• Aggregateresults• TranslateCRliftintolearning andcustomer

insights• Adjustyourtestingprofile,youroptimization

goals

Asyouaccumulatelearning,revisityourrules.

Yourturn!

27

1 2 3

Doyourresearch Buildprioritizationrules Keepimprovingthem

We’reHiring!

PaulineMarol

pmarol@hotwire.com

JosephineFoucher

jfoucher@hotwire.com