use data analytics for best return on marketing spend€¦ · the best return on your marketing...

C O N F I D E N T I A L

Use data analytics

for best return

on marketing spend

Jonathan.Isernhagen@travelocity.com @Jon_Isernhagen

EyeForTravel Smart Travel Analytics

2/12/2014

C O N F I D E N T I A L

Use Data Analytics To Get The Best Return On Your Marketing Spend

1) Implement a data-driven SEO strategy for higher ad position,

better conversion rates, higher ROIs, and lower cost per click

2) Use attribution wisely for accurate media spending in 2014

3) Only buy the media that work: track visitor behavior and

every dollar spent

4) Measure and track behavior to improve your mobile website

or application and use it to engage with customers

5) Time your marketing mix for maximum return using attribution

and reservation data.

C O N F I D E N T I A L

Agenda

1) Goal of channel spend

2) Consequences of inaccurate attribution

3) Approaches/levels of attribution a) Algorithmic attribution

b) Arbitrary attribution

i. Bracketing

ii. Tactical spend decisions by channel

C O N F I D E N T I A L

Positive Return on Incremental Channel Investment

- 4 -

ROI = (VCM – Spend)

Spend

Abbreviation Term Definition

ROI Return on Investment Indicator of investment profitability.

Positive = good.

VCM Variable Contribution Margin The amount of profit driven by a

given transaction.

Spend Channel Spend The amount spent driving traffic to

the site during the period in question

Calculated over a specified time period of investment and return.

C O N F I D E N T I A L

Which marketing treatment(s) triggered the purchase?

"Half the money I spend

on advertising is wasted;

the trouble is,

I don't know which half.“

-John Wanamaker

Father of Modern Advertising

C O N F I D E N T I A L

“Hard to Know” ≠ “Random”

We may not find the correct answer, but the answer does exist.

C O N F I D E N T I A L

Agenda

1) Goal of channel spend

2) Consequences of inaccurate attribution

3) Approaches/levels of attribution a) Algorithmic attribution

b) Arbitrary attribution

i. Bracketing

ii. Tactical spend decisions by channel

C O N F I D E N T I A L

Google Analytics attribution options

Google offers several attribution

mechanisms out of the box

Choosing the wrong model

can have consequences.

C O N F I D E N T I A L

How badly do you want to know?

Raylan: You'll pay to find that out. Boyd: What are you packing?

C O N F I D E N T I A L

Choosing your attribution strategy

Do you want/need

true attribution*?

Do you have

data guys?

Do they have

bandwidth for this?

Yes

No

No

Use your site

metric solution’s

attribution

Yes

Yes Hire a vendor

Can you access

your data?

Yes

No

*Do you have:

1) Large enough budget?

2) Multiple channels?

3) Belief in ROI “knowability?”

DIY

No

C O N F I D E N T I A L

Agenda

1) Goal of channel spend

2) Consequences of inaccurate attribution

3) Approaches/levels of attribution a) Algorithmic attribution

b) Arbitrary attribution

i. Bracketing

ii. Tactical spend decisions by channel

(skip)

C O N F I D E N T I A L

• Sends

• Opens

Collecting the necessary data

Analysis

space

• Clicks

• Visitors

• Bookings

• VCM

• Media impressions

Site metrics tool

Back office system

• Channel-specific

phone #s

Email distribution tool

Call Center IVR

• Impressions

• Spend

Ad server

e.g. SAS, Revolution Analytics,

SPSS, Teradata Warehouse Miner

SEM Bid Management tool

• Impressions

• Clicks

• Spend

• GRPs

• Spend

Television plan

• Impressions

• Clicks

• Spend

Meta search feeds

• Spend

Manual spend entry table

C O N F I D E N T I A L

Connecting the necessary data

Transactions Profiles Customers

Sessions

∞ 1 ∞ 1 Clicks:Transactions Clicks

∞ 1 1 ∞

∞ 1 ∞ 1

Calls

Sends

Opens ∞

1

∞

1

Email

IVR

GRPs

TV plan

Impressions

Ad server

Impressions

∞ 1

∞ 1

∞ 1

Back office Site monitoring tool CRM system

1 ∞

C O N F I D E N T I A L

Forrester 2012Q2 survey of attribution vendors

Our Stats PhD audited the players, settled on pilot with new upstart Datasong (formerly “Upstream.”)

C O N F I D E N T I A L

“Given all the history we know, how likely is this shopper to convert soon?”

Datasong answers with a 2-stage model:

-Are Brand shoppers (BSEM, BSEO, DTI) likely to book? Yes

-Did your last TV spot cause their brand loyalty? Maybe

Algorithmic attribution: Survival Modeling

Model 2 Accuracy 81%

Model 1 Accuracy 68%

0

0.5

1

0 0.5 1

NonConverters Converters

0

0.5

1

0 0.5 1

NonConverters Converters

C O N F I D E N T I A L

Algorithmic attribution: Survival Modeling

1) The Axes show the time since last channel exposure

2) Dots represent converters vs. nonConverters

3) The Orange line represents the relationship between

2 variables, e.g.: a) time since last email and

b) time since last affiliate visit

4) Responders: everyone above the line. We count the

folks above the line and see what our accuracy is

5) Attribution: once satisfied with a model(s), we’d: a) take a given order,

b) see the time since last email and affiliate and based on

the timing, and;

c) where we are on the orange line, so;

d) we have a means to allocate which was more causal.

0

0.5

1

0 0.2 0.4 0.6 0.8 1

NonConverters Converters

C O N F I D E N T I A L

Survival Modeling: DIY

Recommended by one of our

PhD statisticians.

“SAS usage is not necessary.”

C O N F I D E N T I A L - 18 -

Algorithmic attribution: Interaction Method

Brand

SEM E-mail Affiliate

5.6%

Conversion

Brand

SEM Affiliate

5.8%

Conversion

Some channel interactions may actually take conversion backwards.

Only by examining all interactions can you know for sure.

C O N F I D E N T I A L

Running the process

1) For each site visitor a) Assemble visit history

b) Create variables to represent: i. Channel impressions

ii. Channel clicks

iii.Past purchases

2) Regress or use machine learning algorithm a) Ascertain which channel touches predict booking

b) Give VCM credit to causal channels

3) Calculate ROI a) Use each channel’s VCM and spend

b) Where ROI is positive, spend up

c) Where ROI is negative, cut spend or change tactics

C O N F I D E N T I A L

Agenda

1) Goal of channel spend

2) Consequences of inaccurate attribution

3) Approaches/levels of attribution a) Algorithmic attribution

b) Arbitrary attribution

i. Bracketing

ii. Tactical spend decisions by channel

(skip to end)

C O N F I D E N T I A L

“Round up all the clicks!”

• Offline bookings

• Multiple profile activity

• Impressions (Display or Television/Radio/etc)

• Cleared-cookie clicks

- 21 -

Leaves out:

C O N F I D E N T I A L

55% of shoppers clear cookies every 3 months

Cookie clearing is a serious problem causing the true impact of online

channels to be massively understated.

“The Consumer Side of Analytics” - Forrester/Jupiter Research 2009

C O N F I D E N T I A L

Bracketing

1) Assume for the sake of argument that one of the site tool’s attribution

schemes reflects reality, but you don’t know which one.

2) Find (or create) the scheme that is most and least generous to digital

channels. a) Last click: usually the least generous to digital channels

b) 3-tier: a scheme we created to favor NBSEM and Meta/Travelsearch

3) Assume that the true attribution (& ROI) lies between these extremes

4) Adjust spend and see which scheme better predicts reality.

C O N F I D E N T I A L

Tactical decision-making

Ignore the big picture and focus on the borderline projects in each channel

• Brand advertising

• SEM (Brand and Nonbrand)

• Email

C O N F I D E N T I A L

Tactical decisions: Brand

Easy decision: advertising heavily in geographies of known high

customer concentration during periods before high demand.

Hard decisions:

• national vs. regional

• whether uniform or in intermittent bursts

• invest more in air time or fresh creative

Tactical solutions:

1) A/B test across “identical” municipalities/regions, observe rise in

Brand-attributable (BSEM, BSEO, DTI) bookings.

2) Hire Comscore to measure brand awareness/intent after broadcast.

C O N F I D E N T I A L

Tactical decisions: Search Engine Marketing

Easy decision: spending to capacity on Brand SEM, which looks

profitable under almost all attribution schemes

Hard decision: whether to spend up on NonBrand SEM keywords which

don’t appear profitable no matter how you attribute.

Tactical solution:

• A/B test whether reduced BSEM spend is offset by increased BSEO

booking volume

C O N F I D E N T I A L

Tactical decisions: Email distribution

Easy decision:

• emailing up to 1x/week, hitting each site visitor with the message he or

she is most likely to convert from.

• sending messages advertising exceptional deals.

Hard decisions:

• whether to upgrade to rules-based campaign (e.g. Responsys, ET)

• whether to saturation-bomb your e-mail list, raising overall yield but

boosting unsubscription.

• whether to push loss-leading promo code, boosting overall conversion

but training site shoppers to expect discounts

C O N F I D E N T I A L

Take-away lessons

- 28 -

1) Be intentional about your attribution decisions, and creative about how to test your assumptions.

2) For top competitors in each vertical, the question is no longer “should we do algorithmic attribution?” but rather “How can we operationalize our attribution model?”

3) If your data aren’t yet in shape to support attribution & other ad hoc analyses, get that project underway.

Jonathan.Isernhagen@travelocity.com @Jon_Isernhagen

C O N F I D E N T I A L

Attribution: arbitrary methods

# Channel Rationale

1 Last click ABC: Always Be Closing.

2 Last paid click Direct is a point of sale, not a marketing channel.

3 Average/hyperbolic Every channel is special.

4 First click First attracted the consumer

5 First relevant click Started the conversation that led to the purchase.

Which channel(s) gets the credit in the ROI calculation for any given booking?

- 30 -

T C C C C C

NYC hotel

1/1/13

DTI

NYC hotel

1/1/13

BSEM

NYC hotel

1/1/13

E-mail

NYC hotel

1/1/13

Meta search

ATL-NYC flight

1/1/13

SEO

W. Carib. cruise

6/1/13

1 2

3

4 5

45-day

lookback

All of these make assumptions about causality but give no evidence of causality.

C O N F I D E N T I A L

DataSong’s Modeling Architecture

• ETL

• N marketing channels

• Behavioral variables

• Promotional data

• Overlay data

• Functions to read Hadoop output; xdf creation

• Exploratory data analysis

• GAM survival models

• Scoring for inference

• Scoring for prediction

• 5 billion scores per day

per customer

UPSTREAM DATA

FORMAT (UDF)

CUSTOM VARIABLES

(PMML)

C O N F I D E N T I A L

What analysis method was used?

Survival Response Model on Accounts and Cookies

Analyze effects of marketing on response over time

In terms of both what worked and what didn’t

Time-dependent:

Inputs: estimate the speed of marketing effects’ decay

Outcome: look at times where a bkg has or hasn’t yet been made

Create models specific to:

LOB: the impact of SEM varies from auto to packages

Customer: repeat purchaser motivations are more knowable

Controls for non-marketing effects:

Anniversary effects: many sales driven by season demand

Customer-driven activity: many customers driven by loyalty

C O N F I D E N T I A L

VCM

Transaction revenue minus variable non-marketing expenses:

• Revenue: –Supplier Commissions,

–GDS incentives,

–Overrides (lumpy: average/booking must be assumed)

–Media (not transaction-driven, but has to be modeled in somewhere)

–Attached bookings / Lifetime value: try to gauge full value without double-

counting

• Expenses: –Website hosting/capacity costs

–Data processing expenses

The all-inclusive profitability of the transactions driven

- 33 -

C O N F I D E N T I A L

Marketing ≈ Farming

1) Fertilizing/irrigating = brand advertising

2) Planting = presence on the digital shelf

- 34 -

Which factor or factors caused you to have a good

(or bad) crop yield?

C O N F I D E N T I A L

Attribution options - broad view

Provider Method Cost

Timing Comments 1x monthly

Adobe Attribution model Varies - 4-10 wks Consulting arm willing to address attribution questions

as one-off modeling projects.

Adometry Econometric

model Low High 12 wks

Forrester-acclaimed industry co-leader.

Highly display-focused.

Datasong

Attribution model

w/

Customer

scoring

Med

Med

6 wks

Comprehensive model which pulls in both online and

offline spend impacts.

Med Per-customer demand curve to predict the expected

return on every campaign

Google

Analytics Interactions

model - - now

Has delivered digital-only channel interactions model

to beta clients.

Gauge LLC Simple

regression High - 10 wks

General consulting analysts, not on the efficient

frontier for this deliverable.

Kenshoo Unknown

algorithm - Med 4 wks

Primarily targeted towards SEM spend, but can be

expanded to comprise other digital channels

Visual IQ Econometric and

Attribution

models

High - 4-8 wks Econometric model. Offers cross-channel allocation

guidance, including offline spend.

Med-

High - 3-6 wks

Attribution model. Offers within-channel campaign-

level spend guidance. No offline.