always be testing: how eroi + taco bell built a testing agenda (and what happened next)
TRANSCRIPT
1
ALWAYS BE TESTING
How eROI + Taco Bell Built a Testing Agenda(and What Happened Next)
Cher FullerSENIOR DIGITAL STRATEGIST
@cherfuller
Christine LiE-COMMERCE INNOVATION
@li_tacobeta
FULL BLEED ANCHORS
MARGINS
MARGINS
2
3
FULL BLEED ANCHORS
MARGINS
MARGINS
“Without data, you’re just another person with an OPINION.”
-W. Edwards Deming, data scientist
Here are some other people with opinions….
4
FULL BLEED ANCHORS
MARGINS
MARGINS
5
FULL BLEED ANCHORS
MARGINS
MARGINS
Product testing TOLEDO, OH.
6
FULL BLEED ANCHORS
MARGINS
MARGINS
Digitally engaged brandsswim in OCEANS OF DATA.
7
FULL BLEED ANCHORS
MARGINS
MARGINS
More data is being collected and analyzed today than at ANY OTHER TIME IN HISTORY.
8
FULL BLEED ANCHORS
MARGINS
MARGINS
2.5 billion gigabytesof data are collected
every day.
90% of globally stored data was collected in
the past two years.
IBM: What is Big Data? https://www-01.ibm.com/software/data/bigdata/what-is-big-data.html
9
A
B A
B
By A/B TESTING, right?
1 2 3 4 5
10
FULL BLEED ANCHORS
MARGINS
MARGINS
But A/B testing is step 5.
1 2 3 4 5
11
FULL BLEED ANCHORS
MARGINS
MARGINS
But A/B testing is step 5.
You need to start at step 1.
12
FULL BLEED ANCHORS
MARGINS
MARGINS
To swim this ocean of data,first TEST THE WATER.
13
Build a Learning Agenda
FULL BLEED ANCHORS
MARGINS
MARGINS
14
FULL BLEED ANCHORS
MARGINS
MARGINS
A LEARNING AGENDA asks specific questions in a specific order to set the framework for a testing strategy.
LEARNING AGENDA
FULL BLEED ANCHORS
MARGINS
MARGINS
15
HIERARCHYWHY
Information hierarchy strongly influences user engagement.
By testing how prominently we present messages, we can measure what our users really care about—and why.
BENEFITS CONTENT TYPESWHY
To understand consumer incentive, we must pinpoint the benefit to online ordering.
Without this incentive, online ordering loses its value and risks shifting to in-store purchases.
WHY
Test engagement within several modules against a control to learn if specific types of content are more engaging than
others.Once we understand the value of certain modules, we can
make strategic decision around when/where to place them in triggered emails.
Hierarchy
Benefits Content Types
Information hierarchy strongly influences user engagement.
By testing how prominently we present messages, we can measure what our users really care about—and why.
Create an outlineThe success of a deck relies on a solid outline, which should happen well before you create a new deck.Drafting a deck outline will not only prevent hours of unused work, but it will be the nucleus for team alignment and effective storytelling.
The deck outline can also serve as a helpful means to define who speaks to what points, and function as speaking notes!
To understand consumer incentive, we must pinpoint the benefit to online ordering.
Without this incentive, online ordering loses its value and risks shifting to in-store purchases.
Gangsta’s ParadiseMINUTE AFTER MINUTE, HOURI’m the kinda G that little homieswant to be like.They keep spending most their lives living in the gangsta’s paradise. They keep spending most their lives living in the gangsta’s paradise. They keep spending most their lives living...They keep spending most their lives living in the gangsta’s paradise.They keep spending most their lives living in the gangsta’s paradise.They keep spending most their lives living in the gangsta’s paradise.
TOO MUCH TELEVISION WATCHING GOT ME CHASING DREAMS.I’M EDUCATED FOOL WITH MONEY ON MY MIND.
Test engagement within several modules against a control to learn if specific types of content are more engaging than others.
Once we understand the value of certain modules, we can make strategic decision around when/where to place them in triggered emails.
16
Before starting any learning agenda, ask...
1. What are you testing?2. Why are you testing it?3. What do you want to learn?4. What will you do with your insights?
FULL BLEED ANCHORS
MARGINS
MARGINS
P
StHa
Before starting any LEARNING AGENDA, ask…
What do you want to learn?
What will you do
with your insights?
What are you
testing?
Why are you
testing it?
What are you testing?
What do you want to learn?
Why are you testing it?
What will you do with your insights?
1
2
3
4
17
SEE TEXT ON SLIDEFULL BLEED
ANCHORS
MARGINS
MARGINS
P
StHa Why does a brand care about the DATA STORY?
18
Where We Sometimes Misstep
FULL BLEED ANCHORS
MARGINS
MARGINS
19
FULL BLEED ANCHORS
MARGINS
MARGINS
NOT EVERYONE speaks data.We need to translate it.
20
WHAT WE WILL COVER TODAY
FULL BLEED ANCHORS
MARGINS
MARGINS
Make the complexSIMPLE. A LEARNING AGENDA
might test:
21
FULL BLEED ANCHORS
MARGINS
MARGINS
TELL STORIES,not reports.
22
So We RevisualizedThe Learning Agenda.
FULL BLEED ANCHORS
MARGINS
MARGINS
23
FULL BLEED ANCHORS
MARGINS
MARGINS
BEFORE AFTER
24
SEE TEXT ON PAGE
FULL BLEED ANCHORS
MARGINS
MARGINS
P
StHa This learning agendahelped Taco Bell’s brand teamTAKE TRULY DATA-DRIVEN ACTIONS.
25
We RevisualizedOur Performance Reports Too
FULL BLEED ANCHORS
MARGINS
MARGINS
26
FULL BLEED ANCHORS
MARGINS
MARGINS
Digital Strategists
Performance Analysts
27
FULL BLEED ANCHORS
MARGINS
MARGINS
P
StHa PERFORMANCE REPORTSinfluence brand decisions
PERFORMANCEReports
29
FULL BLEED ANCHORS
MARGINS
MARGINS
qualitative data branding operations quantitative data
performanceAudienceBrand positionMessage / toneVisual style directionContent strategy
30
Data Storytelling Opens the Door to Bigger (!) Creative Possibilities
FULL BLEED ANCHORS
MARGINS
MARGINS
TOP Emails 2016
31
FULL BLEED ANCHORS
MARGINS
MARGINS
headline
CTA
headline
CTA
copy
1
2
3
4
5
CYBER MONDAYDelivered 42% more clicksthan average
And a 24% higher click-to-open rate than average
FULL BLEED ANCHORS
MARGINS
MARGINS
32
FULL BLEED ANCHORS
MARGINS
MARGINS
NAKED CHICKEN CHALUPABy placing the secondary messagefirst, we gained +10% total clicksand 2x higher conversions.
34
To Recap
FULL BLEED ANCHORS
MARGINS
MARGINS
35
FULL BLEED ANCHORS
MARGINS
MARGINS
TO RECAPData is cool
36
FULL BLEED ANCHORS
MARGINS
MARGINS
TO RECAPFirst ask WHY, then WHAT
37
FULL BLEED ANCHORS
MARGINS
MARGINS
TO RECAPTell stories, not reports
38
FULL BLEED ANCHORS
MARGINS
MARGINS
TO RECAPCollaborate!
39
FULL BLEED ANCHORS
MARGINS
MARGINS
TO RECAPInformed creative > Pretty things
40
Any questions?
FULL BLEED ANCHORS
MARGINS
MARGINS