video a/b testing deconstructed: what? how? where? (with real data and examples)
TRANSCRIPT
Video A/B Testing Deconstructed: What? How? Where?
(with REAL Data & Examples)
#WistiaFest / @acapland
#WistiaFest / @acapland
Hey, I’m Andrew
• Lead the Growth team at Wistia
• Huge marketing & data nerd
• Surfer & snowboarder
• 90’s hip-hop fan
#WistiaFest / @acapland
I like testing things
#WistiaFest / @acapland
I testing things
#WistiaFest / @acapland
#WistiaFest / @acapland
We test things, like CTAs
#WistiaFest / @acapland
And page layouts
#WistiaFest / @acapland
And value props
#WistiaFest / @acapland
And self-guided tours
#WistiaFest / @acapland
Then we redesign and do it all again
#WistiaFest / @acapland
testing worksBecause marketers are crazy
#WistiaFest / @acapland
WHY aren’t marketers testing
with video?
#WistiaFest / @acapland
Video is more compelling than static content
#WistiaFest / @acapland
SCREEN SHOT OF MY TALK
#WistiaFest / @acapland
That’s my post!
ads
WHY?
#WistiaFest / @acapland
#WistiaFest / @acapland
Video is a powerful tool that humanizes your brand, and compels
your viewers to take action
Marketers don’t understand the conversion power of video
#WistiaFest / @acapland
Action Increase signups from website
Drive product activation Accelerate sales process
Decrease churn
Emotional Storytelling
Humanize brand Establish connections
Let people get to know you
#WistiaFest / @acapland
Action Increase signups from website
Drive product activation Accelerate sales process
Decrease churn
You can easily test and measure these!
#WistiaFest / @acapland
You need to be testing your videos
Two main ways to a/b test with video
#WistiaFest / @acapland
Video content Video placement
#WistiaFest / @twitter-handle
1. Why should I test my videos?
2. Where should I test my video?
3. Video testing lessons from Wistia
Let’s dig in more:
#WistiaFest / @acapland
1. Why should I test my videos?
#WistiaFest / @acapland
This is the Wistia creative team
Danielle
Dan
Liat
Trevor
Chris
#WistiaFest / @acapland
They make awesome things
Danielle
Dan
Liat
Trevor
Chris
#WistiaFest / @acapland
Hey we’re redesigning our About Us page,
should it have a video?
Of course, let’s make it weird. Everyone
loves weird!
Oh yeah! That would be sick.
Let’s put it on the top of the page.
Let’s film it with our new lens. It’ll
look awesome. Oh, I‘m jacked up!
#WistiaFest / @acapland
About Us page
#WistiaFest / @acapland
80’s themed pizza party to celebrate!
#WistiaFest / @acaplandJAN FEB MAR APR MAY JUNE JULY
Conversion)Rate
Ruh roh
Conversion Rate
#WistiaFest / @acapland
#WistiaFest / @acapland
Was it this?
#WistiaFest / @acaplandJAN FEB MAR APR MAY JUNE JULY
Conversion)Rate
Too late
Test to avoid unexpected results
#WistiaFest / @acapland
JAN FEB MAR APR MAY JUNE JULY
Conversion)Rate
More please
Test to understand what works
#WistiaFest / @acapland
LEARN:
WHAT WORKSWHAT DOESN’T WORK
#WistiaFest / @acapland
Don’t embed it and forget itSee Margot de Cunah’s talk
Don’t publish videos and just
hope for the best
#WistiaFest / @acapland
We get 30k monthly visits to our page, but few signups. I wonder
how we could get more?
And what video might get us the most signups?
Hmmn. I wonder if a video would help?
We could film a few different versions? Oh, I am jacked up!
#WistiaFest / @acapland
1. Set clear goals 2. Brainstorm 3. Rank 4. Execute (quickly) 5. Iterate
#WistiaFest / @acapland
Sometimes your most boring ideas are the most effective
#WistiaFest / @acapland
You need to be testing your videos
Learn wicked fast
#WistiaFest / @acapland
2. Where should I test my videos?
#WistiaFest / @acapland
Hint: not everywhere
#WistiaFest / @acapland
Test in actionable places where you can learn
#WistiaFest / @acapland
Also, somewhere that gets a decent amount of traffic
#WistiaFest / @acapland
Product page(s) Email
About us page Pricing page
Onboarding flow Homepage
#WistiaFest / @acapland
Optimizely Sample size calculator - wi.st/sizecalc
A/B confidence calculator - wi.st/statsig
How to run an a/b test
#WistiaFest / @acapland
3. Lessons we learned from testing video at Wistia
ReminderThere are two main ways you can a/b test with video
#WistiaFest / @acapland
Video content Video placement
#WistiaFest / @acapland
1. Set clear goals 2. Brainstorm 3. Rank 4. Execute (quickly) 5. Iterate
Remember the system
#WistiaFest / @acapland
Test 1: Welcome to Wistia page
#WistiaFest / @acapland
Test 1: Is this the best video for our goal?
#WistiaFest / @acapland
Test 1: Version A
#WistiaFest / @acapland
Test 1: Welcome to Wistia page
Hypothesis: Teaching people about
the tool will increase account activations.
#WistiaFest / @acapland
#WistiaFest / @acapland
Test 1: What do you think?
8% increase in activations
#WistiaFest / @acapland
Well, like duh!
#WistiaFest / @acapland
Learnings
Straightforward educational content is more helpful
(for new users)
#WistiaFest / @acapland
Test 1: Educational follow-up
Hypothesis: Highlighting product
differentiation will increase activations
#WistiaFest / @acapland
Test 1: Follow-up
#WistiaFest / @acapland
Test 1: What do you think?
3% decrease in activations
#WistiaFest / @acapland
Learnings
Teaching how to use tool more helpful than highlighting features
(for new users).
#WistiaFest / @acapland
Test 2: Wistia product page
#WistiaFest / @acapland
Test 2: Should this page have a video?
#WistiaFest / @acapland
Test 2: Automated Emails
#WistiaFest / @acapland
Test 2: Product Page
Hypothesis: Static content above the fold may be more
effective.
#WistiaFest / @acapland
Test 2: What do you think?
9% increase in signups!
#WistiaFest / @acapland
Test 2: Learnings
New visitors may want to browse quickly on the
product page
#WistiaFest / @acapland
Test 3: Onboarding flow
#WistiaFest / @acapland
Test 3: Is this the best video?
#WistiaFest / @acapland
Test 3: Onboarding flow
Hypothesis: Sharing how to use all
the features will increase action with
new users.
#WistiaFest / @acapland
#WistiaFest / @acapland
Test 3: What do you think?
30% increase in usage!
#WistiaFest / @acapland
Test 3: Learnings
Reinforced our “how to” learning. Showing better than telling.
Context is king.
#WistiaFest / @acapland
You should be testing your videos
Everyone wins!
#WistiaFest / @acapland
RECAP
1. Set clear goals 2. Test to learn faster & understand impact 3. Test where it’s actionable (not everywhere) 4. Test your way to success
#WistiaFest / @acapland
Use technology to find hidden gems
Collect data (Google Analytics to ID high traffic pages) Watch users (FullStory, Inspectlet)
Survey visitors (Qualaroo, Intercom)
#WistiaFest / @acapland
Optimizely Sample size calculator - wi.st/sizecalc
A/B confidence calculator - wi.st/statsig
A/B Testing Resources