video a/b testing deconstructed: what? how? where? (with real data and examples)

Video A/B Testing Deconstructed: What? How? Where?

(with REAL Data & Examples)

#WistiaFest / @acapland

Hey, I’m Andrew

• Lead the Growth team at Wistia

• Huge marketing & data nerd

• Surfer & snowboarder

• 90’s hip-hop fan

I like testing things

I testing things

We test things, like CTAs

And page layouts

And value props

And self-guided tours

Then we redesign and do it all again

testing worksBecause marketers are crazy

WHY aren’t marketers testing

with video?

Video is more compelling than static content

SCREEN SHOT OF MY TALK

That’s my post!

Video is a powerful tool that humanizes your brand, and compels

your viewers to take action

Marketers don’t understand the conversion power of video

Action Increase signups from website

Drive product activation Accelerate sales process

Decrease churn

Emotional Storytelling

Humanize brand Establish connections

Let people get to know you

Action Increase signups from website

Drive product activation Accelerate sales process

Decrease churn

You can easily test and measure these!

You need to be testing your videos

Two main ways to a/b test with video

Video content Video placement

#WistiaFest / @twitter-handle

1. Why should I test my videos?

2. Where should I test my video?

3. Video testing lessons from Wistia

Let’s dig in more:

1. Why should I test my videos?

This is the Wistia creative team

Danielle

Trevor

They make awesome things

Danielle

Trevor

Hey we’re redesigning our About Us page,

should it have a video?

Of course, let’s make it weird. Everyone

loves weird!

Oh yeah! That would be sick.

Let’s put it on the top of the page.

Let’s film it with our new lens. It’ll

look awesome. Oh, I‘m jacked up!

About Us page

80’s themed pizza party to celebrate!

#WistiaFest / @acaplandJAN FEB MAR APR MAY JUNE JULY

Conversion)Rate

Ruh roh

Conversion Rate

Was it this?

#WistiaFest / @acaplandJAN FEB MAR APR MAY JUNE JULY

Conversion)Rate

Too late

Test to avoid unexpected results

JAN FEB MAR APR MAY JUNE JULY

Conversion)Rate

More please

Test to understand what works

LEARN:

WHAT WORKSWHAT DOESN’T WORK

Don’t embed it and forget itSee Margot de Cunah’s talk

Don’t publish videos and just

hope for the best

We get 30k monthly visits to our page, but few signups. I wonder

how we could get more?

And what video might get us the most signups?

Hmmn. I wonder if a video would help?

We could film a few different versions? Oh, I am jacked up!

1. Set clear goals 2. Brainstorm 3. Rank 4. Execute (quickly) 5. Iterate

Sometimes your most boring ideas are the most effective

You need to be testing your videos

Learn wicked fast

2. Where should I test my videos?

Hint: not everywhere

Test in actionable places where you can learn

Also, somewhere that gets a decent amount of traffic

Product page(s) Email

About us page Pricing page

Onboarding flow Homepage

Optimizely Sample size calculator - wi.st/sizecalc

A/B confidence calculator - wi.st/statsig

How to run an a/b test

3. Lessons we learned from testing video at Wistia

ReminderThere are two main ways you can a/b test with video

Video content Video placement

1. Set clear goals 2. Brainstorm 3. Rank 4. Execute (quickly) 5. Iterate

Remember the system

Test 1: Welcome to Wistia page

Test 1: Is this the best video for our goal?

Test 1: Version A

Test 1: Welcome to Wistia page

Hypothesis: Teaching people about

the tool will increase account activations.

Test 1: What do you think?

8% increase in activations

Well, like duh!

Learnings

Straightforward educational content is more helpful

(for new users)

Test 1: Educational follow-up

Hypothesis: Highlighting product

differentiation will increase activations

Test 1: Follow-up

3% decrease in activations

Learnings

Teaching how to use tool more helpful than highlighting features

(for new users).

Test 2: Wistia product page

Test 2: Should this page have a video?

Test 2: Automated Emails

Test 2: Product Page

Hypothesis: Static content above the fold may be more

effective.

9% increase in signups!

Test 2: Learnings

New visitors may want to browse quickly on the

product page

Test 3: Onboarding flow

Test 3: Is this the best video?

Test 3: Onboarding flow

Hypothesis: Sharing how to use all

the features will increase action with

new users.

30% increase in usage!

Test 3: Learnings

Reinforced our “how to” learning. Showing better than telling.

Context is king.

You should be testing your videos

Everyone wins!

1. Set clear goals 2. Test to learn faster & understand impact 3. Test where it’s actionable (not everywhere) 4. Test your way to success

Use technology to find hidden gems

Collect data (Google Analytics to ID high traffic pages) Watch users (FullStory, Inspectlet)

Survey visitors (Qualaroo, Intercom)

Optimizely Sample size calculator - wi.st/sizecalc

A/B confidence calculator - wi.st/statsig

A/B Testing Resources

Thank you

Ask me questions anytime:

andrew@wistia.com @acapland

video a/b testing deconstructed: what? how? where? (with real data and examples)

Marketing

video page examples

infographics, social media and video examples

3g video real world examples

pop trends deconstructed - hit songs deconstructed · pdf...

still life, a deconstructed story

deconstructed goat cheese cheesecake & figs

coaching competencies deconstructed

deconstructed hitz

examples of video shots and composition

boy deconstructed

chancellor zeppos deconstructed

spirituality in health (-deconstructed)

the deconstructed broccoli gratin soup

deconstructed stolen poem

artificially deconstructed - cabinet, cage, stage

deconstructed websites

the core deconstructed - complimentary preview

multikulturalism deconstructed

dooh deconstructed

typography deconstructed