a step-by-step guide to a/b testing research
TRANSCRIPT
![Page 1: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/1.jpg)
A Step-By-Step Guide to
A/B Testing ResearchSIDDHARTH DESWAL
![Page 2: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/2.jpg)
Thue is the Kissmetrics Webinar Wizard and Marketing Ops Manager. Before joining forces with Kissmetrics, he was a Lyft driver in SF,
which is also how he ended up as a KISSmetrics marketer. Whenever Thue is not trying to automate everything around him, you can find him
hiking in the Sierras.
THUE MADSENMarketing Operations Manager, Kissmetrics
@ThueLMadsen
Siddharth runs demand generation at VWO. In addition to being interested in all things CRO, he’s also passionate about B2B SaaS
marketing, Sales, growth hacking and human resource management.
SIDDHARTH DESWALSenior Marketer, Visual Website Optimizer
@SiddharthDeswal
![Page 3: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/3.jpg)
@NEILPATEL #KissWebinar @Kissmetrics
#KissWebinar
@SiddharthDeswal
![Page 4: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/4.jpg)
1 Find your top optimization opportunities
2 Ask some questions
3 Easy usability testing
TABLE OF CONTENTS
4 Visual analysis
5 Build your hypotheses
5 Putting it all together
![Page 5: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/5.jpg)
WATCH WEBINAR RECORDING NOW
![Page 6: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/6.jpg)
Step 1 - Find Your Top Optimization Opportunities
![Page 7: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/7.jpg)
LANDING PAGES WITH HIGH BOUNCE
![Page 8: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/8.jpg)
FUNNEL PAGES WITH HIGH EXIT
Image source
![Page 9: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/9.jpg)
Start with your most valuable traffic sources
ANALYTICS - TOP TRAFFIC SOURCE
![Page 10: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/10.jpg)
Landing pages with high bounce and high entries are optimization gold
ANALYTICS - MOST VALUABLE ENTRY POINTS
![Page 11: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/11.jpg)
Landing pages with high bounce and high entries are optimization gold
ANALYTICS - MOST VALUABLE ENTRY POINTS
![Page 12: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/12.jpg)
Funnel pages with high exit are the holes in the leaking bucket
ANALYTICS - FUNNEL DROP OFF
![Page 13: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/13.jpg)
ANALYTICS - FUNNEL DROP OFF
![Page 14: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/14.jpg)
LOG IN WITH GOOGLE
Start Your Free Kissmetrics Trial
![Page 15: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/15.jpg)
Segment Your Traffic For Accurate Analysis
![Page 16: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/16.jpg)
USEFUL SEGMENTS - NEW VISITORS WHO DIDN’T BOUNCE
![Page 17: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/17.jpg)
USEFUL SEGMENTS - BEHAVIOR SIMILAR TO CONVERTERS
Create a segment about your goal converters (in our case, Free Trial signups)
![Page 18: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/18.jpg)
USEFUL SEGMENTS - BEHAVIOR SIMILAR TO CONVERTERS
Map their key metrics
![Page 19: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/19.jpg)
USEFUL SEGMENTS - BEHAVIOR SIMILAR TO CONVERTERS
Create a segment to see those who behaved like goal converters but did not convert
![Page 20: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/20.jpg)
USEFUL SEGMENTS - BEHAVIOR SIMILAR TO CONVERTERS
You can use this segment in multiple ways1. Retarget them with a discount or promotional offer2. Give them higher lead scores in your marketing automation system3. Show them a right hand side pop-up4. If you have their email, trigger a specific message if they don’t convert
and leave the website5. If they’ve opted in to browser push-notifications, trigger a message
![Page 21: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/21.jpg)
Step 2 - Ask Some Questions
![Page 22: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/22.jpg)
ON-SITE SURVEYS
![Page 23: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/23.jpg)
ON-SITE SURVEYS - SAAS
Page Question Insights drawn
Pricing / Features / Trial-Signup
What’s stopping you from signing up with us?
Common concerns, objections, anxieties
Top exit pages Is there something you were looking for that you couldn’t find on the website?
Reasons that users leave, Information that they needed to make a decision but couldn’t find
Top landing pages / Homepage
How familiar are you with [business offering]?
Understand how much convincing/explaining do your visitors need
Top landing pages / Homepage
Where did you come to know of us?
Top marketing channels
![Page 24: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/24.jpg)
ON-SITE SURVEYS - SAAS
![Page 25: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/25.jpg)
ON-SITE SURVEYS - ECOMMERCE
Page Question Insights drawn
Product page What’s stopping you from adding this product to cart?
Common concerns, objections, anxieties or ‘just looking’
Top exit pages (product or category)
Is there something you were looking for that you couldn’t find on the website?
Reasons that users leave, Information that they needed to make a decision but couldn’t find
Top landing pages / Homepage
Where did you come to know of us?
Top marketing channels
Checkout / Cart What’s stopping you from buying this product?
Why are visitors not completing purchases in spite of adding to cart
![Page 26: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/26.jpg)
Step 3 - Easy Usability Testing
![Page 27: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/27.jpg)
EASY USABILITY TESTING
Putting 5 to 10 users in front of your website or landing pages and observing them (or hearing them vocalize their thoughts) can teach you far more than most other forms of research.
Caveat: these users have to be representative of your target customers.
![Page 28: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/28.jpg)
USABILITY TESTING - ASKING THE RIGHT QUESTIONS
“You received a damaged product, try and return it”
vs.
“You received a damaged product, what do you do now?”
![Page 29: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/29.jpg)
USABILITY TESTING - ASKING THE RIGHT QUESTIONS
“You received a damaged product, try and return it”
vs.
“You received a damaged product, what do you do now?”
1) User searches for ‘Returns Policy’ page and then proceeds to follow the steps listed
![Page 30: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/30.jpg)
USABILITY TESTING - ASKING THE RIGHT QUESTIONS
“You received a damaged product, try and return it”
vs.
“You received a damaged product, what do you do now?”
1) User searches for ‘Returns Policy’ page and then proceeds to follow the steps listed
1) User opens ‘Live Chat’ and asks rep what to do, or2) Takes to Twitter to complain to the brand, or3) Looks for the ‘Returns’ page, or4) Something that none of us were able to anticipate
![Page 31: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/31.jpg)
EASY USABILITY TESTING
Choose your usability tasks/problems based on these parameters:1. Do you want to understand problems in a specific, goal driven process,
or2. Do you want to understand how users approach a problem they are
looking to solve?
![Page 32: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/32.jpg)
Step 4 - Visual Analysis
![Page 33: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/33.jpg)
HEATMAPS
Heatmaps can reveal some surprising insights
![Page 34: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/34.jpg)
HEATMAPS
More people clicked on the top-right “Free Trial” CTA instead of the primary, middle-of-page CTA
![Page 35: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/35.jpg)
HEATMAPS
What is it that we want our visitors to do, versus what are they doing instead?
The goal should be to align our interests and theirs’ to a common outcome.
Heatmaps quickly show you the difference.
![Page 36: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/36.jpg)
By now, problems on your pages should have ‘bubbled’ out.
![Page 37: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/37.jpg)
Step 5 - Hypothesis
![Page 38: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/38.jpg)
HOW TO CREATE GREAT HYPOTHESIS
Research
Hypothesis
![Page 39: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/39.jpg)
HOW TO CREATE GREAT HYPOTHESIS
Image source: MarketingExperiments.com
![Page 40: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/40.jpg)
HOW TO CREATE GREAT HYPOTHESIS
The Change: Effect model depends on
1. Presumed problem
2. Proposed solution
3. Anticipated result
Image source: MarketingExperiments.com
![Page 41: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/41.jpg)
Putting It Together
![Page 42: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/42.jpg)
PUTTING IT ALL TOGETHER
1. Start by analyzing your top traffic source2. Look at the landing pages that have high traffic and relatively higher
bounce3. Look at pages that have the highest exit rates in a funnel4. Segment your traffic to get more accurate insights (goal converters
vs. non converters, new visitors who didn’t bounce)5. Run on-site surveys on your important landing or funnel pages6. Ask live users to complete tasks or solve problems that involve these
pages7. Run visual analysis on your most important pages8. At the end of this process, issues with the pages should bubble
out, i.e. they will become obvious9. Create hypotheses and start testing!
![Page 43: A Step-By-Step Guide to A/B Testing Research](https://reader031.vdocuments.us/reader031/viewer/2022030317/586fddb51a28ab18428b68d5/html5/thumbnails/43.jpg)
SIDDHARTH DESWALSenior Marketer, Visual Website Optimizer
THUE MADSENMarketing Operations Manager, Kissmetrics
Questions?