widemile and microsoft multivariate testing case study
DESCRIPTION
A joint presentation at DMDays09 New York, showcasing a case study between Widemile and Microsoft Office Live Small Business, which through Widemile's multivariate testing platform improved Microsoft's online conversions by 40%.TRANSCRIPT
![Page 1: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/1.jpg)
Microsoft Case Study:
40% Conversion Rate Increase of SEM Traffic using
Multivariate Testing
Michael Schultz Director Product Management & Marketing, Microsoft Office Unmanaged Services
Frans KeylardDirector of Optimization, Widemile
![Page 2: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/2.jpg)
Agenda
• A/B & Multivariate Testing Overview
• Microsoft Office Live Small Business Case Study
– Business Overview
– Case Study Details
• Top 10 Page Optimization Tips
![Page 3: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/3.jpg)
Cus
tom
er
Sal
es F
unne
l
• Email marketing• Loyalty Programs• Newsletters• Personalization• Coupons /Surveys
Widemile Focus
• Key web pages• Sales funnels• Landing pages• Registration pages
• Search marketing• Display advertising• Affiliate marketing• Contextual ads• Ad targeting
Conversion Optimization
Conversion
Acquisition
Retention
*Source: eMarketer – April 2009
Investment recommendation:5% to 15% of acquisition budget
Testing and segmentation can improve retention performance
Automated, data-driven action that increases revenue by optimizing a user’s online experience.
$37 Billion annual US spend by 2013. $23 Billion spent in 2008.*
![Page 4: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/4.jpg)
Testing Options – Approach Overview
Basic Analytics
Ideal use: > Validates traffic and conversion rates
Limitation: > Simple measurement; no optimization
A/B Split Testing
Ideal use: > Best for testing two or more radically different pages <i.e. template testing>
Limitation: > No insight into why one page is better, just which won among traffic tested
Multivariate Testing (MVT)
Ideal use: > Test different content, simultaneously determining most effective combination> Segmentation can be applied for more granular optimization in follow-on tests
Limitation: > Requires sufficient traffic & conversions
![Page 5: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/5.jpg)
MVT Testing Options - Full & Fractional Factorial
Fractional FactorialFull Factorial
5
0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170
Full Factorial
Fractional Factorial
Number of Days to Complete Test
![Page 6: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/6.jpg)
Expertise
Requirements for Success
6
User Interface Proven Architecture Reporting Interface
Analyst Creative Web Developer
Technology
![Page 7: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/7.jpg)
2009 – Year for Conversion Optimization
![Page 8: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/8.jpg)
Microsoft Office Live Small Business
Audience:
• Small businesses, 1-10 employees(real estate agent, local bakery, video store, restaurant)
• Non-profits/Organizations(community groups, soccer clubs, PTSAs)
Benefits:
• Free website and email
• Free services to view, store and share documents online
![Page 9: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/9.jpg)
Goals Summary
Goals:
• Improve sign up conversions
• Better understand audience
• Identify highest impact page factors
Testing Strategy:
• Leverage proven landing page template
• Execute multivariate test
• Fractional factorial approach to minimize time
![Page 10: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/10.jpg)
Microsoft Case Study
Getting started…1. Define page to be tested
2. Analyze outside factors
1. In-bound traffic
2. Traffic volume
3. UX - Funnel
3. Review page & conversion goals
Original web page
![Page 11: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/11.jpg)
Testing Considerations
What is the best hero shot?
Does the headline grab the customer’s attention?
Do these resource links belong here?
Does the button maximize conversions?
Does this copy help or hurt?
![Page 12: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/12.jpg)
Indentifying Factors to Test
1
2
3
4
5
![Page 13: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/13.jpg)
Defining a Test Matrix
No. Factor (Variable) Levels (versions)
1 Hero Shot 4
2 Title 4
3 Description 4
4 Call-To-Action Button 4
5 Resource Links 2
![Page 14: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/14.jpg)
Audience Participation – Best CTA Button
Level 2
Level 3
Level 4
Level 1
![Page 15: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/15.jpg)
Efficiency Through Technology
1 of 16 2 of 16
3 of 16 4 of 16
0 100 200 300 400 500
Full Factorial
Fractional Factorial
Comparison - # of Experiments Required for Test by Approach
512
16
![Page 16: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/16.jpg)
And the Winner is…
40%Improvement
![Page 17: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/17.jpg)
Which Factors Made the Biggest Difference
High Influence
Medium Influence
Low Influence
Hero Shot
Title
Description
Call-To-Action
Resource Link Location
![Page 18: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/18.jpg)
High Influence Factor - Key Learning
Factor/level insights
• “Get Started Free” won
• The free aspect of the service is very important to get people to try it out
• More task focused terminology like “Set up my business!” or “Build My Online Business” lost
• “Sign Up Free!” which is the typical text found on so many CTA buttons also lost out.
• Future testing around the positioning of the FREE service
![Page 19: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/19.jpg)
Case Study Summary
Microsoft Office Live Small Business’ testing and optimizing investment:
1. Delivered a 40% lift in conversion
2. Reduced customer acquisition cost
3. Maximized SEM performance
Testing and optimizing content is a fast and effective way to maximize the performance of your SEM spend.
![Page 20: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/20.jpg)
10 Optimization Tips
1. Offer Clarityo Make sure the offer is clear, relevant, and compelling
o Keep your pages simple, clear, and appealing
2. Path to Conversiono Make sure the path to conversion is simple and straightforward
o Keep conversion paths efficient, focused, uncluttered
3. Performanceo Prospects will not tolerate slow pages or links
o Google penalizes slow page loads
4. Test Design and User Experienceo Avoid confusing landing page designs and difficult to read text
o Make transitions seamless and keep the consistency high (both messaging and look & feel)
5. Seasonalityo Seasonally optimize your pages and PPC campaign
o Leverage historical SEM in planning current and future campaigns
20
![Page 21: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/21.jpg)
10 Optimization Tips
6. Target Audienceo Optimize your content based on audience insight
o Capture contact information early to better inform your marketing
7. Segment Audienceo Optimize for distinct audiences to convert
o Segment your traffic and test to see if unique landing pages would perform better
8. Offers and Goalso Create and test offers that have a high perceived value but a low cost to fulfill
o Force your page to have only one clear goal and link ads to landing pages
9. Consistencyo Build consistency and relevancy between PPC campaigns and where you send your traffic
o Reinforce the call to action/offer on order and lead forms
10. Competitiono Look at what your competition is doing, but also what they are not doing
o Differentiation can often help, so test new approaches, offers, and incentives
![Page 22: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/22.jpg)
4 Additional Tips for Marketers
1. Drink the Kool-Aid and start now!
2. Think of multivariate testing as a “fixed” marketing expense and allocate budget appropriately
3. Ensure collaboration between Creative, Optimization Testing and SEM
4. Make sure Optimization vendor learns the business and understands the audience
22
![Page 23: Widemile and Microsoft Multivariate Testing Case Study](https://reader035.vdocuments.us/reader035/viewer/2022062405/5557b890d8b42a696c8b4ff6/html5/thumbnails/23.jpg)
Questions?
PresentersFrans Keylard – [email protected]
Michael Schultz – [email protected]
Content Informationwww.widemile.com
www.officelive.com