build 2016 - p510 - a/b testing for uwp apps: experiment for success

#Build2016

A/B Testing for UWP Apps:Experiment for SuccessJason WatsonSr Program Manager, Windows Dev Center

A/B Testing ScenariosTest variations of your app to discover optimizations that boost conversions and engagement:In-app offersAd placementsUser flowSign-up experienceGame level configuration

Make data-driven decisions about app changes

ExampleCan I boost conversions by changing in-app offer text?Would “Disable Ads Now!” encourage more clicks?My conversion rate is 9% today....

11% conversion rate

45% conversion rate

4514 users (Variation A)

4512 users (Variation B)

Winning Variation

11% conversion rate

45% conversion rate

4514 users (Variation A)

4512 users (Variation B)

Winning Variation

45% conversion rate

9026 users (Variation B)

Winning Variation

How does it work?Update code in your app once to accommodate future experimentsAdd reference to the Microsoft Store Engagement and Monetization SDKBuild app logic around A/B testable settingsCreate logging events for user views and conversions

Use Windows Dev Center to run experimentsRun experiments to identify winning app variationsSwitch your audience to winning app variations

Demo #1: Update App Code for ExperimentationJason Watson

Demo #2: Act on A/B Test ResultsJason Watson

AdvantagesExperiment with your audience using one code base

Identify app variations that boost conversions

Switch your audience to a variation in two clicks

Best PracticesMaximize your test sample sizesUse split distributions for variation assignmentsLimit your experiment to two variations, including the control group

Run experiments for at least 2 weeksDo peek at data after 1 day, but don’t peek afterwards Allow for time to collect statistically significant dataData from specific time of day can skew results (ex. Saturday mornings)

• Click on Experimentation in Windows Dev Center to get started.

• Download the Microsoft Store Engagement and Monetization SDK: http://aka.ms/store-em-sdk

• Re-visit Build on Channel 9.• Continue your education at

Microsoft Virtual Academy online.

Call to Action

© 2016 Microsoft Corporation. All rights reserved.