product madness - a/b testing

GIAF 3 OCTOBER 2013A/B TESTS

andy.toben@productmadness.com

What is an A/B test ?“A/B testing is a methodology in advertising of using randomized experiments with two variants, A and B, which are the control and treatment in the controlled experiment.”

- wikipedia.org

What Can We A/B Test?Basically anything:• Page Colour • Layout• Call to action • Images• If you can change it, you can test it!

Step by Step OptimizationTesting one variable at the time works, but can create issues:

- Time consuming:Needs a lot of traffic or data

- Local verses Global Optimum:Can lead to accepting the best variant for that test, but it is not optimised against all possible variants

Multi Variant TestingThis allows you to test all combinations at once however:

- Requires massive data sets – much more than Step by Step

- Requires mathematical tools

- No intuitive / building insights

How do we know who’s winning ?

How do we know who’s winning ?

Advertising is a game, so how about games ?

Strategies

What can we use A/B testing to discover?

• Search - Option A / Option B• Optimize - Option A / Option A+ / Option A-• Change management - treatment verses control• Measure - No treatment verses treatment

A/B Testing User ExperienceWhen A/B testing user experience you can test out the following and create a big impact… for better or worse!

- Order of game levels / features- Bonus system- Available content- P2P assignment - Pricing and Economy

Challenges We FaceSome A/B tests are very expensive - You may need to commit development or art resource for something that may not work.

Variant assignment needs to persistent  - In some cases you can’t just change a player’s track to the

“winning” branch  What to measure:

• Responsiveness (CTR)• Engagement• Retention • Monetization

Branching user experience

Understanding Results

Feature cost 1st day retention 7th day retention User value

10,000 coins 37% 10% 10 USD

25,000 coins 36% 6% 12 USD

Which metric is the best one to look at, and which result should we take action on?

But after all, It’s just a tool

Allocate Variant

No Change

No

Should we test the User?

User

NoIs there an

active Experiment?

Yes

Yes

Questions ?

andy.toben@productmadness.com