how experiments drive product growth at viki

38
How Experiments drive product at Viki Ishan Agrawal Software Engineer, Data Viki

Upload: ishanagrawal90

Post on 07-Jul-2015

343 views

Category:

Data & Analytics


0 download

DESCRIPTION

We extensively experiment with all aspects of product at Viki. These experiments take the form of AB testing, cohort analysis and various customisations. We collect and analyse the data on how these experiments affect user behaviour and other metrics. We are always in a continuous product improvement cycle heavily influenced by these experiments and corresponding data. We have developed our own custom Experimentation Framework called Turing for this purpose. In this talk I will explain how we carry out these experiments at Viki. I will also talk about Turing, why and how we built it.

TRANSCRIPT

Page 1: How experiments drive product growth at Viki

How  Experiments  drive  product  at  Viki

Ishan Agrawal

Software Engineer, Data Viki

Page 2: How experiments drive product growth at Viki

Why  Experiment?

Page 3: How experiments drive product growth at Viki

What  is  AB  testing?

Page 4: How experiments drive product growth at Viki

How  to  AB  test?

Page 5: How experiments drive product growth at Viki

What  can  you  test?

Anything that affects user behavior

Page 6: How experiments drive product growth at Viki
Page 7: How experiments drive product growth at Viki

To  build  or  not  to  build…

More control

Our own Data

Page 8: How experiments drive product growth at Viki

…  in  walks  Turing Our own Experimentation framework built in Golang

and Rails!

Page 9: How experiments drive product growth at Viki

The  Challenges

Flexibility Speed Scalability

User  Partitioning Stickiness

Page 10: How experiments drive product growth at Viki

Architecture

Page 11: How experiments drive product growth at Viki

How  to  make  it  fast  and  furious!

•  Api is in Golang

•  Store everything in-memory hashes (timely syncs)

•  No external calls (not even a redis cache)

~  300  micro  seconds

Page 12: How experiments drive product growth at Viki

Advantages  of  using  Go  lang •  Run as a binary anywhere!

•  Static Type checking

•  A good trade-off between performance and development speed

Page 13: How experiments drive product growth at Viki

Stickiness  and  User  partitions  without  a  cache?

Page 14: How experiments drive product growth at Viki

Points  to  note

•  Server side determination

•  Clients log the experiments tied to the current session.

•  A user is under only 1 experiment at a time

Page 15: How experiments drive product growth at Viki

So  how  do  you  create  experiments?

Features and Variations on them

Page 16: How experiments drive product growth at Viki

Feature  and  Variations  Example

Feature: Homepage Variations: Default : { version : 0 }! Setting1 : { version : 1 }!conditions: { country : ‘us’}!

Experiment1 : {version : 2}!

!!conditions: { country : ‘sg’, start_time : today, end_time : tomorrow}!!

Experiment2 : {version : 0}!!conditions: { country : ‘sg’, start_time : today, end_time : tomorrow}!!

Page 17: How experiments drive product growth at Viki

The  magic! •  Settings

o  Power of customizations

o  Different result for everyone •  Country •  Regions •  User type •  Time Ranges •  Application versions

Page 18: How experiments drive product growth at Viki
Page 19: How experiments drive product growth at Viki
Page 20: How experiments drive product growth at Viki

Data  Collection  and  Analysis

Page 21: How experiments drive product growth at Viki

Making  it  faster

Each release version is tagged with the enabled features.

Page 22: How experiments drive product growth at Viki

Sometimes  users  define  the  product…

•  Feature toggling

•  Feature tweaking / improvement (with data feedback)

•  Feature customization

•  Slow feature roll-outs •  In app messages

Page 23: How experiments drive product growth at Viki

Some  Experiments  we  ran…

Page 24: How experiments drive product growth at Viki

Viki  Pass  Landing  Page(1)

Page 25: How experiments drive product growth at Viki

Viki  Pass  Landing  Page(2)

Page 26: How experiments drive product growth at Viki

Viki  Pass  Landing  Page(3)

Page 27: How experiments drive product growth at Viki

And  the  winner  is…

Page 28: How experiments drive product growth at Viki

Home  page  experiments

Page 29: How experiments drive product growth at Viki

Home  page  experiments

20%

Conversion  Rate

Page 30: How experiments drive product growth at Viki

Ads  optimization

Page 31: How experiments drive product growth at Viki

Ads  optimization

Page 32: How experiments drive product growth at Viki

Ads  optimization

Page 33: How experiments drive product growth at Viki

We  are  still  figuring  it  out..

Page 34: How experiments drive product growth at Viki

     

Thanks!  @ishanagr  

[email protected]        

Any  questions?

Page 35: How experiments drive product growth at Viki

Appendix  

Page 36: How experiments drive product growth at Viki

In-­‐‑app  messaging

Page 37: How experiments drive product growth at Viki

AB  testing  Favorite  CTA

Page 38: How experiments drive product growth at Viki

AB  testing  like  buYon  CTA