churn prediction in mobile social games: towards a ...yokozunadata.com/research/churnslides.pdf ·...
TRANSCRIPT
![Page 1: Churn Prediction in Mobile Social Games: Towards a ...yokozunadata.com/research/churnSlides.pdf · Churn prediction in Free-To-Play games We focus on the top spenders: the whales](https://reader030.vdocuments.us/reader030/viewer/2022040515/5e71b1fefde9247d465719e7/html5/thumbnails/1.jpg)
Churn Prediction inMobile Social Games: Towards a Complete Assessment Using Survival Ensembles
1
África Periáñez, Alain Saas, Anna Guitart and Colin MagneIEEE/ACM DSAA 2016Montreal, October 19th, 2016
![Page 2: Churn Prediction in Mobile Social Games: Towards a ...yokozunadata.com/research/churnSlides.pdf · Churn prediction in Free-To-Play games We focus on the top spenders: the whales](https://reader030.vdocuments.us/reader030/viewer/2022040515/5e71b1fefde9247d465719e7/html5/thumbnails/2.jpg)
About us
2
Who are we?● Game and technology company based in Tokyo (spin-off of
Silicon Graphics)
● Research project to provide Game Data Science as a Service
● Goals: predict player behavior, scale to big data and intuitive result visualization
![Page 3: Churn Prediction in Mobile Social Games: Towards a ...yokozunadata.com/research/churnSlides.pdf · Churn prediction in Free-To-Play games We focus on the top spenders: the whales](https://reader030.vdocuments.us/reader030/viewer/2022040515/5e71b1fefde9247d465719e7/html5/thumbnails/3.jpg)
3
● Free-to-play mobile social games● in-app purchases and activity behavioral data
Our data
![Page 4: Churn Prediction in Mobile Social Games: Towards a ...yokozunadata.com/research/churnSlides.pdf · Churn prediction in Free-To-Play games We focus on the top spenders: the whales](https://reader030.vdocuments.us/reader030/viewer/2022040515/5e71b1fefde9247d465719e7/html5/thumbnails/4.jpg)
4
Churn prediction in Free-To-Play games
We focus on the top spenders: the whales ➔ 0.2% of the players, 50 % of the revenues➔ Their high engagement make them more likely to answer positively to
action taken to retain them➔ For this group, we can define churn as 10 days of inactivity
◆ The definition of churn in F2P games is not straightforward
![Page 5: Churn Prediction in Mobile Social Games: Towards a ...yokozunadata.com/research/churnSlides.pdf · Churn prediction in Free-To-Play games We focus on the top spenders: the whales](https://reader030.vdocuments.us/reader030/viewer/2022040515/5e71b1fefde9247d465719e7/html5/thumbnails/5.jpg)
The modelSurvival Ensembles
5
![Page 6: Churn Prediction in Mobile Social Games: Towards a ...yokozunadata.com/research/churnSlides.pdf · Churn prediction in Free-To-Play games We focus on the top spenders: the whales](https://reader030.vdocuments.us/reader030/viewer/2022040515/5e71b1fefde9247d465719e7/html5/thumbnails/6.jpg)
Challenge: modeling churn
◎ Survival analysis focuses on predicting the time-to-event, e.g. churn○ when a player will stop playing?
◎ Classical methods, like regressions, are appropriate when all players have left the game
◎ Censoring Problem: dataset with incomplete churning information
◎ Censoring is the nature of churn
➔ Survival analysis is used in biology and medicine to deal with this problem
➔ Ensemble learning techniques provide high-class prediction results
6
![Page 7: Churn Prediction in Mobile Social Games: Towards a ...yokozunadata.com/research/churnSlides.pdf · Churn prediction in Free-To-Play games We focus on the top spenders: the whales](https://reader030.vdocuments.us/reader030/viewer/2022040515/5e71b1fefde9247d465719e7/html5/thumbnails/7.jpg)
◎ We focus on whales◎ Churn definition as 10 days of inactivity◎ Cumulative survival probability (Kaplan-Meier estimates) ◎ Step function that changes every time that a player churns
7
Output of the model
![Page 8: Churn Prediction in Mobile Social Games: Towards a ...yokozunadata.com/research/churnSlides.pdf · Churn prediction in Free-To-Play games We focus on the top spenders: the whales](https://reader030.vdocuments.us/reader030/viewer/2022040515/5e71b1fefde9247d465719e7/html5/thumbnails/8.jpg)
◎ Two approaches:○ Churn as a binary classification○ Churn as a censored data problem
◎ One model: Conditional Inference Survival Ensembles1 ○ deals with censoring ○ high accuracy due to ensemble learning
Survival Analysis
➔ Survival analysis methods (e.g. Cox regression) does not follow any particular statistical distribution: fitted from data
➔ Fixed link between output and features: efforts to model selection and evaluation
1) Hothorn et al., 2006. Unbiased recursive partitioning: A conditional inference framework 8
Challenge: modeling churn
![Page 9: Churn Prediction in Mobile Social Games: Towards a ...yokozunadata.com/research/churnSlides.pdf · Churn prediction in Free-To-Play games We focus on the top spenders: the whales](https://reader030.vdocuments.us/reader030/viewer/2022040515/5e71b1fefde9247d465719e7/html5/thumbnails/9.jpg)
Survival Tree➔ Split the feature space
recursively
➔ Based on survival statistical criterion the root node is divided in two daughter nodes
➔ Maximize the survival difference between nodes
➔ A single tree produces instability predictions
Conditional Survival Ensembles➔ Outstanding predictions
➔ Make use of hundreds of trees ➔ Conditional inference survival
ensemble use a Kaplan-Meier function as splitting criterion
➔ Overfit is not present
➔ Robust information about variable importance
➔ Not biased approach
9
Conditional inference survival ensembles
![Page 10: Churn Prediction in Mobile Social Games: Towards a ...yokozunadata.com/research/churnSlides.pdf · Churn prediction in Free-To-Play games We focus on the top spenders: the whales](https://reader030.vdocuments.us/reader030/viewer/2022040515/5e71b1fefde9247d465719e7/html5/thumbnails/10.jpg)
Conditional inference survival tree partition with Kaplan-Meier estimates of the survival time which characterizes the players placed in every terminal node group
10
Linear rank statistics as splitting criterion
Survival tree
![Page 11: Churn Prediction in Mobile Social Games: Towards a ...yokozunadata.com/research/churnSlides.pdf · Churn prediction in Free-To-Play games We focus on the top spenders: the whales](https://reader030.vdocuments.us/reader030/viewer/2022040515/5e71b1fefde9247d465719e7/html5/thumbnails/11.jpg)
◎ Two steps algorithm:
○ 1) the optimal split variable is selected: association between covariates and response
○ 2) the optimal split point is determined by comparing two sample linear statistics for all possible partitions of the split variable
Random Survival Forest
➔ RSF is based on original random forest algorithm1
➔ RSF favors variables with many possible split points over variables with fewer
111) Breiman L. 2001. Random Forests.
Conditional inference survival ensembles
![Page 12: Churn Prediction in Mobile Social Games: Towards a ...yokozunadata.com/research/churnSlides.pdf · Churn prediction in Free-To-Play games We focus on the top spenders: the whales](https://reader030.vdocuments.us/reader030/viewer/2022040515/5e71b1fefde9247d465719e7/html5/thumbnails/12.jpg)
Features selection◎ Game independent features:
○ player attention:
● time spent per day
○ player loyalty :
● number of days connecting (loyalty index)
● days from registration to first purchase
● days since last purchase
○ player intensity:
● number of actions, sessions, etc.
● amount in-app purchases
◎ Game dependent features:● player level: (concept common to most games)
12
![Page 13: Churn Prediction in Mobile Social Games: Towards a ...yokozunadata.com/research/churnSlides.pdf · Churn prediction in Free-To-Play games We focus on the top spenders: the whales](https://reader030.vdocuments.us/reader030/viewer/2022040515/5e71b1fefde9247d465719e7/html5/thumbnails/13.jpg)
Features selection
◎ Game independent features:
○ player attention: time spent per day, lifetime
○ player loyalty : number of days connecting, loyalty index (number of days played over lifetime), days from registration to first purchase, days since last purchase
○ player intensity: number of actions, sessions, amount in-app purchases, action activity distance (total average actions compared to last days behaviour)
○ player level: concept common to most games)
◎ Game dependent features researched but ultimately not part of our model:
○ participation in a guild (social feature)
○ actions measured by categories
13
![Page 14: Churn Prediction in Mobile Social Games: Towards a ...yokozunadata.com/research/churnSlides.pdf · Churn prediction in Free-To-Play games We focus on the top spenders: the whales](https://reader030.vdocuments.us/reader030/viewer/2022040515/5e71b1fefde9247d465719e7/html5/thumbnails/14.jpg)
The ResultsWith “Age of Ishtaria” Game Data
14
![Page 15: Churn Prediction in Mobile Social Games: Towards a ...yokozunadata.com/research/churnSlides.pdf · Churn prediction in Free-To-Play games We focus on the top spenders: the whales](https://reader030.vdocuments.us/reader030/viewer/2022040515/5e71b1fefde9247d465719e7/html5/thumbnails/15.jpg)
15
Binary classification results and comparison with other models
![Page 16: Churn Prediction in Mobile Social Games: Towards a ...yokozunadata.com/research/churnSlides.pdf · Churn prediction in Free-To-Play games We focus on the top spenders: the whales](https://reader030.vdocuments.us/reader030/viewer/2022040515/5e71b1fefde9247d465719e7/html5/thumbnails/16.jpg)
16
Predicted Kaplan-Meier survival curves as a function of time (days) for new or existing players
Censored data problem results
![Page 17: Churn Prediction in Mobile Social Games: Towards a ...yokozunadata.com/research/churnSlides.pdf · Churn prediction in Free-To-Play games We focus on the top spenders: the whales](https://reader030.vdocuments.us/reader030/viewer/2022040515/5e71b1fefde9247d465719e7/html5/thumbnails/17.jpg)
17
Validation -- Churn prediction
![Page 18: Churn Prediction in Mobile Social Games: Towards a ...yokozunadata.com/research/churnSlides.pdf · Churn prediction in Free-To-Play games We focus on the top spenders: the whales](https://reader030.vdocuments.us/reader030/viewer/2022040515/5e71b1fefde9247d465719e7/html5/thumbnails/18.jpg)
18
Validation -- Churn prediction
1000 bootstrap cross-validation error curves for the survival ensemble model and Cox regression
![Page 19: Churn Prediction in Mobile Social Games: Towards a ...yokozunadata.com/research/churnSlides.pdf · Churn prediction in Free-To-Play games We focus on the top spenders: the whales](https://reader030.vdocuments.us/reader030/viewer/2022040515/5e71b1fefde9247d465719e7/html5/thumbnails/19.jpg)
◎ Censoring problem is the right approach○ the median survival time, i.e. time when the percentage of
surviving in the game is 50%, can be used as a time threshold to categorize a player in the risk of churning
◎ Binary problem -- static model○ also bring relevant information○ useful insight for a short-term prediction
◎ SVM, ANN, Decision Trees, etc. are useful tools for regression or classification problems.○ in their original form cannot handle with censored data○ 1) modification of algorithm or 2) transformation of the data
19
Survival ensembles approach
![Page 20: Churn Prediction in Mobile Social Games: Towards a ...yokozunadata.com/research/churnSlides.pdf · Churn prediction in Free-To-Play games We focus on the top spenders: the whales](https://reader030.vdocuments.us/reader030/viewer/2022040515/5e71b1fefde9247d465719e7/html5/thumbnails/20.jpg)
◎ Application of state-of-the-art algorithm “conditional inference survival ensembles” ○ to predict churn ○ and survival probability of players in social games
◎ Model able to make predictions every day in operational environment
◎ adapts to other game data: Democratize Game Data Science
◎ relevant information about whales behaviour ○ discovering new playing patterns as a function of time○ classifying gamers by risk factors of survival experience
◎ Step towards the challenging goal of the comprehensive understanding of players
20
Summary and conclusion
![Page 21: Churn Prediction in Mobile Social Games: Towards a ...yokozunadata.com/research/churnSlides.pdf · Churn prediction in Free-To-Play games We focus on the top spenders: the whales](https://reader030.vdocuments.us/reader030/viewer/2022040515/5e71b1fefde9247d465719e7/html5/thumbnails/21.jpg)
21
Other work related to Game Data Science
Discovering Playing Patterns:Time Series Clustering of Free-To-Play Game DataAlain Saas, Anna Guitart and África PeriáñezIEEE CIG 2016
Special Session on Game Data ScienceChaired by Alain Saas and África PeriáñezIEEE/ACM DSAA 2016www.gamedatascience.org