massively multiplayer data challenges in mobile game analytics
TRANSCRIPT
Massively Multiplayer Data: Challenges in Mobile Game
AnalyticsJak Marshall, Sega Hardlight
Take-aways of this talk
An introductory overview of a very modern, complex, and lucrative industry that doesn’t usually get a lot of press in academic circles.
Present challenges relating to free to play apps and games.
Discuss the current best practices in the industry.
Tempt both ardent academics and those considering industrial work that the games industry offers a lot of ‘low hanging fruit’ and real hardcore challenges also.
Talk about videogames! Ask me anything!
About Me (Fun Version)
Lifelong Gamer
Content Creator (Lunch Time Game Review)
Tabletop Gamer (board and card games, mostly)
Blogger (103% Complete Gaming Blog)
LU Comedy Society
Vegan Runner
About Me (Career)
(2006 - 2010): Msci Maths and Stats at Lancaster University
(2008): Study Abroad at UC Berkeley
(2010): Data Intern at Unilever (Next Generation Methods, Port Sunlight
(2010): First Cohort of Stor-i DTC MRes
(2011 - 2016): PhD (currently writing up corrections)
(2015): Exient Malta : Lead Games Data Analyst
(2016): Sega Hardlight : Games Data Analyst
Softography (that I can tell you about)
Sega Hardlight
Created in 2012, based in the West Midlands.
Leamington Spa (very close to Warwick, home of JLR and Aston Martin).
Owned by and a part of the Sega group of companies (Did I mention SONIC!?).
Speciality in Mobile Games Development.
Leamington Spa is also known as Silicon Spa for the surprisingly large amount of game developers and tech companies in the area.
This is because the Oliver Twins (Codemasters) set up there originally.
Games: Narrowing it down
Games: Narrowing it down
Mobile Games: Narrowing it down (by business plan)
Paid: Traditional, consumer pays an upfront cost and gets the complete game.
Subscription: Pay regular installments to have access to game. No upfront fee most of the time.
Free to Play: No upfront cost or subscription tiers, in-app purchases and in-app advertising drive revenue.
Mobile Games: Narrowing it down (by business plan)
Paid: Traditional, consumer pays an upfront cost and gets the complete game.
Subscription: Pay regular installments to have access to game. No upfront fee most of the time.
Free to Play: No upfront cost or subscription tiers, in-app purchases and in-app advertising drive revenue.
How does Free to Play work generally?
Economies of Scale
Since the price of entry is $0 (and sometimes incentivised!), the volume of users using the app is large.
Not just games either! Most of us adopted Dropbox, Spotify, Skype etc because it was free to install and access to most of the functionality of the product was given away for nothing.
Even if only 10% of users ever purchase additional features and add-ons, that’s 10% of a large volume of users, so this pricing model still covers costs, in theory.
Success for a mobile app
Some definitions:
Churned user: Someone who has ‘permanently’ stopped using the app.
Mean Lifetime value (LTV) of a user: The mean amount of $revenue that an individual user generates for the business by using the app and buying in-app goods before they churn forever.
Cost per Install (CPI): The mean cost of acquiring a single user.
What we want to see: User Volume * (LTV - CPI) > Overheads + Dev Costs
A business model built on miserly users!?
Yes! But even those who do not pay any of their own money still have real money value to the business.
1. They contribute to the virality of app, reducing the cost of user acquisition depending on the k-factor of that user (more on that later).
2. Passive and incentivised advertising costs nothing to the user but provides revenue to the business if consumed.
3. Their sheer numbers incentivise others to spend!
We’ll talk about ‘virality’ in some more detail/
What on earth is Virality/k-factor?
It’s actually very costly to get users to download your app/game!
$4 to get a loyal user on board.
Loyal doesn’t necessarily mean ‘paying customer’!
What on earth is Virality/k-factor?Virality for apps means that your current users acquire even more users for you.
Back in the ‘bad old days’ this meant that your Facebook feed was awash with invites to play Candy Crush Saga.
As annoying as that was, it was a monster success for King,
But how is Virality defined?
Open problem! One of many!
The k-factor method counts the average number of ‘free users’ you get from each existing user.
If k = 0, that means nobody is inviting their friends to the party. If k > 1, then you typically have a ‘viral game’ as the user base grows exponentially.
The Catch! -- Attribution is very hard and messy. How do you know why a user decided to join your game? Especially when companies ‘lie’ about it.
You can track invites and clicks, but you can’t track word-of-mouth and response to TV/Cinema ads very accurately! We needs better models for this...
Truth: Games Data Analytics is a Mess
1. Academic research in this area is very fragmented.
2. Many mobile studios are currently made up from former console or tech start-up talent.
1.Academic WorkAnalysts like myself don’t publish their work and we’re not unionised yet.
We ‘pull’ inspiration from the (free!) literature as we need it but we don’t ‘push back’. Not great links between academia and industry (Let’s change that?).
Operational pressures means that the industry can’t formalise its learning.
There are very few academic practitioners like there are in medicine and energy for example. There’s not a prominent ‘games data’ themed journal or portal.
This is very sad because game development, publishing and live operations offer a vast amount of potential projects for Stats, OR, and CS students and faculty.
Even sadder because I know how many of you are game nerds!
2a. Former console talent
A great proportion of mobile studios consist of former console devs, making ‘finished’ box products (Think Assassin’s Creed, Tomb Raider, Bok-tai)
Traditional console studios never included analysts that worked on the games themselves.
Analysts in the industry usually worked in publishing and market intelligence side of things out of studio.
The presence of analysts in a games studio is an idea which isn’t even a decade old at this point. Even big studios with centralised Business Intelligence Departments are flying by the seat of their pants at this point in time.
2b. Technology companies who also make games.
These companies are awash with very bright analysts, computer scientists and tech heads.
They’re ahead of the field in terms of solving a lot of complicated technical problems and optimising processes.
However, they lack any real pedigree in game design experience and user experience knowledge.
Probably not the kind of company that are interested in the ‘science of fun’ and more interested in simply grinding money out of people in a soulless kind of way.
They found that Mariah Carey gets more installs than Kate Upton through AB Testing though!
One more thing… you have to do this on the fly.You have to collect this data while your
game is running.
If you have a complex real-time tactical experience, you have to be able to grab everything you need while the game is happening, to a potentially huge concurrent roster of players on a server.
A large proportion of it will be available to sample at the time that the analytics events you want will fire. E.g. high-score at Game Over.
So now we get to what I do for a living!1. Providing regular ‘health checks’ on the games that are live for players right
now: Identifying risks and areas for improvement in our KPIs.
2. Overall data strategy: What we decide to track, how we track it and which technology we use. The studio look to me to provide that direction.
3. Insights: Deep dives, post mortems, market analysis… that’s on me too!
4. Optimisation: Experimenting on players to learn what makes them play for longer and spend more.
Let’s talk briefly about ‘passive reporting’1. Providing regular ‘health checks’ on the games that are live for players right
now: Identifying risks and areas for improvement in our KPIs.
2. Overall data strategy: What we decide to track, how we track it and which technology we use. The studio look to me to provide that direction.
3. Insights: Deep dives, post mortems, market analysis… that’s on me too!
4. Optimisation: Experimenting on players to learn what makes them play for longer and spend more.
Top level Reporting - Always available to all
Non-technical execs want to be able to look at ‘high-level stats’ without adult supervision.
This tends to be non-game specific for cross title comparisons
There’s a culture of ‘let’s track what everyone else is tracking’ but there’s certainly room for improvement here.
Trying to improve our workflow and toolchain.1. Providing regular ‘health checks’ on the games that are live for players right
now: Identifying risks and areas for improvement in our KPIs.
2. Overall data strategy: What we decide to track, how we track it and which technology we use. The studio look to me to provide that direction.
3. Insights: Deep dives, post mortems, market analysis… that’s on me too!
4. Optimisation: Experimenting on players to learn what makes them play for longer and spend more.
The tech I’m using.
It’s also a matter of evangelising the need to think about analytics from design to servicing.
‘Big Ticket’ Analysis - Deep Dive Reporting.1. Providing regular ‘health checks’ on the games that are live for players right
now: Identifying risks and areas for improvement in our KPIs.
2. Overall data strategy: What we decide to track, how we track it and which technology we use. The studio look to me to provide that direction.
3. Insights: Deep dives, post mortems, market analysis… that’s on me too!
4. Optimisation: Experimenting on players to learn what makes them play for longer and spend more.
5. Systems Design: Not my core remit, but I’m often drafted in to balance the economies and metagame of our more complex titles.
Example of Recent Deep Dive: Churn Detection.What are the key predictors of churn? (leaving a game, never to return)
It largely depends on the game and the audience.
Took a lot of inspiration from some pioneer’s on Kaggle working in World of Warcraft.
https://www.kaggle.com/thibalbo/d/mylesoneill/warcraft-avatar-history/wow-dataset-exploratory-analysis
It’s important that we can predict which of our users are at risk of leaving us so we can decide when and how to intervene.
What we’ve found so far.
Social obligation reduces churn: Online ‘teams’ can create a sense of loyalty.
Social proof also reduces churn: Just seeing that your friends are playing puts you at lower risk.
Changepoints in login streak: If engagement starts ‘to wobble’, there’s a good chance it will fall off altogether.
Seasonal: There’s definite ‘churnfests’ as seasons change, particularly the end of school holidays, the start of academic terms, and the week after New Years.
But how do we intervene!?1. Providing regular ‘health checks’ on the games that are live for players right
now: Identifying risks and areas for improvement in our KPIs.
2. Overall data strategy: What we decide to track, how we track it and which technology we use. The studio look to me to provide that direction.
3. Insights: Deep dives, post mortems, market analysis… that’s on me too!
4. Optimisation: Experimenting on players to learn what makes them play for longer and spend more.
AB Testing Approach (Boo!)It seems to be the industry standard.
You randomly allocate users into separate groups and then change the game experience for people in that group.|
Candy Crush changes the order and difficulty of its levels all the time to optimise the trade off between selling extra moves and people getting fed up with it.
It’s slow, inefficient, and doesn’t control for an awful lot of things. It’s simple.
It’s also tricky when you have multiplayer games and clever clogs on forums who spot that things are a bit different.
Bayes Optimisation: The Multi-Armed Bandit FutureYou want to expose the fewest number of users as
possible to potential variants of the game experience while still learning about the impact of potential changes you want to make.
It’s very expensive to lose players, particularly if they are influential spenders
Bandits seem to be the best way to balance the delivery of product insights and the overheads associated with doing so.
Gaming is behind the curve on this front, as it is with a lot of its data practice, so the service providers that deliver Bayesian versions of ‘off-shelf’ tools and backend services stand to do well. (I’m doing my best from the front lines!)
The vital take-aways from all of this.
Mobile Gaming remains a growth industry with plenty of money flying around.
Large outfits such as Sega, Nintendo, and even King are far from yielding the additional revenue to be gained from the proper application of analytics.
If you love games and you want to help make them, you can try and make these changes from the belly of the beast like I’m doing.
If you’re more academically inclined there’s a whole lot of interesting problems and applications up for grabs if we can establish good links between publishers/studios and academic institutions!
