driving business goals with recommender systems @ yac/m 2015
TRANSCRIPT
Driving business goals with Recommender Systems
Konstantin Savenkov, COO Bookmate
[email protected], http://bookmate.com
Target audience
4
B2C Services
B2B Recom- mender Platforms
run a pilot estimate costs and benefits
determine fair price or scale to
start with
PROFIT
determine value for potential
clients run a pilot
set fair pricing model
Agenda
• Recommender Systems: Academy, Technology, Business
• Recommender Systems for content discovery
• B2C Content Services: overview and business model
• Driving business goals with Recommender Systems
• customer acquisition cost • lifetime value • catalogue exploitation
• Bookmate – E-Contenta case
6
RS
$ $ $ $
Agenda
• Recommender Systems: Academy, Technology, Business
• Recommender Systems for content discovery
• B2C Content Services: overview and business model
• Driving business goals with Recommender Systems
• customer acquisition cost • lifetime value • catalogue exploitation
• Bookmate – E-Contenta case
7
RS
$ $ $ $
Academy vs. Tech vs. Business
8
How to improve performance by X%
How hard is to implement that?
A: T:
B: When gains match costs?
Evaluation of Recommender Systems
9
academy business offline evaluation
online evaluation
economic evaluation
• user behavior history
• RMSE • MAP • NDCG • etc.
• live users • actual UX • actual
inventory
• NDCG • CTR • funnels • response
time
• live users • actual UX • actual
inventory • business
model
• CAC • LTV • COGS • …PROFIT!
10
“It’s tempting, if the only tool you have is a hammer, to treat everything as a nail.”
* Recommender systems are cool, but they don’t substitute old good traffic quality, UX and pricing.
Abraham Maslow, The Psychology of Science, 1966
Agenda
• Recommender Systems: Academy, Technology, Business
• Recommender Systems for content discovery
• B2C Content Services: overview and business model
• Driving business goals with Recommender Systems
• customer acquisition cost • lifetime value • catalogue exploitation
• Bookmate – E-Contenta case
11
RS
$ $ $ $
Recommender Systems for Content Discovery
• preference elicitation
• hard to describe preferences in a textual form
• weak textual relevance
• limited catalogue
12
“I WANT TO READ SOMETHING…”
EVEN FOR BOOKS!
LOOKING FOR UNKNOWN UNKNOWNS
REGIONAL SEGMENTATION
Recommender Systems in the interface • Any place in the interface, when number of objects to
show exceeds available space
• Most of the interfaces are list-based
• Hence, order and size of the list can be defined by either personalized or non-personalized algorithm
• Explaining recommendations is a different topic
14
There is no “no recommender system” setting. If there’s “just something” or “popularity sorted”, that’s your RS. !
Agenda
• Recommender Systems: Academy, Technology, Business
• Recommender Systems for content discovery
• B2C Content Services: overview and business model
• Driving business goals with Recommender Systems
• customer acquisition cost • lifetime value • catalogue exploitation
• Bookmate – E-Contenta case
16
RS
$ $ $ $
B2C Content Services
17
subscription, PPD or hybrid
limited attention and time
content may have different cost
Unit Economics
18
Business at scale (marginal revenue and expenses per user)
LTV
Cost of content
CAC
user
life
time
ARPU
ARPU
…
PROFIT!
How the product works
• Each connection here is driven and improved by business activities
• The content itself fits into a sort of a BCG matrix:
19
GROWTH
CO
STS
CAC
×
÷
Driving Business Goals
20
CAC
LTV
Content Costs
Marketing Expenses
New Customers
ARPU
Lifetime
Consumed Content Mix
Conversion
Retention
Reactivation
Exposed Content Mix
×
÷
Driving Business Goals
21
CAC
LTV
Content Costs
Marketing Expenses
New Customers
ARPU
Lifetime
Consumed Content Mix
Conversion
Retention
Reactivation
Exposed Content Mix
*
* the recommendation fairy
Agenda
• Recommender Systems: Academy, Technology, Business
• Recommender Systems for content discovery
• B2C Content Services: overview and business model
• Driving business goals with Recommender Systems
• customer acquisition cost • lifetime value • catalogue exploitation
• Bookmate – E-Contenta case
22
RS
$ $ $ $
Option I: Improving conversion / CAC
Hypotheses to prove:
1. There’re enough users who will use RS output 2. Their conversion will be above average
A/B testing is the only way:
§ different channels convert with up to 20x difference § current traffic mix is unpredictable and hard to control in case of app installs
Do pilots:
§ Run with limited resources, then extrapolate and decide if run full-scale
24
Option I: Improving conversion / CAC
Two approaches to estimate:
1. increase of revenue 2. decrease of CAC
Suits for estimating various models:
§ upfront costs (when the investments return)
§ flat fee (monthly license or added headcount)
§ variable costs (CPO or PaaS model)
25
UNIT ECONOMICS!
subscribers
marketing budget
Case Study: Bookmate + E-Contenta
Sounds promising!
Did 40% more users become converted?
Not really, as there was just 7% who didn’t know the book to start with.
26
Group A Group B
Decided to use this channel
Converted
3 starter books from editors
3 starter books from a cold-
start RS
2.17*X% X%
Y% 0.65*Y%
Overall conversion Z% 1.4*Z%
three-sigma
Let’s check the economics *
• In case of using a third-party RS on a CPO basis, in this case the CPO is limited by $0.14 (actually, much less)
• In case of a flat fee of $1000**/month, this is feasible starting from 7143 new subscribers/month, or $35K of marketing budget.
27 * CAC and marketing budget are model data ** some arbitrary number
1000 CAC = $5
Group A Group B
Blended conversion C% Blended conversion 1.028*C%
Increased conversion 1.4x for 7% of users
CAC = $4.86 +28
Blended conversion across all channels is C%
$5000 of traffic
Option 2: Improving retention / LTV Hypotheses to prove:
1. User pays as long as he finds what to read 2. There’re enough users who will use RS output 3. This channel has a discoverability above average
Ideal experiment:
§ A/B, then count actual lifetime § with lifetime close to year, it’s too long to wait
Solution:
§ do separate A/B for different user cohorts (new, 1 month old, 2 months old etc.) § estimate significant change in month-to-month retention for each cohort
28
Model case: estimating LTV improvement Let’s assume Recommender System led to 0.5%-3%* increase of month-to-month retention (old cohorts / new cohorts), Group A estimated lifetime is 9 months*.
29 * model data provided for illustration
That’s an equivalent for: • increase of the lifetime by 2.6
months for Group B • increase of LTV by 29% for
Group B
The area between the curves is equal to # of additional ARPUs
Option 3: Better catalogue exploitation
30
Just to give an idea:
• long-tail content in general is cheaper (niche, back-catalogue and public domain) • driving user out of search already improves margins • adding a recommender system really changes a balance • once you have the data from the pilot, estimation is quite straightforward
Conclusions for B2C services
• The simplest recommender system would likely give you 80% of all possible upside. If it doesn’t, the problem is most likely not in the algorithm.
• If you want to go beyond, run a pilot to assess costs and benefits, then estimate if you have enough scale to afford the solution.
• If you deal with a third-party Recommender System convince them to fair pricing (e.g. free period until you have enough scale).
• And, again 31
Conclusions for B2B Recommender Platforms • Based on amount of traffic, price of marketing budget you can estimate value of
your solution for potential customers.
• Based on pilot integrations, you may either define a fair price point for a particular customer or develop PaaS-style tiered pricing model.
• Doing just a UX-applicable Recommender Systems leaves you a quite tight margin between LTV and CAC+COGS. Better take on the full user acquisition vertical.
• TEASER: Bookmate + E-Contenta 2.0: E-Contenta integrates with remarketing solution and provides traffic, not just recommendations.
32