playlist consumption on spotify - wordpress.complaylist consumption on spotify getting the success...
TRANSCRIPT
![Page 1: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/1.jpg)
Personalizing the user experience and playlist consumption on Spotify
Getting the success metrics right
Mounia LalmasDirector of Research @ Spotify11 October 2018, London
![Page 2: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/2.jpg)
This talk
1. A bit about myself2. A bit about user engagement
3. Personalising Spotify Home4. Accounting for user experience and
playlist consumption
Work by many at Spotify, and in particular Rishabh Mehrotra, James McInerney and Paolo Dragone (intern from University of Trento)
![Page 3: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/3.jpg)
A bit about myself
![Page 4: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/4.jpg)
A bit about myself
1998 to 2008 from Lecturer to Professor at Queen Mary University of London working on information retrieval, structured document retrieval and running INEX, a worldwide evaluation framework for XML retrieval
2008 to 2010 Microsoft Research/Royal Academy of Engineering Research Professor at University of Glasgow working on quantum-inspired models of interactive information retrieval
2011 to 2013 Visiting Principal Research Scientist at Yahoo Labs Barcelona working on user engagement in search, social media and news
2013 to 2017 Director of Research at Yahoo (now Oath) London working on advertising sciences
2017 to now Director of Research at Spotify London working on personalisation and discovery in the User Engagement mission
![Page 5: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/5.jpg)
Fields
Machine Learning
Language Technologies
Human Computer Interaction
Areas
Information RetrievalRecommender systems
User ModellingContent Analysis
EvaluationAlgorithmic bias
Tech Research @ User Engagement
Boston
London
New York
Incorporate and extend the state of the art in technologies relevant to current & future Spotify products.
![Page 6: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/6.jpg)
A bit about user engagement
![Page 7: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/7.jpg)
Characteristics of user engagement
Novelty
Richness and control
Reputation, trust and expectation
Motivation, interests, incentives, and benefits
Focused attention
Positive Affect
Aesthetics
Endurability
What is user engagement? … “Success”
Lalmas, O’Brien & Yom-Tov, “Measuring user engagement”, Morgan & Claypool Publishers, 2014.
User engagement is the quality of the user experience that emphasizes the positive aspects of interaction – in particular the fact of wanting to use the technology longer and often. Attfield, Kazai, Lalmas & Piwowarski, “Towards a science of user engagement”, WSDM Workshop, 2011.
![Page 8: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/8.jpg)
Characteristics of user engagement
Novelty
Richness and control
Reputation, trust and expectation
Motivation, interests, incentives, and benefits
Focused attention
Positive Affect
Aesthetics
User engagement is a quality of the user experience that emphasizes the positive aspects of interaction – in particular the fact of wanting to use the technology longer and often (Attfield et al., 2011).
Quality of the user experience
People remember “successful” experiences and
want to repeat them
We need metrics to quantify the quality of the
user experience → Metrics of “success”
Lalmas, O’Brien & Yom-Tov, “Measuring user engagement”, Morgan & Claypool Publishers, 2014.
Endurability
![Page 9: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/9.jpg)
Measures, metrics & key performance indicators
Measurement:
process of obtaining one or more quantity values that can reasonably be attributed to a quantity
e.g. number of clicks
(signals)
Metric:
a measure is a number that is derived from taking a measurement … in contrast, a metric is a calculation
e.g. click-through rate
Key performance indicator (KPI):
metric demonstrating how effectively key business objectives are being achieved
e.g. daily active users
https://www.klipfolio.com/blog/kpi-metric-measure
a measurement can be used as metric but not all metrics are measurementsa KPI is a metric but not all metrics are KPIs
![Page 10: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/10.jpg)
Three levels of metrics of success
Business metrics -- KPIs
Behavioral metrics -- online metrics
Optimisation metrics -- target metrics used to train machine learning algorithms
Thes
e le
vels
are
con
nect
ed
![Page 11: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/11.jpg)
Why not one optimisation metric?
Playlists differ in their listening patterns.
Genres and moods can be viewed as sub-hubs, each with some common patterns.
Searching has a particular pattern.
Home can be viewed as a hub with a “star” style pattern.
Media type and freshness lead to different patterns.
Different listening patternsDifferent consumption patterns
↓↑Different expectations of what success is
![Page 12: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/12.jpg)
Metrics and the Machine Learning ecosystem
Algorithm(s)
Training
Features(item)
Features(user)
Metric(s)
Signals (interaction) and online learning
Evaluation offline and online
![Page 13: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/13.jpg)
Personalising on Spotify Home
![Page 14: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/14.jpg)
ARTISTS FANS
Spotify’s mission is to unlock the potential of human creativity—by giving a million creative artists the opportunity to live off their art and billions of
fans the opportunity to enjoy and be inspired by it.
![Page 15: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/15.jpg)
User Engagement mission
Match fans and creators in a personal and relevant way.
ARTISTS FANS
![Page 16: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/16.jpg)
BaRT: Machine learning algorithm for Spotify Home
Streaming
UserBaRT
J. McInerney, B. Lacker, S. Hansen, K. Higley, H. Bouchard, A. Gruson, R. Mehrotra. “Explore, Exploit, Explain: Personalizing Explainable Recommendations with Bandits”, RecSys 2018.
![Page 17: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/17.jpg)
BaRT (Bandits for Recommendations as Treatments)
How to rank playlists (cards) in each shelf first, and then how to rank the shelves?
![Page 18: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/18.jpg)
Explore vs Exploit
Flip a coin with given probability of tail
If head, pick best card in M according to predicted reward r → EXPLOIT
If tail, pick card from M at random → EXPLORE
There are other methods
Multi-armed bandit algorithms
success = reward function
https://hackernoon.com/reinforcement-learning-part-2-152fb510cc54
Li, Chu, Langford & Schapire, “A contextual Bandit Approach to Personalized News Article Recommendation”, WWW 2010.
![Page 19: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/19.jpg)
BaRT: Multi-armed bandit algorithm for Spotify Home
![Page 20: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/20.jpg)
Accounting for user experience and
playlist consumption
![Page 21: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/21.jpg)
Accounting for user experience and playlist consumption by revisiting the reward function
Streaming
Reward
User
Binarised Streaming Time
BaRT
Success is when user streams the playlist for at least 30s.success = reward function
![Page 22: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/22.jpg)
Is success the same for all playlists?
Aggregate over playlists
Consumption time of a sleep playlist is longer than average playlist consumption time.
![Page 23: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/23.jpg)
Is success the same for all users?
Aggregate over users
Jazz listeners consume Jazz and other playlists for longer period than average. users.
![Page 24: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/24.jpg)
What we propose as success metrics
Distribution-based reward functions
![Page 25: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/25.jpg)
One reward function for all users and all playlists
Users Playlists
How BaRT currently works … success independent of user & playlist
reward function
![Page 26: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/26.jpg)
reward functions
. . .
Users Playlists
One reward function per user x playlist
Ideal case … success depends on user and playlist
Likely too granular, sparse, noisy, and costly to generate and maintain.
![Page 27: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/27.jpg)
. . .
reward functions
User groups Playlist groups
One reward function per group of users & playlists
Trade-off … success depends on group of users listening to group of playlists
How to generate the groups and how to derive reward functions from these?
![Page 28: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/28.jpg)
Generating the groups of users x playlists
We use co-clustering
![Page 29: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/29.jpg)
Co-clustering
Co-clustering
Dhillon, Mallela & Modha, "Information-theoretic co-clustering”, KDD, 2003.
Caveat no theoretical foundation for selecting the number of co-clusters apriori
group = clustergroup of user x playlist = co-cluster
![Page 30: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/30.jpg)
Co-clustering for Spotify Home
Users
Playlists
User groups
Playlist groups
Any (interaction) signal can be used to generate the co-clusters.
![Page 31: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/31.jpg)
Playlist groups
User groups
Co-clustering for Spotify Home using streaming time (5 user groups x 7 playlist groups)
Distribution plot of streaming time for that co-cluster
![Page 32: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/32.jpg)
Co-clustering for Spotify Home using streaming time (5 user groups x 7 playlist groups)
![Page 33: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/33.jpg)
Most frequent genre in co-clusterspopindo
francophone
francophone
francophone
danish
dansband
frenchvariety
classical
classical
frenchvariety
desi
desi
country
k pop
dansband
inspiration
danish
funkbrazil
francophone
folk americana
inspiration
dutch
folk americana
country
inspiration
dansband
dutch
dansband
mandopop
dutch
francophone
colombian
danish
colombian
desi
![Page 34: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/34.jpg)
Generating the reward functions
We use various thresholding methods
![Page 35: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/35.jpg)
3 reward functions based on● Mean● Additive● Continuous
Reward function per co-cluster using distribution of streaming time
![Page 36: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/36.jpg)
Mean-based reward functionUse the mean as threshold to define successA common methodology to determine a threshold
![Page 37: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/37.jpg)
Additive-based reward functionAdditive reward based on mean and standard deviationDifferentiate success for streaming time around the mean
![Page 38: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/38.jpg)
Generalisation of additive-based reward functionBased on cumulative distribution, where longer streaming time indicates higher success
Continuous-based reward function
![Page 39: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/39.jpg)
Offline evaluation
Experiments and Results
![Page 40: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/40.jpg)
Set-up of the offline evaluation
● Counterfactual methodology, which works like offline A/B testing, and aims to address data bias, provide good indication of causality and allows reusability
○ Unbiased counterfactual evaluation possible due to randomized data collection, where we show small set of users random content & log interaction
● Test all proposed thresholding methods (mean, additive, continuous & random)● Baseline (one threshold), playlists only, users only, both
● One week dataset made of random sample of over 800K users and 900K playlists on approximately 8M user-playlist interactions
● Expected stream rate (reported here), precision, recall & F score
Li, Chu, Langford & Wang, “Unbiased Online Evaluation of Contextual-bandit-based News Article Recommendation Algorithms”, WSDM 2011.
![Page 41: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/41.jpg)
Baseline is when success = streaming for at least 30 secondsPerformance measured with expected stream rate
Results
![Page 42: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/42.jpg)
Distribution-aware threshold better than fixed threshold for global single model (no clustering on anything)
![Page 43: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/43.jpg)
Among distribution-aware methods, mean better than additive itself better than continuous itself better than fixed threshold
mean
![Page 44: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/44.jpg)
Clustering on either user or playlist performs better than fixed thresholdAmong the two, user- and playlist-clustering perform similarly
![Page 45: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/45.jpg)
Co-clustering performs better than fixed thresholds, distribution-aware global thresholds and separate user-only & playlist-only clustering
![Page 46: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/46.jpg)
Distribution awareness in co-clustering is important as random threshold on co-cluster does not perform as well
![Page 47: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/47.jpg)
Recap of the main findings● Distribution-aware threshold >> fixed threshold even without any clustering
● Among distribution-aware methods: Mean > Additive > Continuous >> Fixed
● Clustering on either user or playlist >> fixed threshold
● User- and playlist- clustering perform similarly
● Co-clustering performs better than (i) fixed thresholds, (ii) distribution-aware global thresholds and (iii) user-only & playlist-only clustering
● Distribution-aware in co-cluster >> random threshold in co-cluster
![Page 48: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/48.jpg)
Conclusions and what next?
![Page 49: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/49.jpg)
Conclusions
Accounting for user experience and playlist consumption matters.
Co-clustering users and playlists surfaces patterns of user experience x playlist consumption.
Using one interaction signal and a simple thresholding method can already provide effective personalised success metrics.
![Page 50: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/50.jpg)
What next?
● Going deeper
○ Number of co-clusters → hyperparameter
○ Semantics of co-clusters → in-depth analysis
● Online experiments
● Future research directions
○ How do we account for changing user behavior and seasonality?
○ Can we learn distribution-aware reward functions?
![Page 51: playlist consumption on Spotify - WordPress.complaylist consumption on Spotify Getting the success metrics right Mounia Lalmas Director of Research @ Spotify ... 2017 to now Director](https://reader030.vdocuments.us/reader030/viewer/2022041019/5ecd26205f998d2c8838e090/html5/thumbnails/51.jpg)
Thank you!