appfunnel: a framework for usage-centric evaluation of recommender systems that suggest mobile...
TRANSCRIPT
AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications
Matthias BöhmerLyubomir GanevAntonio Krüger
!
Introduction
‣Number of mobile applications steadily increasing
‣ Finding good apps can become a difficult task
‣Recommender systems can help- Mobile apps special type of items- Context is important
‣ „Dead apps“ on smartphones- Less than 1/2 of apps are actively used- Installation counts good for evaluation?
Our paper...
(1) Introduces a usage-centric evaluation framework(2) Presents results of a case study
3
4
‣ AppFunnel
‣ Idea: evaluate recommendation based on app engagement
‣Concept of funnels adopted from advertising domain
‣Apps can reach different stages after recommendation
‣ Tracking engagement with apps along stages
Concept of AppFunnel
VIEW INSTALLATION DIRECT USAGE LONG-TERM USAGE
RECOMMENDATION
6
‣ Case Study
Testbed: appazaar
‣Recommender system for Android applications
‣Available for end-users on Google Play Market for free
‣ 6,680+ users of appazaar, worldwide distribution
‣ 45 users contributed to case study over three months
Recommender Engines under Test
Non-personalized Personalized
Context-less - App popularity - Usage-based CF
Context-aware - App-aware filtering - Location-aware CF- Time-aware CF
Conversion Stages
Ave
rage
num
ber o
f occ
urre
nces
per r
ecom
men
datio
n lis
t
0.0
0.2
0.4
0.6
0.8
1.0
Location−aware Collaborative
Filtering
App−popularity Filtering
App−aware Filtering
Usage−based Collaborative
Filtering
Time−aware Collaborative
Filtering
AppFunnel stage
view view market installation direct usage long−term usage
App-popularity
Usage-based CF
App-aware
Time-aware CF
Location-aware CF
Conversion Rates
Con
vers
ion
rate
s in
per
cent
0102030405060708090
100
Location−aware Collaborative
Filtering
App−popularity Filtering
App−aware Filtering
Usage−based Collaborative
Filtering
Time−aware Collaborative
Filtering
Conversion
view to installation installation to direct usage installation to long−term usage
App-popularity
Usage-based CF
App-aware
Time-aware CF
Location-aware CF
Results of Case Study
‣AppFunnel shows difference in engines‘ performances, e.g.- Context-less engines: more views- Context-aware engines: better from installation to direct use
‣AppFunnel enables validation of engines‘ design goals, e.g.- App-aware engine: high installation to direct-usage rate- Context-less engines: higher rates for long-term usage
‣AppFunnel helped us to trace other issues of engines, e.g.- Location-aware engine: many views but no installations
Limitations
‣Recommender engines under test are rather simplistic
‣Updates and removal not taken into account- Update does not tell much about app engagement- Removal events needs further refinement of AppFunnel
‣ Future work:- Improve recommender engines by incorporating
results of case study
Discussion
‣Choosing from AppFunnel‘s metrics- Metrics might counteract (e.g., direct vs. long-term usage)- Metrics should reflect design goal (context -> direct usage)
‣Usage-centric evaluation valuable for other domains- When engagement with item can be traced- Changing view from „selling items“ to UX of items
‣ Conclusion
14
Conclusion
‣We presented AppFunnel- Usage-centric evaluation of recommender engines- Recommendation of mobile applications
‣We presented a case study of AppFunnel in the wild- Traces down performance beyond installations- Shows differences between engines
VIEW INSTALLATION DIRECT USAGE LONG-TERM USAGE
RECOMMENDATION