![Page 1: AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications](https://reader030.vdocuments.us/reader030/viewer/2022020217/554e93d7b4c90573338b4fc9/html5/thumbnails/1.jpg)
AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications
Matthias BöhmerLyubomir GanevAntonio Krüger
!
![Page 2: AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications](https://reader030.vdocuments.us/reader030/viewer/2022020217/554e93d7b4c90573338b4fc9/html5/thumbnails/2.jpg)
Introduction
‣Number of mobile applications steadily increasing
‣ Finding good apps can become a difficult task
‣Recommender systems can help- Mobile apps special type of items- Context is important
‣ „Dead apps“ on smartphones- Less than 1/2 of apps are actively used- Installation counts good for evaluation?
![Page 3: AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications](https://reader030.vdocuments.us/reader030/viewer/2022020217/554e93d7b4c90573338b4fc9/html5/thumbnails/3.jpg)
Our paper...
(1) Introduces a usage-centric evaluation framework(2) Presents results of a case study
3
![Page 4: AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications](https://reader030.vdocuments.us/reader030/viewer/2022020217/554e93d7b4c90573338b4fc9/html5/thumbnails/4.jpg)
4
‣ AppFunnel
![Page 5: AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications](https://reader030.vdocuments.us/reader030/viewer/2022020217/554e93d7b4c90573338b4fc9/html5/thumbnails/5.jpg)
‣ Idea: evaluate recommendation based on app engagement
‣Concept of funnels adopted from advertising domain
‣Apps can reach different stages after recommendation
‣ Tracking engagement with apps along stages
Concept of AppFunnel
VIEW INSTALLATION DIRECT USAGE LONG-TERM USAGE
RECOMMENDATION
![Page 6: AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications](https://reader030.vdocuments.us/reader030/viewer/2022020217/554e93d7b4c90573338b4fc9/html5/thumbnails/6.jpg)
6
‣ Case Study
![Page 7: AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications](https://reader030.vdocuments.us/reader030/viewer/2022020217/554e93d7b4c90573338b4fc9/html5/thumbnails/7.jpg)
Testbed: appazaar
‣Recommender system for Android applications
‣Available for end-users on Google Play Market for free
‣ 6,680+ users of appazaar, worldwide distribution
‣ 45 users contributed to case study over three months
![Page 8: AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications](https://reader030.vdocuments.us/reader030/viewer/2022020217/554e93d7b4c90573338b4fc9/html5/thumbnails/8.jpg)
Recommender Engines under Test
Non-personalized Personalized
Context-less - App popularity - Usage-based CF
Context-aware - App-aware filtering - Location-aware CF- Time-aware CF
![Page 9: AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications](https://reader030.vdocuments.us/reader030/viewer/2022020217/554e93d7b4c90573338b4fc9/html5/thumbnails/9.jpg)
Conversion Stages
Ave
rage
num
ber o
f occ
urre
nces
per r
ecom
men
datio
n lis
t
0.0
0.2
0.4
0.6
0.8
1.0
Location−aware Collaborative
Filtering
App−popularity Filtering
App−aware Filtering
Usage−based Collaborative
Filtering
Time−aware Collaborative
Filtering
AppFunnel stage
view view market installation direct usage long−term usage
App-popularity
Usage-based CF
App-aware
Time-aware CF
Location-aware CF
![Page 10: AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications](https://reader030.vdocuments.us/reader030/viewer/2022020217/554e93d7b4c90573338b4fc9/html5/thumbnails/10.jpg)
Conversion Rates
Con
vers
ion
rate
s in
per
cent
0102030405060708090
100
Location−aware Collaborative
Filtering
App−popularity Filtering
App−aware Filtering
Usage−based Collaborative
Filtering
Time−aware Collaborative
Filtering
Conversion
view to installation installation to direct usage installation to long−term usage
App-popularity
Usage-based CF
App-aware
Time-aware CF
Location-aware CF
![Page 11: AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications](https://reader030.vdocuments.us/reader030/viewer/2022020217/554e93d7b4c90573338b4fc9/html5/thumbnails/11.jpg)
Results of Case Study
‣AppFunnel shows difference in engines‘ performances, e.g.- Context-less engines: more views- Context-aware engines: better from installation to direct use
‣AppFunnel enables validation of engines‘ design goals, e.g.- App-aware engine: high installation to direct-usage rate- Context-less engines: higher rates for long-term usage
‣AppFunnel helped us to trace other issues of engines, e.g.- Location-aware engine: many views but no installations
![Page 12: AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications](https://reader030.vdocuments.us/reader030/viewer/2022020217/554e93d7b4c90573338b4fc9/html5/thumbnails/12.jpg)
Limitations
‣Recommender engines under test are rather simplistic
‣Updates and removal not taken into account- Update does not tell much about app engagement- Removal events needs further refinement of AppFunnel
‣ Future work:- Improve recommender engines by incorporating
results of case study
![Page 13: AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications](https://reader030.vdocuments.us/reader030/viewer/2022020217/554e93d7b4c90573338b4fc9/html5/thumbnails/13.jpg)
Discussion
‣Choosing from AppFunnel‘s metrics- Metrics might counteract (e.g., direct vs. long-term usage)- Metrics should reflect design goal (context -> direct usage)
‣Usage-centric evaluation valuable for other domains- When engagement with item can be traced- Changing view from „selling items“ to UX of items
![Page 14: AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications](https://reader030.vdocuments.us/reader030/viewer/2022020217/554e93d7b4c90573338b4fc9/html5/thumbnails/14.jpg)
‣ Conclusion
14
![Page 15: AppFunnel: A Framework for Usage-centric Evaluation of Recommender Systems that Suggest Mobile Applications](https://reader030.vdocuments.us/reader030/viewer/2022020217/554e93d7b4c90573338b4fc9/html5/thumbnails/15.jpg)
Conclusion
‣We presented AppFunnel- Usage-centric evaluation of recommender engines- Recommendation of mobile applications
‣We presented a case study of AppFunnel in the wild- Traces down performance beyond installations- Shows differences between engines
VIEW INSTALLATION DIRECT USAGE LONG-TERM USAGE
RECOMMENDATION