prefminer:)mining)user’s)preferences)for ...axm514/paper/slides/ubicomp16.pdf ·...
TRANSCRIPT
PrefMiner: Mining User’s Preferences for Intelligent Mobile No5fica5on Management
Abhinav Mehrotra University College London University of Birmingham
Robert Hendley University of Birmingham
Mirco Musolesi University College London
No5fica5ons Inform Users About a Variety of Events
System, Tools and Others
Online Social Networks Communica5on
Are All No5fica5ons Useful?
NO
Users are not interested in many no5fica5ons they receive and feel annoyed on receiving them.
Interrup5bility Management System: State-‐of-‐the-‐art
Predict opportune moments by using:
• context of the user. • content of no5fica5on.
Limita8on No past study has considered stopping or detec5ng
no5fica5ons that are not useful for the user.
Interrup5bility Management System: State-‐of-‐the-‐art
Evaluated in the offline sePngs.
Limita8on Never evaluated in the real-‐world scenario.
Contribu5ons
1. A mechanism for learning the types of informa5on users prefer to receive via no5fica5ons in different situa5ons.
2. A technique to make the predic5ve model transparent to
users.
Mining User Preferences
Construc5ng Associa5on Rules
Removing Reminder
No5fica5ons
No5fica5on Clustering
Mining User Preferences
Construc5ng Associa5on Rules
Removing Reminder
No5fica5ons
No5fica5on Clustering
If an applica5on’s click rate is zero then all
no5fica5ons from that applica5on are treated as reminder no5fica5ons.
Click Rate Number of accepted no5fica5ons Total number of no5fica5ons *
Some no5fica5ons are dismissed because they do not require any further ac5on from the user.
= 100
S1: Removing Reminder No5fica5ons
Mining User Preferences
Construc5ng Associa5on Rules
Removing Reminder
No5fica5ons
No5fica5on Clustering
S2: No5fica5on Clustering
1. Cleaning No5fica5on Titles • Conversion of the text to lower-‐case • Removal of punctua5on, numbers and stop words • Removal of the sender and applica5on names • Stemming of words
2. Clustering No5fica5ons • Removing sparse terms • Term Frequency-‐based clustering
No5fica5on Title Applica5on Name
S2: No5fica5on Clustering
Removing sparse terms
TFthreshold Number of par5cipa5on days N * Total number of no5fica5ons =
This ensures that at least one no5fica5on containing the term is triggered in N days.
0
1000
2000
3000
1 2 3 4 5 6 7N−value
Term
s C
ount
Filtered Remaining Not much difference in the number of filtered terms with N ∈ [2, 7]. We use N = 2 for Tfthreshold .
Mining User Preferences
Construc5ng Associa5on Rules
Removing Reminder
No5fica5ons
No5fica5on Clustering
S3: Construc5ng Associa5on Rules We use the AIS algorithm for mining associa5on rules.
X → Y
X is the antecedent (never contains no5fica5on response) Y as the consequent (only contains no5fica5on response)
S3: Construc5ng Associa5on Rules We use the AIS algorithm for mining associa5on rules.
X → Y
X is the antecedent (never contains no5fica5on response) Y as the consequent (only contains no5fica5on response)
Support: Number of no5fica5ons of the no5fica5on-‐type covered in the rule
Total number of no5fica5ons in the dataset Confidence:
Occurrence count of the no5fica5on pabern covered in the rule Occurrence count of the no5fica5on pabern covered in the antecedent of rule
* 100
* 100
S3: Construc5ng Associa5on Rules We use the AIS algorithm for mining associa5on rules.
X → Y
X is the antecedent (never contains no5fica5on response) Y as the consequent (only contains no5fica5on response)
{N1} → {Dismiss} {N2,Home} → {Accept} {N2,Other} → {Accept} {N2,Work} → {Dismiss}
Let us consider an example where the user: • always dismiss N1 no5fica5ons. • does not accept N2 no5fica5ons at work.
Procedure
1. Evaluate (offline) the proposed intelligent no5fica5on mechanism. 2. Funnel the findings in an intelligent no5fica5on library: MyPref. 3. Implement an app on top of MyPref and deploy it to evaluate the
library in a real-‐world scenario.
Dataset
Study: My Phone and Me
Par5cipants: 18
Minimum ac5ve days: 14
No5fica5on samples: 11,185
Abhinav Mehrotra, Veljko Pejovic, Jo Vermeulen, Robert Hendley, and Mirco Musolesi. 2016. My Phone and Me: Understanding People’s Recep5vity to Mobile No5fica5ons. In CHI’16.
Types of Associa5on Rules
Assoc Rule 1: no5fica5on response with no5fica5on type. Assoc Rule 2: no5fica5on response with no5fica5on type and ac5vity. Assoc Rule 3: no5fica5on response with no5fica5on type and arrival 5me. Assoc Rule 4: no5fica5on response with no5fica5on type and loca5on. Assoc Rule 5: no5fica5on response with no5fica5on type, ac5vity, arrival 5me and loca5on.
Results
Recall: ra5o between the number of no5fica5ons that are correctly predicted as dismissed and the total number of no5fica5ons that are actually dismissed. Precision: ra5o between the number of no5fica5ons that are correctly predicted as dismissed and the total number of no5fica5ons that are predicted as dismissed.
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
AR4 AR5
Minimum Confidence: 65-‐95% Minimum Support: Dpar5cipate / ( 2 ∗ Ncount )
Dpar5cipate: Days of par>cipa>on Ncount: number of no5fica5ons collected for a user This ensures that the type of no5fica5on covered in each rule has arrived at least once in two days.
Assoc Rule 1 Assoc Rule 2 Assoc Rule 3
Results
Recall: ra5o between the number of no5fica5ons that are correctly predicted as dismissed and the total number of no5fica5ons that are actually dismissed. Precision: ra5o between the number of no5fica5ons that are correctly predicted as dismissed and the total number of no5fica5ons that are predicted as dismissed.
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
Assoc Rule 4 Assoc Rule 5
Assoc Rule 1 Assoc Rule 2 Assoc Rule 3
Results
Recall: ra5o between the number of no5fica5ons that are correctly predicted as dismissed and the total number of no5fica5ons that are actually dismissed. Precision: ra5o between the number of no5fica5ons that are correctly predicted as dismissed and the total number of no5fica5ons that are predicted as dismissed.
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
Best Performance
Assoc Rule 1 Assoc Rule 2 Assoc Rule 3
Assoc Rule 4 Assoc Rule 5
Results
Recall: ra5o between the number of no5fica5ons that are correctly predicted as dismissed and the total number of no5fica5ons that are actually dismissed. Precision: ra5o between the number of no5fica5ons that are correctly predicted as dismissed and the total number of no5fica5ons that are predicted as dismissed.
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
AR4 AR5
Best Performance
User’s preference for receiving a no8fica8on depends only on the type of informa8on it contains
and the loca8on.
Assoc Rule 1 Assoc Rule 2 Assoc Rule 3
Op5mizing the System for High Precision
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
Assoc Rule 4
• We should aim to have fewer false-‐nega5ves (i.e., incorrectly predic5ng a no5fica5on as non-‐interes5ng for the user).
• This can be achieved by ensuring that the precision remains close to 100%.
Op5mizing the System for High Precision
0102030405060708090
100
70 80 90Confidence Value (%)
Res
ult (
%)
PrecisionRecall
• We should aim to have fewer false-‐nega5ves (i.e., incorrectly predic5ng a no5fica5on as non-‐interes5ng for the user).
• This can be achieved by ensuring that the precision remains close to 100%.
90% precision 35% recall
Assoc Rule 4
Online Learning
● ●
●
●
●
●●
●
●
●
● ● ●
●●
● ●
●
● ●●
90
100
1 3 5 7 9 11 13 15 17 19 21Day
Prec
isio
n (%
) Confidence● 65%
70%75%80%85%90%95%
● ●
●
●
●
●●
●
●
●
●● ● ● ●
● ● ●● ● ●
0
10
20
30
40
1 3 5 7 9 11 13 15 17 19 21Day
Rec
all (
%)
Confidence● 65%
70%75%80%85%90%95%
MyPref Library An intelligent interrup5bility management library that can predict the type of no5fica5ons that users would prefer to
receive on their mobile phones in specific contexts.
PlaQorm Implemented for the Android OS.
Evalua8on
It takes not more than 61 seconds for mining associa5on rules from 1500 no5fica5ons for any features.
Released as an open source project
hbps://github.com/AbhinavMehrotra/PrefMiner
In-‐the-‐wild Evalua5on Results
Par5cipants: 18
Dura5on: 15 days (minimum)
Data of 2 par5cipants was not considered because they never accepted/dismissed any suggested rule.
Accepted Rules
80%79%
77%
75%
73%73%
67%
64%
57%50%
50%
47%
45%
34%
25%
20%
0
5
10
15
20
25
30
35
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16User
Rul
es C
ount
AcceptedRejected
Overall, 102 out of suggested 179 rules were accepted. 56.98% acceptance rate.
Accepted Rules
80%79%
77%
75%
73%73%
67%
64%
57%50%
50%
47%
45%
34%
25%
20%
0
5
10
15
20
25
30
35
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16User
Rul
es C
ount
AcceptedRejected
Overall, 102 out of suggested 179 rules were accepted. 56.98% acceptance rate.
70% (11 out of 16) users accepted 50% (and above) rules.
Filtered No5fica5ons
Average filter recall for 16 users: 45.81%
0
20
40
60
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16User
Filte
r Rec
all (
%)
Filter Recall N-‐Auto N-‐Auto + N-‐Manual * = 100
N-‐Auto: Number of filtered no5fica5ons by the system N-‐Manual: Number of no5fica5ons dismissed manual
PrefMiner: Exit Ques5onnaire
Triggered aUer 15 days
Ques8on Average Response
I found the app useful for learning my preferences to filter no5fica5ons.
4.25
The app filtered most of the no5fica5ons that I didn’t want to receive.
4.33
The app incorrectly filtered no5fica5ons that I wanted to receive.
1.58
1:Strongly disagree -‐ 5:Strongly agree
• First in-‐the-‐wild study for automa5c extrac5on of rules that reflect user’s preferences.
• An open-‐source library (MyPref) that can learn the type of no5fica5ons that users would prefer to receive on their mobile phones in specific contexts.
• PrefMiner represents a very effec5ve, yet transparent, solu5on for interrup5bility management for mobile devices.
Take Aways
42
Ques5ons?
Abhinav Mehrotra
Email: [email protected]
GitHub: hbps://github.com/AbhinavMehrotra
Play Store: hbps://play.google.com/store/apps/details?id=com.nsds.prefminer