the role of domain knowledge in a large scale data mining project kopanas i., avouris n., daskalaki...
TRANSCRIPT
The role of Domain Knowledge in a large scale Data Mining Project
Kopanas I., Avouris N., Daskalaki S.
University of Patras
University of Patras, HCI Group - SETN02 2
Outline of the talk
• Knowledge in a DM process
• Case study in a large DM project: Prediction of customer insolvency in Telecommunications business
• The role of domain expertise (and domain experts ) in the process
• Summary and conclusions
University of Patras, HCI Group - SETN02 3
Data Mining• Evolution of knowledge-based systems
• Key partners in Data Mining
– Data analyst / statistician
– Knowledge Engineer
– Domain Expert
• Role of domain knowledge in Data Mining
University of Patras, HCI Group - SETN02 4
DM phases(a) Problem definition
(b) Creating target data set
(c ) Data pre-processing and transformation
(d ) Feature and algorithm selection
(e) Data Mining
(f) Evaluation of learned knowledge
(g) Fielding the knowledge base
University of Patras, HCI Group - SETN02 5
Case study: Prediction of Customer Insolvency in Telecommunications businessPredict the insolvent customers to be, that is the
customers that will refuse to pay their telephone bills in the next payment due date, while there is still time for preventive (and possibly avertive) measures
• Problem ObjectivesProblem Objectives– Detect as many insolvent customers as possible
– Minimize false alarms (solvent customers classified as insolvent)
University of Patras, HCI Group - SETN02 6
Case study: problem characteristics
• Significant loss of revenue for the company
• Human behavior is (generally) unpredictable
• Insolvency cases are rare compared to non-insolvencies
• Information can be retrieved only after processing huge amounts of data from several sources
University of Patras, HCI Group - SETN02 7
The billing process (domain knowledge)
Jun Jul Aug Sept Feb AprMarOct Nov JanDec
Billing Period
Due Date
Issue of Bill
Service Interruption
Nullification
University of Patras, HCI Group - SETN02 8
Target data set definition (semantic value of data)• Data from 3 different cities (combination of rural,
urban and touristic areas)
• Types of data – Customer data (coded)
– Data from billing and payments
– Call detail records (from switching centers)
• Time span of data studied– Cases of collected and uncollected bills (10/99-2/01)
– Calls records (8/99-12/00)
University of Patras, HCI Group - SETN02 9
Data pre-processing (knowledge-based reduction of search space)
• Eliminated inexpensive calls (< 0.3 €)
• Synchronizing data
• Removing noise
• Missing values
• Data aggregation by period
DATA
WAREHOUSE
University of Patras, HCI Group - SETN02 10
Dataset for model fitting
• Stratified sample of solvent customers– Class distribution: 90% solvent customers and 10%
insolvent customers
• 2066 total number of cases and 46 variables – 2 variables describing the phone account
– 4 variables describing customer attitude towards previous phone bills
– 40 variables summarizing customer call habits over fifteen 2-week periods
University of Patras, HCI Group - SETN02 11
Data mining
• Classification problemClassification problem
– 2 classes: solvent and insolvent customers
– Distribution among classes in original dataset: 99% of solvent customers and 1% of insolvent customers
– Very small number of insolvencies
– Very different costs of misclassification between the two classes of customers
University of Patras, HCI Group - SETN02 12
Criteria for evaluation of predictionThe precision of the classifier, defined as the percentage of the actually insolvent customers in those, predicted as insolvent by the classifier.
The accuracy of the classifier, defined as the percentage of the correctly predicted insolvent out of the total cases of insolvent customers in the data set.Precision > 30% & Accuracy > 70%
University of Patras, HCI Group - SETN02 13
Features selected (most popular in 50 classifiers)
• NewCust
• Latency
• Count_X_charges
• CountResiduals
• StdDif
• TrendDif11
• TrendDif10
• TrendDif7
• TrendDif6
• TrendDif3
•TrendUnitsMax
•TrendDif5
•TrendDif8
•Average_Dif
•Type
•MaxSec
•TrendUnits5
•AverageUnits
•TrendCount5
•CountInstallments
TrendDifxx , StdDif dispersion of called telephone numbers in a given time interval xx
University of Patras, HCI Group - SETN02 14
Deployment of the Knowledge-based system
• The classifiers are combined (voting algorithms have been used)
• Heuristics are used as applicability criteria
• Visualization plays an important role in the design of the system
• The roles of the user and the knowledge-based system have to be carefully defined
University of Patras, HCI Group - SETN02 15
Stepwise Discriminant Analysis
Classification Results E3 Predicted
Category 0 1 Total 0 78 58 136 Count 1 28 1184 1212 0 57.35 42.65 100
Original
% 1 2.31 97.69 100 0 77 59 136 Count 1 35 1177 1212 0 56.62 43.38 100
Cases Selected Cross-
validated %
1 2.89 97.11 100 0 36 28 64 Count 1 22 632 654 0 56.25 43.75 100
Cases not Selected
Original
% 1 3.36 96.64 100
93.6% of selected original grouped cases correctly classified 93.02% of selected cross-validated cases correctly classified 93.04% of unselected original grouped cases correctly classified
University of Patras, HCI Group - SETN02 16
Decision Tree
CCaatteeggoorryy 00 11 TToottaall
CCoouunntt 00 110011 3355 113366 11 99 11220033 11221122
%% 00 7744..2266 2255..7744 110000 11 00..7744 9999..2266 110000
CCoouunntt 00 4422 2222 6644 11 1166 663388 665544
%% 00 6655..6622 3344..3388 110000 11 22..4455 9977..5555 110000
CCllaassssiiffiiccaattiioonn RReessuullttss EE2211 Predicted Group
Original
Cases not Selected
Original
Cases Selected
University of Patras, HCI Group - SETN02 17
Neural Network
Category 00 11 Total
CCoouunntt 00 6655 6699 113366 11 88 11220033 11221122
%% 00 4477..77 5500..77 110000 11 00..66 9999..22 110000
CCoouunntt 00 2244 4400 6644 11 1111 664433 665544
%% 00 3377..55 6622..55 110000 11 11..66 9988..33 110000
Classification Results E30 Predicted Group
Original
Cases not Selected
Original
Cases Selected
University of Patras, HCI Group - SETN02 18
Evaluation of classifiers (example)
• Performance over 90% in the majority class and over 83% in the minority class.
• precision = 113/2844= 3.9%
• accuracy = 113/136= 83%,
Predicted cases
Category Insolvent (0) Solvent (1)
Insolvent (0) 113
(83.1 %) 23
(16.9%) Actual cases
Solvent (1) 2731
(9.8 %) 25081
(90.2 %)
University of Patras, HCI Group - SETN02 19
stage DK Type of DK(a) Problem definition HIGH Business and domain knowledge,
requirements Implicit, tacit knowledge
(b) Creating target data set
MEDIUM Attribute relations, semantics of corporate DB
(c ) Data pre-processing
HIGH Tacit and implicit knowledge for inferences
(d ) Feature and algorithm selection
MEDIUM Interpretation of the selected features
(e) Data Mining
LOW Inspection of discovered knowledge
(f) Evaluation of learned knowledge
MEDIUM Definition of criteria related to business objectives
(g) Fielding the knowledge base
HIGH Supplementary domain knowledge necessary for implementing the system
University of Patras, HCI Group - SETN02 20
Selection of DM tool (Elder 98)
University of Patras, HCI Group - SETN02 21
Conclusion• Data mining is a knowledge-driven process
• All stages contribute to the success of the process
• Domain experts play significant role in most phases of the process
• Need for selection of algorithms and techniques that support interpretation of mined knowledge
• Need for integrated tools and adequate techniques to support involvement of domain experts in the process