fuzzy interpretation of discretized intervals dr. xindong wu andrea porter april 11, 2002
Post on 21-Dec-2015
216 views
TRANSCRIPT
![Page 1: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/1.jpg)
Fuzzy Interpretation of Discretized Intervals
Dr. Xindong Wu
Andrea Porter
April 11, 2002
![Page 2: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/2.jpg)
Plan For Presentation
Introduction to Problem, HCVDiscretization Techniques/Fuzzy
BordersA Hybrid Solution for HCVExperiments and ResultsConclusion
![Page 3: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/3.jpg)
Introduction
Real-world data contains both numerical and nominal data, must be able to deal with different types of data.
Existing systems discretize numerical domains into intervals and treat intervals as nominal values during induction.
Problems occur if test examples are not covered in training data (no-match, multiple match)
The solution is a hybrid approach using fuzzy intervals for no-match problem.
![Page 4: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/4.jpg)
HCV
Attribute based rule induction algorithm, extension matrix approach Divide positive examples into intersecting groups Find a heuristic conjunctive rule in each group that
covers all PE and no NE
HCV can find a rule in the form of variable-valued logic
More compact than the decision trees/rules of ID3 and C4.5
![Page 5: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/5.jpg)
Variable Valued Logic and Selectors
Represents decisions where variables can take a range
Selector:
[ X # R ]
X = attribute
# = relational operator ( = , <, >, . . . )
R = Reference, list of 1 or more values
e.g [ Windy = true] [Temp > 90]
, , ,...
![Page 6: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/6.jpg)
HCV Software
C++ implementationCan work with noisy and real-valued
domains as well as nominal and noise-free databases
Provides a set of deduction facilities for the user to test the accuracy of the produced rules on test examples
![Page 7: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/7.jpg)
Example DB ORDER X1 X2 X3 X4 CLASS
1 1 a a 1 F
2 1 a b 1 F
3 1 a c 1 F
4 1 a a 0 F
5 1 b c 1 T
6 0 b b 0 T
7 0 a c 1 T
8 1 b a 0 T
9 1 b a 1 T
10 1 c c 0 F
11 1 c b 1 F
12 0 c b 0 T
13 0 a a 0 T
14 0 c c 1 F
15 0 c a 0 T
16 1 a b 0 F
17 0 a a 1 T
18 0 b a 1 T
![Page 8: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/8.jpg)
C4.5 Results vs. HCV
C4.5:The T classX2 = b
X1 = 0 & X3 = a X1 = 0 & X3 = b X1 = 0 & X2 = a
HCV:The T classX2 = b
X1 = 0 & X2 = a
X1 = 0 & X4 = 0
C4.5:The F classX1 = 1 & X2 = a
X1 = 1 & X2 = c X2 = c & X3 = c
![Page 9: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/9.jpg)
Deduction of Induction Results
Induction generates knowledge from existing data Deduction applies induction results to interpret
new data. With real-world data, induction can not be
assumed to be perfect Three cases:
1) no-match (measure of fit)2) single-match3) multiple-match (estimate of probability)
![Page 10: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/10.jpg)
Discretization
Occurs during rule induction Discretize numerical domains into intervals and
treat similar to nominal values. The challenge is to find the right borders for the
intervals Possible Methods:
1) Simplest Class-Separating Method
2) Information Gain Heuristic (implemented in HCV)
![Page 11: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/11.jpg)
Simplest Class- Separating Method:
Interval Borders are places between each adjacent pair of examples which have different classes.
If attribute is very informative - method is efficient and useful.
If attribute is not informative - method produces too many intervals
![Page 12: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/12.jpg)
Information Gain Heuristic
Use IGH to find more informative border.
x = (xi + xi+1)/2 for (i = 1, …, n-1) x is a possible cut point if xi and xi+1 are of different
classes. Use IGH to find best x Recursively split on left and right To stop recursive splitting:
1) stop if IGH is same on all possible cut points.2) stop if # of examples to split is less than a predefined number3) limit the number of intervals
![Page 13: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/13.jpg)
Fuzzy Borders
Discretization of continuous domains does not always fit accurate interpretation.
Instead of using sharp borders, use a membership function, measures the degree of membership.
A value can be classified into a few different intervals at the same time (e.g. single to multiple match)
![Page 14: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/14.jpg)
Fuzzy Borders (2)
Fuzzy matching - deduction with fuzzy borders of discretized intervals.
Take the interval with the greatest degree as the value’s discrete value.
3 functions to fuzzify borders:
1) linear2) polynomial3) arctan
Definitionss = spread parameter l = length of original
xleft, xright = left/right sharp borders
xleft xright
l
![Page 15: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/15.jpg)
a = -kxleft + 1/2 b = kxright + 1/2
linleft(x) = kx + a
lin right(x) = -kx + b
lin(x) = MAX(0, MIN(1,linleft(x),linright(x)))
Linear Membership Function
k = 1/2sl
xleft xright
l
sl
![Page 16: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/16.jpg)
Arctan Membership Function
![Page 17: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/17.jpg)
*Polynomial Membership Function
polyside(x) = asidex3 + bsidex2 + csidex + dside
aside = 1/(4(ls)3)
bside = -3asidexside side {left,right}
cside = 3aside(xside2 - (ls)2)
dside = -a(xside3 -3xside(ls)2 + 2(ls)3)
polyleft(x), if xleft -ls x xleft + ls
poly(x) = polyright(x), if xright -ls x xright +ls
1, if xleft +ls x xright -ls
0, otherwise
![Page 18: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/18.jpg)
Match Degree
Selector method - take the max membership degree of the value in all the intervals involved. If 2 adjacent intervals have the same class, values close to the border will have low membership.
Conjunction method - adds with fuzzy plus
ab=a + b - ab
![Page 19: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/19.jpg)
No-Match Resolution
Largest Class
Assign all no match examples to the largest class, the default class.
Works well, if the number of classes in a training set is small and one class is clearly larger.
Deteriorates if there is a larger number of classes and the examples are evenly distributed
![Page 20: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/20.jpg)
No-Match Resolution
Measure of FitCalculate the measure of fit for each class:
1) calculate MF for each selector (sel)MF(sel, e) = 1, if sel is satisfied by e
n/|x|, otherwise2) calculate MF for each conjunctive rule(conj)MF(conj, e) = MF(sel, e) * n(conj)/N
![Page 21: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/21.jpg)
No-Match Resolution
Measure of Fit (2)3) calculate MF for each class c
MF(c, e) = MF(conj1, e) + MF(conj2, e) - MF(conj1,e)MF(conj2,e)
* For more than two rules, apply formula recursively.
* Find maximum MF - determines which class is closest to the example
![Page 22: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/22.jpg)
Multiple-Match
Caused by over-generalization of the training examples at induction time
Example (X1 = a, X2 = 1)
All PE cover X1 = a
All NE cover X2 = 1 Multiple Match
![Page 23: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/23.jpg)
Multiple-Match Resolution
First Hit Use first rule which classifies the example Produces reasonable results if the rules from
induction have been ordered according to a measure of reliability
Advantages - straightforward, efficient Disadvantages - have to sort rules at
induction time
![Page 24: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/24.jpg)
Multiple-Match Resolution
Largest Rule
Similar to largest class method from no-match resolution
Choose conjunctive rule that covers the most examples in the training set.
![Page 25: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/25.jpg)
Multiple-Match Resolution
Estimation of Probability Assign EP value to each class based on the
size of the satisfied conjunctive rules.
1) Find EP for each conjunctive rule (conj):EP(conj, e)= { n(conj)/N, if conj is satisfied
by e0, otherwise
n(conj) = number of examples covered by conjN = number of total examples
![Page 26: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/26.jpg)
Multiple-Match Resolution
Estimation of Probability (2)
2) Find EP value for each class:
EP(c, e) = EP(conj1, e) + EP(conj2, e) - EP(conj1,e)EP(conj2,e).
* For more rules, apply formula recursively
* Choose class with highest EP value
![Page 27: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/27.jpg)
Hybrid Interpretation
Used because fuzzy borders only add conflicts because they don’t reduce the number rules that are applicable
HCV - use sharp borders during induction and use fuzzy borders only during deduction
Algorithm:* Single match - use class indicated by rules* Multiple match - use estimation probability
(EP) with sharp borders* No match - use fuzzy borders with polynomial
membership function to find closest rule
![Page 28: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/28.jpg)
The Data
Used 17 databases from the Machine Learning Database Repository, U. of California, Irvine.
Databases selected because:
1) All include numerical data
2) All lead to situations where no rules clearly apply.
![Page 29: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/29.jpg)
Results – Predictive Accuracy
![Page 30: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/30.jpg)
Results (cont.)
The results shown for C4.5 and NewID are the pruned ones These were usually better than the
unpruned ones in this experimentHCV did not fine tune different
parameters because this would be loss of generality and applicability of the conclusions
![Page 31: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/31.jpg)
Accuracy Results
HCV(hybrid) - 9 databasesC4.5 (R 8) - 7 databasesC4.5 (R 5) - 6 databasesHVC (large) - 3 databasesHCV (fuzzy) - 2 databases
![Page 32: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/32.jpg)
HCV Comparison
HCV (fuzzy) generally performs better than the simple largest class method
HCV’s performance improves significantly when the fuzzy borders (for no match) are combined with probability estimation (for multiple match) in HCV (hybrid)
![Page 33: Fuzzy Interpretation of Discretized Intervals Dr. Xindong Wu Andrea Porter April 11, 2002](https://reader034.vdocuments.us/reader034/viewer/2022042702/56649d585503460f94a3807f/html5/thumbnails/33.jpg)
Conclusions
Fuzzy borders are constructed and used at deduction time only when a no match case occurs.
This hybrid method performs more accurately than several other current deduction programs.
Fuzziness is strongly domain dependent, HCV allows the user to specify their own intervals and fuzzy functions.