examples of classification methods csit5210. content knn decision tree naïve bayesian bayesian...
TRANSCRIPT
Examples of classification methods
CSIT5210
Content
• KNN• Decision Tree• Naïve Bayesian• Bayesian Belief Network• Naïve Neural Network• Multilayer Neural Network• SVM
KNN• Question: Assignment 1 Q1• Solution:
1)Understand the distance function: # of different attributes.
The distance between tuple 2 and tuple 3:
one attribute is the same and three attributes are different
Dist(2,3) = |{Height(low!=med), Weight(med!=high), BloodPressure(med!=high)}| = 3
KNN2) Calculate the Distance table.
Dist 1 2 3 4 5 6 7 8 9 10
11 4 3 2 2 3 3 2 3 2 2
12 1 4 3 3 3 3 2 3 2 4
13 4 4 2 2 4 3 2 4 4 1
14 4 3 2 2 4 3 3 3 3 1
15 2 2 4 2 4 3 3 3 4 3
16 4 3 3 3 4 3 3 3 4 2
17 0 4 4 3 3 2 2 4 3 4
18 2 4 3 2 3 2 2 3 3 3
19 3 3 3 2 2 3 2 3 3 3
20 2 4 3 4 3 1 4 4 3 3
Testing data
Training data
KNN3) For k=1 find the nearest 1 neighbor(choose the smaller id in ties), compare actual and prediction result.
Dist 1(N) 2(Y) 3(Y) 4(Y) 5(N) 6(N) 7(N) 8(Y) 9(N) 10(Y)11(Y==Y) 4 3 2 2 3 3 2 3 2 212(N==N) 1 4 3 3 3 3 2 3 2 413(Y==Y) 4 4 2 2 4 3 2 4 4 114(Y==Y) 4 3 2 2 4 3 3 3 3 115(N==N) 2 2 4 2 4 3 3 3 4 316(N!=Y) 4 3 3 3 4 3 3 3 4 217(N==N) 0 4 4 3 3 2 2 4 3 418(Y!=N) 2 4 3 2 3 2 2 3 3 319(Y==Y) 3 3 3 2 2 3 2 3 3 320(N==N) 2 4 3 4 3 1 4 4 3 3
There are 2 errors in the the 10 test data(11-20), so the error rate is 2/10 = 0.2
We choose the one with smaller id to break tie
KNN• 4) For k=3 repeat the above procedure. Use the
majority win rule to get prediction result.Dist 1(N) 2(Y) 3(Y) 4(Y) 5(N) 6(N) 7(N) 8(Y) 9(N) 10(Y)
11(Y==Y) 4 3 2 2 3 3 2 3 2 2
12(N==N) 1 4 3 3 3 3 2 3 2 4
13(Y==Y) 4 4 2 2 4 3 2 4 4 1
14(Y==Y) 4 3 2 2 4 3 3 3 3 1
15(N!=Y) 2 2 4 2 4 3 3 3 4 3
16(N!=Y) 4 3 3 3 4 3 3 3 4 2
17(N==N) 0 4 4 3 3 2 2 4 3 4
18(Y!=N) 2 4 3 2 3 2 2 3 3 3
19(Y!=N) 3 3 3 2 2 3 2 3 3 3
20(N==N) 2 4 3 4 3 1 4 4 3 3
There are 4 errors, so the error rate is 4/10 = 0.4
Decision Tree• Question: Assignment 1 Q2• Solution:– There are 6 yes and 6 no in the training data. So: Info(D) = I(6,6)
– For all the 4 attributes, calculate the Information gained by branching on it:• E.g. : If branching on the attribute “age”, the data will
be split into: • D(age=old) = {4 yes, 0 no } • D(age=young) = {2 yes, 6 no}
• Infoage(D) = 8/12 * I(2,6) + 4/12 * I(4,0)
Decision Tree
– Age has the largest gain so we choose Age as the root.
– For the age=old branch, all decisions are yes, cannot split any more.
– For the age=young branch, repeat the gain calculation again.
Decision Tree
Then we choose married for splitting. The remaining data is:Dmarried=yes = {4 approved=no}Dmarried=no = {2 approved=yes, 2 approved=no}
We choose approved=yes for the branch married=no. Here is the final tree:
Decision Tree Age
MarriedYes
Yes No
old young
no yes
Apply the tree on the testing data:error rate = 4/6 = 0.667
Naive Bayesian
• Question: Assignment 1 Q3• Answer: – In the training data, there are 6 approved=yes and 6
approved=no, so • P(C1) = P(approved=yes) = 6/12 = 0.5• P(C2) = P(approved=no) = 6/12 = 0.5
– For every attribute and class, compute P(X|Ci)• P(Sex = “male” | C1) = 4/6 = 0.667• P(Sex = “female” | C1) = 2/6 = 0.333• P(Sex = “male” | C2) = 4/6 = 0.667• P(Sex = “female” | C2) = 2/6 = 0.333
Naive Bayesian
• P(Age = “old” | C1) = 4/6 = 0.667 • P(Age = “young” | C1) = 2/6 = 0.333 • P(Age = “old” | C2) = 0/6 = 0• P(Age = “young” | C2) = 6/6 = 1
• P(Housing = “yes” | C1) = 1/6 = 0.167 • P(Housing = “no” | C1) = 5/6 = 0.833 • P(Housing = “yes” | C2) = 4/6 = 0.667• P(Housing = “no” | C2) = 2/6 = 0.333
Naive Bayesian
• P(Employed = “yes” | C1) = 4/6 = 0.667 • P(Employed = “no” | C1) = 2/6 = 0.333 • P(Employed = “yes” | C2) = 1/6 = 0.167 • P(Employed = “no” | C2) = 5/6 = 0.833
– For the first testing data:X1 = (Sex = “female”, Age = “young”, Housing = “yes”, Employed = “yes”)
P(X1|C1) = 0.333×0.333×0.167×0.667 = 0.012 P(X1|C2) = 0.333×1×0.667×0.167 = 0.037P(X1|C1) * P(C1) = 0.012 * 0,5 = 0.006P(X1|C2) * P(C2) = 0.037 * 0,5 = 0.019 > P(X|C1) * P(C1)
So X1 belongs to C2 (Approved=no)
Naive Bayesian
• For the remaining testing data, repeat the same procedure.
• So the error rate is 2/3 = 0.667
id sex age housing employed Approved(actual) prediction
13 female young yes yes yes no
14 male young yes yes yes no
15 female young yes no no no
Bayesian Network
• Question:• Smoking is prohibited on High-Speed trains. If
someone smokes, the alarm may sound, and also other passengers may report it to the police. If the police hear the alarm or get the report, he will, very possibly, come and arrest the smoker.
• This can be modeled in the following Bayes network:
Bayesian Network
Smoking
ReportAlarm
Police comes
The police comes if he believes there is someone smoking. They don’t trust the alarm very much and they may, rarely, patrol on the train.
S P(A=F) P(A=T)
T 0.4 0.6
F 0.8 0.2
A R P(P=T) P(P=F)
T T 0.8 0.2
F T 0.6 0.4
T F 0.4 0.6
F F 0.01 0.99
S P(R=F) P(R=T)
T 0.6 0.4
F 0.9 0.1
The alarm is not accurate enough, it ignores some smoking and sometimes sound for nothing.
Not every passenger want to report smokers and some passengers make mistakes.(The alarm does not affect passengers.)
Bayesian Network
Suppose the probability of someone smoking is 0.5, what is the probability of the police comes?• Answer:
– P(S=T)=0.5 and P(S=F)=0.5,
– The alarm sounds: • P(A) = P(A|S)*P(S) + P(A|¬S)*P(¬S) = 0.4
– Passengers report:• P(R) = P(R|S)*P(S) + P(R|¬S)*P(¬S) = 0.25
– Police comes:P(P) = P(P|AR) * P(A) * P(R) + P(P|A¬R) * P(A) * P(¬R) +
P(P|¬AR) * P(¬A) * P(R) + P(P|¬A¬R) * P(¬A) * P(¬R)= 0.08 + 0.12 + 0.09 + 0.0045 = 0.2945
Naïve Neural Network
• Question: • Given a perceptron, the training samples are given in the
table below.
• In addition, the initial weights are also given: W0=0.5, W1=0.4, W2=0.5. The learning rate α is 0.2. Please use the sample data as training data and update W0, W1, and W2.
Naïve Neural Network
• Answer:Step 1: – a = = -0.5 +0.4*0 + 0.5*0 = -0.5 < 0– y = 0 = T1, so no need to change weights.
Step 2: – a = = -0.5 +0.4*0 + 0.5*1 = 0– y = 1 = T2, so no need to change weights.
Naïve Neural Network
• Step 3: – a = = -0.5 +0.4*1 + 0.5*1 = -0.1 < 0– y = 0 ≠ T3,
– ∆w0 = α (t-y) x0 = 0.2 * 1 * (-1) = -0.2
– ∆w1 = α (t-y) x1 = 0.2 * 1 * 1 = 0.2
– ∆w2 = α (t-y) x2 = 0.2 * 1 * 0 =0
• Thus, – w0 = w0 + ∆w0 =0.5 – 0.2 =0.3
– w1 = w1 + ∆w1 = 0.4 + 0.2 = 0.6
– w2 = w2 + ∆w2 = 0.5
Naïve Neural Network
• Step 4– a = = -0.3 +0.6*1 + 0.5*1 = 0.8 > 0– Y = 1 = T4 , no need to change weights.
• So, the final weights are: – w0 =0.3
– w1 =0.6
– w2 =0.5
Multilayer Neural Network
• Given the following neural network with initialized weights as in the picture(next page), we are trying to distinguish between nails and screws and an example of training tuples is as follows: – T1{0.6, 0.1, nail}– T2{0.2, 0.3, screw}
• Let the learning rate (l) be 0.1. Do the forward propagation of the signals in the network using T1 as input, then perform the back propagation of the error. Show the changes of the weights. Given the new updated weights with T1, use T2 as input, show whether the predication is correct or not.
Multilayer Neural Network
Multilayer Neural Network
• Answer:• First, use T1 as input and then perform the
back propagation.– At Unit 3:• a3 =x1w13 +x2w23+θ3 =0.14
• o3 = = 0.535
– Similarly, at Unit 4,5,6: • a4 = 0.22, o4 = 0.555
• a5 = 0.64, o5 = 0.655
• a6 = 0.1345, o6 = 0.534
Multilayer Neural Network
• Now go back, perform the back propagation, starts at Unit 6:– Err6 = o6 (1- o6) (t- o6) = 0.534 * (1-0.534)*(1-0.534) = 0.116 – ∆w36 = (l) Err6 O3 = 0.1 * 0.116 * 0.535 = 0.0062– w36 = w36 + ∆w36 = -0.394– ∆w46 = (l) Err6 O4 = 0.1 * 0.116 * 0.555 = 0.0064 – w46 = w46 + ∆w46 = 0.1064 – ∆w56 = (l) Err6 O5 = 0.1 * 0.116 * 0.655 = 0.0076 – w56 = w56 + ∆w56 = 0.6076– θ6 = θ6 + (l) Err6 = -0.1 + 0.1 * 0.116 = -0.0884
Multilayer Neural Network• Continue back propagation: • Error at Unit 3:
Err3 = o3 (1- o3) (w36 Err6) = 0.535 * (1-0.535) * (-0.394*0.116) = -0. 0114
w13 = w13 + ∆w13 = w13 + (l) Err3X1 = 0.1 + 0.1*(-0.0114) * 0.6 = 0.09932
w23 = w23 + ∆w23 = w23 + (l) Err3X2 = -0.2 + 0.1*(-0.0114) * 0.1 = -0.2001154
θ3 = θ3 + (l) Err3 = 0.1 + 0.1 * (-0.0114) = 0.09886• Error at Unit 4:
Err4 = o4 (1- o4) (w46 Err6) = 0.555 * (1-0.555) * (-0.1064*0.116) = 0.003
w14 = w14 + ∆w14 = w14 + (l) Err4X1 = 0 + 0.1*(-0.003) * 0.6 = 0.00018
w24 = w24 + ∆w24 = w24 + (l) Err4X2 = 0.2 + 0.1*(-0.003) * 0.1 = 0.20003
θ4 = θ4 + (l) Err4 = 0.2 + 0.1 * (0.003) = 0.2003• Error at Unit 5:
Err5 = o5 (1- o5) (w56 Err6) = 0.655 * (1-0.655) * (-0. 6076*0.116) = 0.016
w15 = w15 + ∆w15 = w15 + (l) Err5X1 = 0.3 + 0.1* 0.016 * 0.6 = 0.30096
w25 = w25 + ∆w25 = w25 + (l) Err5X2 = -0.4 + 0.1*0.016 * 0.1 = -0.39984
θ5= θ5 + (l) Err5 = 0.5 + 0.1 * 0.016 = 0.5016
Multilayer Neural Network
• After T1, the updated values are as follows:
• Now, with the updated values, use T2 as input:– At Unit 3:– a3 = x1w13 + x2w23 + θ3 = 0.0586898
– o3 = = 0.515
Multilayer Neural Network
• Similarly, • a4 = 0.260345, o4 = 0.565
• a5 = 0.441852, o5 = 0.6087
• At Unit 6:– a6 = x3w36 + x4w46 + x5w56 + θ6 = 0.13865
– o6 = = 0.5348
• Since O6 is closer to 1, so the prediction should be nail, different from given “screw”.
• So this predication is NOT correct.
SVM• Consider the following data points. Please use SVM to train a classifier,
and then classify these data points. Points with ai=1 means this point is support vector. For example, point 1 (1,2) is the support vector, but point 5 (5,9) is not the support vector.
• Training data:
• Testing data:
SVM
• Question: – (a) Find the decision boundary, show detail
calculation process. – (b) Use the decision boundary you found to
classify the Testing data. Show all calculation process in detail, including the intermediate result and the formula you used.
SVM• Answer:• a) As the picture shows, P1, P2, P3 are support vectors.
SVM• Suppose w is (w1,w2). Since both P1(1,2) and P3(0,1) have y = 1, while
P2(2,1) has y =-1:– w1*1+w2*2+b = 1
– w1*0+w2*1+b = 1
– w1*2+w2*1+b =-1
w1= -1, w2 = 1, b = 0
then, the decision boundary is: – w1 * x1+w2 * x2 + b =0
-x1+x2 = 0
• Showed in the picture next page.
SVM
SVM
• b) Use the decision boundary to classify the testing data:– For the point P9 (2,5)
-x1+x2 = -2+5 = 3 >= 1
So we choose y = 1– For the point P10 (7,2)
-x1+x2 = -7+2 = -5 <= -1
So we choose y = -1 • Showed in the picture next page.
SVM
Q&A