chapter 12 machine learning id: 116 117 name: qun yu (page 1-33) kai zhu (page 34-59) class: cs267...

Chapter 12Machine Learning

ID: 116 117Name: Qun Yu (page 1-33) Kai Zhu (page 34-59)Class: CS267 Fall 2008Instructor: Dr. T.Y.Lin

Introduction

Machine Learning is a key part of A.I. research.

Rough Set Theory can be used for some problems in Machine Learning.

Will discuss 2 cases:1) Learning from Examples2) An Imperfect Teacher

Case1:Learning from Examples

We assume: 2 agents: a knower, a learner Knower’s Knowledge set U U is unchanged, will NOT increase during the

learning process. It is called CWA(Closed World Assumption)

Knower knows everything about U Learner has ability to learn U, in other words,

learner knows some attributes of objects in U

Example For Case 1 KR-System

U a b c d e

1 1 2 0 1 1

2 1 2 0 1 1

3 2 0 0 1 0

4 0 0 1 2 1

5 2 1 0 2 1

6 0 0 1 2 2

7 2 0 0 1 0

8 0 1 2 2 1

9 2 1 0 2 2

10 2 0 0 1 0

Attributes of learner’s knowledge : B={a,b,c,d}

Attributes of knower’s knowledge : e

Example For Case 1 KR-System

U a b c d e

1 1 2 0 1 1

2 1 2 0 1 1

3 2 0 0 1 0

4 0 0 1 2 1

5 22 11 00 22 1

6 0 0 1 2 2

7 2 0 0 1 0

8 0 1 2 2 1

9 22 11 00 22 2

10 2 0 0 1 0

3 concepts of knower’s knowledge:

X0={3,7,10}

X1={1,2,4,5,8}

X2={6,9}

5 concepts of learner’s knowledge:

Y0={1,2}

Y1={3,7,10}

Y2={4,6}

YY3={5,9}={5,9}

Y4={8}

Which Objects are learnable?X0={3,7,10}

X1={1,2,4,5,8}

X2={6,9}

Y0={1,2}

Y1={3,7,10}

Y2={4,6}

Y3={5,9}

Y4={8}

BX0 = Y1 = {3,7,10} = X0 = BX0

X0 is exactly Y-definable and can be learned fully.

BX1 = Y0 Y4 = {1,2,8}

BX1 = Y0 Y2 Y3 Y4 = {1,2,4,5,6,8,9}

X1 is roughly Y-definable, so learner can learn object {1,2,8}, not sure about {4,5,6,9}.

BX2 = Ø

BX1 = Y2 Y3 = {4,5,6,9}

X2 is internally Y-indefinable, so it is NOT learnable.

Learnable:Learnable: The knower’s knowledge can be expressed in terms of learner’s knowledge.

Quality of Learning?BX0 = Y1 = {3,7,10} = X0 = BX0

BX1 = Y0 Y4 = {1,2,8}

BX1 = Y0 Y2 Y3 Y4 = {1,2,4,5,6,8,9}

BX2 = Ø

BX1 = Y2 Y3 = {4,5,6,9}

X0 X1 X2

Positive objects 3,7,10 1,2,8 Ø

Border-line objects Ø 3,7,10 4,5,6,9

Negative objects 1,2,4,5,6,8,9 4,5,6,9 1,2,3,7,8,10

POSB{e}=POSB(X0) POSB(X1)

POSB(X2) = {1,2,3,7,8,10} (6 objects)

U={1,2,3,4,5,6,7,8,9,10} (10 objects)

{e}= 6/10 = 0.6

Simplifier table…step 1U a b c d e

1 1 2 0 1 1

2 1 2 0 1 1

3 2 0 0 1 0

4 0 0 1 2 1

5 22 11 00 22 1

6 0 0 1 2 2

7 2 0 0 1 0

8 0 1 2 2 1

9 22 11 00 22 2

10 2 0 0 1 0

Group the duplicate rows :

U a b c d e

3,7,10

2 0 0 1 0

1,2 1 2 0 1 1

4 0 0 1 2 1

5 22 11 00 22 1

8 0 1 2 2 1

6 0 0 1 2 2

9 22 11 00 22 2

Simplifier table…step 2Remove inconsistent rows :

U a b c d e

3,7,10

2 0 0 1 0

1,2 1 2 0 1 1

8 0 1 2 2 1

U a b c d e

3,7,10

2 0 0 1 0

1,2 1 2 0 1 1

4 0 0 1 2 1

5 22 11 00 22 1

8 0 1 2 2 1

6 0 0 1 2 2

9 22 11 00 22 2

Simplifier table…step 3

{a}, {b}

Find attributes reduct :U a b c d e

3,7,10

2 0 0 1 0

1,2 1 2 0 1 1

8 0 1 2 2 1

U b c d e

3,7,10

0 0 1 0

1,2 2 0 1 1

8 1 2 2 1U a c d e

3,7,10

2 0 1 0

1,2 1 0 1 1

8 0 2 2 1

U a b d e

3,7,10

2 0 1 0

1,2 1 2 1 1

8 0 1 2 1U a b c e

3,7,10

2 0 0 0

1,2 1 2 0 1

8 0 1 2 1

Delete a:

Delete b:

Delete c:

Delete d:

Simplifier table…step 4Find value reduct & decision rules:

3,7,10

1,2 1 1

3,7,10

1,2 2 1

8 1 1a2e0

a1Va0e1

b2Vb1e1

All objects are necessary in learning process?

From the KR-System table, we can see some objects are same by attributes {a,b,c,d,e}:

3 = 7 =10

D1={1,2}

D2={3,7,10}

Therefore, only one object, either 1 or 2 in D1 is necessary in learning process. So is D2.

4,5,6,8,9 are necessary.

U a b c d e

1 1 2 0 1 1

2 1 2 0 1 1

3 2 0 0 1 0

4 0 0 1 2 1

5 2 1 0 2 1

6 0 0 1 2 2

7 2 0 0 1 0

8 0 1 2 2 1

9 2 1 0 2 2

10 2 0 0 1 0

Prove from Decision Rule…(1)

U a b c d e

1 1 2 0 1 1

3 2 0 0 1 0

4 0 0 1 2 1

5 22 11 00 22 1

6 0 0 1 2 2

8 0 1 2 2 1

9 22 11 00 22 2

Remove objects 2,7 and 10 (keep 1 and 3) as Table1,Table1 will provide the same decision rules.

Table1

U a b c d e

2 1 2 0 1 1

4 0 0 1 2 1

5 22 11 00 22 1

6 0 0 1 2 2

8 0 1 2 2 1

9 22 11 00 22 2

10 2 0 0 1 0

Remove objects 1,3 and 7 (keep 2 and 10)as Table 2,Table2 will provide the same decision rules.

Table 2

U a b c d e

1 1 2 0 1 1

2 1 2 0 1 1

3 2 0 0 1 0

5 22 11 00 22 1

6 0 0 1 2 2

7 2 0 0 1 0

9 22 11 00 22 2

10 2 0 0 1 0

Remove objects 4 and 8 as Table3,

The whole new decision algorithm will be:

Table 3

Now concept X2 Now is internally definable. Object 6 is learnable now. So decision algorithm changed.

b2c0 e1

b0c0 e0

b0c1 e2

b2d1 e1

b0d1 e0

b0d2 e2

U a b c d e

1 1 2 0 1 1

2 1 2 0 1 1

3 2 0 0 1 0

4 0 0 1 2 1

5 22 11 00 22 1

6 0 0 1 2 2

7 2 0 0 1 0

8 0 1 2 2 1

10 2 0 0 1 0

Table 4

Remove objects 9 as Table 4,

The whole new decision algorithm will be:

a0b1 e1

a2b1 e1Now object 5 is positive object of X1. So decision algorithm changed.

a2d1 e0

a1d1 e1

a2d2 e1

a0d2 e1

Case2: An Imperfect TeacherWe assume:

2 agents: a knower, a learner

Knower’s Knowledge set U

U is unchanged, will NOT increase during the learning process. It is called CWA(Closed World Assumption)

Knower knows everything about U

Learner has ability to learn U, in other words, learner knows some attributes of objects in U

Example 1 For Case 2 KR-System

U a b c

1 0 2 +

2 0 1 +

3 1 0 +

4 0 1 0

5 1 0 0

6 1 1 -

7 2 1 -

8 0 1 -

9 1 0 -

Attributes of learner’s knowledge : B={a,b}

Attributes of knower’s knowledge : c

Attribute value 0: knower can’t classify this object.

Example 1 For Case 2KR-System

X0={4,5} ignorance region

X+={1,2,3}

X-={6,7,8,9}

X*=X+ X- competence region5 concepts of learner’s knowledge:

Y0={1}

Y1={2,4,8}

Y2={3,5,9}

YY3={6}={6}

Y4={7}

U a b c

1 0 2 +

2 0 1 +

3 1 0 +

4 0 1 0

5 1 0 0

6 11 11 -

7 2 1 -

8 0 1 -

9 1 0 -

Can the learner be able to discover the ignorance region?

BX+ = Y0= {1}

BX+ = Y0 Y1 Y2 = {1,2,3,4,5,8,9}

X+ is roughly Y-definable, so learner can learn object {1}, not sure about {2,3,4,5,8,9}.

BX- = Y3 Y4 = {6,7}

BX- = Y1 Y2 Y3 Y4 = {2,3,4,5,6,7,8,9}

X- is roughly Y-definable, so learner can learn object {6,7}, not sure about {2,3,4,5,8,9}.

BX0 = Ø

BX0 = Y1 Y2 = {2,3,4,5,8,9}

X0 is internally Y-indefinable, so it is NOT learnable.

X+={1,2,3}

X-={6,7,8,9}

X*=X+ X- competence region

Y0={1}

Y1={2,4,8}

Y2={3,5,9}

Y3={6}

Y4={7} Learner can’t discover the ignorance region X0

The ignorance region influences the learner’s ability to learn?

In this example, the answer is NO, because the ignorance region(X0) is not learnable.

Hence, we can prove it:

BNB(X*)= BNB(X+) BNB(X-)

BX+ = Y0= {1}

BX+ = Y0 Y1 Y2 = {1,2,3,4,5,8,9}

BX- = Y3 Y4 = {6,7}

BX- = Y1 Y2 Y3 Y4 = {2,3,4,5,6,7,8,9}

X0={4,5} ignorance regionX* X*

put 4 into X+

X*put 4 into X-

Positive objects

1,6,7 1,6,7 1,6,7

Border-line objects

2,3,4,5,8,9 2,3,4,5,8,9 2,3,4,5,8,9

Negative objects

Ø Ø Ø

Border-line region of the competence region X* remain unchanged, therefore, it doesn't matter knower knows it or NOT.

Put 4 into X+

X+={1,2,3,4}

X-={6,7,8,9}

X0={5}

BX+ = Y0= {1}

BX+ = Y0 Y1 Y2 = {1,2,3,4,5,8,92,3,4,5,8,9}

BX- = Y3 Y4 = {6,7}

BX- = Y1 Y2 Y3 Y4 = {2,3,4,52,3,4,5,6,7,8,98,9}

BNB(X*)= BNB(X+) BNB(X-)

={2,3,4,5,8,9} {2,3,4,5,8,9}

={2,3,4,5,8,9}

X0={4,5} ignorance

region

X+={1,2,3}

X-={6,7,8,9}

X*=X+ X- competence

region

Y0={1}

Y1={2,4,8}

Y2={3,5,9}

Y3={6}

Y4={7}

Calculation Sheet:Border-line objects

U a b c

1 0 2 +

6 1 1 -

7 2 1 -

U a b c

1 0 2 +

2 0 1 +

3 1 0 +

4 0 1 0

5 1 0 0

6 11 11 -

7 2 1 -

8 0 1 -

9 1 0 -

Learner can’t discover the ignorance region {4,5} and the ignorance region won’t affect the learning process.

{a}, {b}

Find attributes reduct :

Delete a:

Delete b:

U a b c

1 0 2 +

6 1 1 -

7 2 1 -U a c

Example 2 For Case 2 KR-System

U a b c

1 0 2 +

2 0 1 +

3 1 0 +

4 0 1 +

5 1 0 0

6 1 1 0

7 2 1 -

8 0 1 -

9 1 0 -

Attributes of learner’s knowledge : B={a,b}

Attributes of knower’s knowledge : c

Attribute value 0: knower can’t classify this object.

Example 2 For Case 2KR-System

X+={1,2,3,4}

X-={7,8,9}

X*=X+ X- competence region5 concepts of learner’s knowledge:

Y0={1}

Y1={2,4,8}

Y2={3,5,9}

YY3={6}={6}

Y4={7}

U a b c

1 0 2 +

2 0 1 +

3 1 0 +

4 0 1 +

5 1 0 0

6 11 11 0

7 2 1 -

8 0 1 -

9 1 0 -

Can the learner be able to discover the ignorance region?

BX+ = Y0= {1}

BX+ = Y0 Y1 Y2 = {1,2,3,4,5,8,9}

X+ is roughly Y-definable, so learner can learn object {1}, not sure about {2,3,4,5,8,9}.

BX- = Y4 = {7}

BX- = Y1 Y2 Y4 = {2,3,4,5,7,8,9}

X- is roughly Y-definable, so learner can learn object {7}, not sure about {2,3,4,5,8,9}.

BX0 = Y3= {6}

BX0 = Y2 Y3 = {3,5,6,9}

X0 is roughly Y-definable, so learner can learn object {6}, not sure about {3,5,9}.

X+={1,2,3,4}

X-={7,8,9}

Y0={1}

Y1={2,4,8}

Y2={3,5,9}

Y3={6}

Y4={7} Learner can discover the ignorance region X0

The ignorance region influences the learner’s ability to learn?

In this example, the answer is YES, because the ignorance region(X0) is roughly learnable, object 6 is important. Hence, we can prove it:

BX+ = Y0= {1}

BX+ = Y0 Y1 Y2 = {1,2,3,4,5,8,9}

BX- = Y4 = {7}

BX- = Y1 Y2 Y4 = {2,3,4,5,7,8,9}

X* X* put 6 into X+

X*put 5 into X-

Positive objects

1, 7 1,6,7 1, 7

Border-line objects

2,3,4,5,8,9 2,3,4,5,8,9 2,3,4,5,8,9

Negative objects

6 Ø 6

Object 6 will fall in positive region if knower knows it (move from X0 to X+).

Therefore, X0 affects the learning process.BNB(X*)= BNB(X+) BNB(X-)

Put 6 into X+

X+={1,2,3,4,6}

X-={7,8,9}

X0={5}

BX+ = Y0= {1,6}

BX+ = Y0 Y1 Y2 Y3 = {1,2,3,4,5,2,3,4,5,6,8,98,9}

BX- = Y4 = {7}

BX- = Y1 Y2 Y4 = {2,3,4,52,3,4,5,7,8,98,9}

POSB(X*)= POSB(X+) POSB(X-)

={1,6} {7}

={1,6,7}

X0={5,6} ignorance

region

X+={1,2,3,4}

X-={7,8,9}

X*=X+ X- competence

region

Y0={1}

Y1={2,4,8}

Y2={3,5,9}

Y3={6}

Y4={7}

Calculation Sheet:Positive objects

U a b c

1 0 2 +

6 1 1 0

7 2 1 -

Object 6 is learnable, therefore learner can discover the ignorance region {4,5} and the ignorance region will affect the learning process.

U a b c

1 0 2 +

2 0 1 +

3 1 0 +

4 0 1 +

5 1 0 0

6 11 11 0

7 2 1 -

8 0 1 -

9 1 0 -

Find attributes reduct :

Delete a:

Delete b:

U a b c

1 0 2 +

6 1 1 0

7 2 1 -U a c

A is indispensable

Object 6 is learnable, so learner can discover the knower’s ignorance.

Chapter 12 Machine LearningChapter 12 Machine LearningInductive LearningInductive Learning

Presented by: Kai ZhuPresented by: Kai Zhu

Professor: Dr. T.Y. LinProfessor: Dr. T.Y. Lin

Class ID: 117Class ID: 117

In the pervious chapters, we assumed that the set of In the pervious chapters, we assumed that the set of instances U is constant and unchanged during the instances U is constant and unchanged during the

learning process. learning process.

In any real life situations however this is not the In any real life situations however this is not the case and new instances can be added to the set U. case and new instances can be added to the set U.

Example Example

Lets consider the K-R system Lets consider the K-R system given in Example 1 in the given in Example 1 in the

pervious section in chapter 12pervious section in chapter 12

Table 9Table 9

U a b c d e

1 1 2 0 1 1

2 1 2 0 1 1

3 2 0 0 1 0

4 0 0 1 2 1

5 2 1 0 2 1

6 0 0 1 2 2

7 2 0 0 1 0

8 0 1 2 2 1

9 2 1 0 2 2

10 2 0 0 1 0

Consistent table:Consistent table:

U a b c d e

1 1 2 0 1 1

2 1 2 0 1 1

3 2 0 0 1 0

7 2 0 0 1 0

8 0 1 2 2 1

10 2 0 0 1 0

Inconsistent table:Inconsistent table:U a b c d e

4 0 0 1 2 1

5 2 1 0 2 1

6 0 0 1 2 2

9 2 1 0 2 2

(e)=card(1,2,3,7,8,10)/card(1,2,3,4,5,6,7,8,9,10)=6/10=0.6

After remove duplicate After remove duplicate and inconsistentand inconsistent

U a b c d e

1 1 2 0 1 1

2 2 0 0 1 0

3 0 1 2 2 1

After computing, get core After computing, get core and reduct values tableand reduct values table

U a b c d e

1 - - - - 1

2 - - - - 0

3 - - - - 1

U a b c d e

1(1) 1 x x x 1

1(2) x 2 x x 1

2(1) 2 x x x 0

2(2) x 0 x x 0

3(1) 0 x x x 1

3(2) x 1 x x 1

3(3) x x 2 x 1

3(4) x x x 2 1

Core tableCore table

Reduct tableReduct table

From reduct table, there are 16 From reduct table, there are 16 Simplified Tables. One of them : Simplified Tables. One of them :

corresponding decision corresponding decision algorithms:algorithms:

U a b c d e

1(1) 1 x x x 1

2(1) 2 x x x 0

3(1) 0 x x x 1

a1a1 e1 e1a2 a2 e0e0a0 a0 e1e1

Table 10Table 10

U a b c d

1 1 2 0 1 1

2 1 2 0 1 1

3 2 0 0 1 0

4 0 0 1 2 1

5 2 1 0 2 1

6 0 0 1 2 2

7 2 0 0 1 0

8 0 1 2 2 1

9 2 1 0 2 2

10 2 0 0 1 0

11 0 1 2 2 1

Inconsistent table:Inconsistent table:

(e)=card(1,2,3,7,8,10,11)/card(1,2,3,4,5,6,7,8,9,10,11)=7/11=0.636

U a b c d

1 1 2 0 1 1

2 1 2 0 1 1

3 2 0 0 1 0

7 2 0 0 1 0

8 0 1 2 2 1

10 2 0 0 1 0

11 0 1 2 2 1

U a b c d

4 0 0 1 2 1

5 2 1 0 2 1

6 0 0 1 2 2

9 2 1 0 2 2

U a b c de

1 1 2 0 1 1

2 2 0 0 1 0

3 0 1 2 2 1

U a b c d e

1 - - - - 1

2 - - - - 0

3 - - - - 1

U a b c d e

1(1) 1 x x x 1

1(2) x 2 x x 1

2(1) 2 x x x 0

2(2) x 0 x x 0

3(1) 0 x x x 1

3(2) x 1 x x 1

3(3) x x 2 x 1

3(4) x x x 2 1

U a b c d e

1(1) 1 x x x 1

2(1) 2 x x x 0

3(1) 0 x x x 1

a1 a1 e1 e1a2 a2 e0e0a0 a0 e1e1

It is obvious that in table 10 the new instance does It is obvious that in table 10 the new instance does not change the decision algorithm, that means that not change the decision algorithm, that means that

the learned concepts will remain the same.the learned concepts will remain the same.

But the quality of learning But the quality of learning (e) changed. (from 0.6 (e) changed. (from 0.6 to 0.636)to 0.636)

Table 11Table 11

U a b c d

1 1 2 0 1 1

2 1 2 0 1 1

3 2 0 0 1 0

4 0 0 1 2 1

5 2 1 0 2 1

6 0 0 1 2 2

7 2 0 0 1 0

8 0 1 2 2 1

9 2 1 0 2 2

10 2 0 0 1 0

11 1 2 0 1 0

(e)=card(3,7,8,10)/card(1,2,3,4,5,6,7,8,9,10,11)=4/11=0.363

U a b c d

3 2 0 0 1 0

7 2 0 0 1 0

8 0 1 2 2 1

10 2 0 0 1 0

U a b c d

1 1 2 0 1 1

2 1 2 0 1 1

4 0 0 1 2 1

5 2 1 0 2 1

6 0 0 1 2 2

9 2 1 0 2 2

11 1 2 0 1 0

U a b c de

1 2 0 0 1 0

2 0 1 2 2 1

a b c d

- - - - 0

- - - - 1

U a b c d

1(1) 2 x x x 0

1(2) x 0 x x 0

1(3) x x 0 x 0

1(4) x x x 1 0

2(1) 0 x x x 1

2(2) x 1 x x 1

2(3) x x 2 x 1

2(4) x x x 2 1

U a b c d

1(1) 2 x x x 0

2(1) 0 x x x 1

a2 a2 e0e0a0 a0 e1e1 (Compare to(Compare to

a1a1 e1 e1a2 a2 e0e0a0 a0 e1)e1)

Table 12Table 12

U a b c d

1 1 2 0 1 1

2 1 2 0 1 1

3 2 0 0 1 0

4 0 0 1 2 1

5 2 1 0 2 1

6 0 0 1 2 2

7 2 0 0 1 0

8 0 1 2 2 1

9 2 1 0 2 2

10 2 0 0 1 0

11 1 0 0 1 3

(e)=card(1,2,3,7,8,10,11)/card(1,2,3,4,5,6,7,8,9,10)=6/10=0.636

U a b c d

1 1 2 0 1 1

2 1 2 0 1 1

3 2 0 0 1 0

7 2 0 0 1 0

8 0 1 2 2 1

10 2 0 0 1 0

11 1 0 0 1 3

U a b c d

4 0 0 1 2 1

5 2 1 0 2 1

6 0 0 1 2 2

9 2 1 0 2 2

U a b c de

1 1 2 0 1 1

2 2 0 0 1 0

3 0 1 2 2 1

4 1 0 0 1 3

U a b c d

1 - 2 - - 1

2 2 - - - 0

3 - - - - 1

4 1 0 - - 3

U a b c d

1(1) x 2 x x 1

2(1) 2 x x x 0

3(1) 0 x x x 1

3(2) x 1 x x 1

3(3) x x 2 x 1

3(4) x x x 2 1

4(1) 1 0 x x 3

U a b c d

1(1) x 2 x x 1

2(1) 2 x x x 0

3(1) 0 x x x 1

4(1) 1 0 x x 3b2 b2 e1e1a2 a2 e0e0a0a0e1e1

a1b0a1b0e3e3

(Compare to(Compare toa1a1 e1 e1a2 a2 e0e0a0 a0 e1)e1)

To sum up, as we have seen from the above To sum up, as we have seen from the above examples, adding a new instance to the universe examples, adding a new instance to the universe

we could face three possibilities:we could face three possibilities:

1. The new instance confirms actual knowledge. 1. The new instance confirms actual knowledge. (table10)(table10)

2. The new instance contradicts the actual 2. The new instance contradicts the actual knowledge. (table11)knowledge. (table11)

3. The new instance is a completely new case.3. The new instance is a completely new case.

(table12)(table12)

Thank youThank you

chapter 12 machine learning id: 116 117 name: qun yu (page 1-33) kai zhu (page 34-59) class: cs267...

Documents

bob lucas nersc/~dhbailey/cs267

david h. bailey based on notes by j. demmel and d. culler...

chapter 9 data analysis cs267 by anand sivaramakrishnan

8. qun liu (dcu) hybrid solutions for translation

cs267 applications of parallel computers

qun wang/ustc/china1 shear and bulk viscosity of qgp in pqcd...

chapter 15.7 buffer management id: 219 name: qun yu class:...

titanium 1 cs267 lecture 8 global address space programming...

bob lucas based on notes by j. demmel and d. culler...

2/19/97 horst d. simon cs.berkeley/cs267

chapter 8 decision making cs267 by gayatri bogineri (12) &...

cs267 l1 introdemmel sp 1999 cs267 / e233 applications of...

qun t essence

comparison and performance evaluation of san file system...

performance debugging techniques for hpc applications david...

cs267 applications of parallel computers lecture 1: ...

cs267 assignment 3: parallelize graph algorithms for de novo...

cs267 – lecture 14 automatic performance tuning and...

cs267 l1 intro cs267 applications of parallel computers...

btrees & bitmap indexes 14.2, 14.7 database systems – the...