iowa state universitycs573x/notes/cs573week3... · 2010-02-07 · 2 ! # department of computer...

1

Department of Computer Science

Artificial Intelligence Research Laboratory

Copyright Vasant Honavar, 2010




2







11, 01,10, 00

11 10

01 00

1 0

c=A c=B

1 0

11 01

11 01

1 0

c=A c=B

10 00

10 00

c=A c=B

x 1 0 x

Tree 1 Tree 2 Data set

x

11, 01, 10, 00

y

Exa

mples

Class

3




C




4







5







6







7







8







9




P.

P , H

P ( )




10







11







P(X = H; Y = H) = 0.2. P(X= H; Y = T) = 0.4P(X = T; Y = H) = 0.3. P(X = T; Y = T) = 0.1H(X,Y)=-0.2log20.2+… 1.85P(X = H) = 0.6. H(X) 0.97P(Y = H) = 0.5. H(Y) = 1.0P(Y= H| X = H) = 0.2/0.6 = 0.333P(Y = T|X = H) = 1-0.333=0.667P(Y= H| X = T)= 0.3/0.4=0.75P(Y= T| X = T)=0.1/0.4=0.25 H(Y|X) 0.88

12




I(X,Y)=0?




13




S1

S2

SmS

Si Ci




14




I1 (t, d, l) +I2 (s, d, l) +I3 (t, b, l) -I4 (t, r, l) -I5 (s, b, l) -I6 (t, b, w) +I7 (t, d, w) +I8 (s, b, w) +

Height (tall, short)Hair (dark, blonde, red) Eye (blue, brown)




I1…I8

Height

I1 I3 I4 I6 I7 I2 I5 I8

t s

St Ss

Hair Hair

15




Hair

rd

+ -Eye

b

wl

+




P (h) h

P (D) D

P (h | D) h D

P (D | h) D h

16







17




hMAP = argmaxh

iHP(D | hi )P(hi )

= argmaxh

iH

log2 P Dhi( ) + log2 P hi( )( )

= argminh

iH

log2 P Dhi( ) log2 P hi( )( )

log2 P hi( ) = description length of hi under optimal encoding

of hypotheses

-log2P(D | hi ) = description length of data D given hi under optimal encoding based on P(D | hi )

MAP hypothesis learning implies Occam's razor!




X,

D = {ei =(Xi , di)|Xi X di=c(Xi )}

D

VH,D

D

19







P D( ) = P(D | hh H

)P(h) =1H

h VH, D

P Xi( )i

+ 0

h VH, D

=VH ,D

HP Xi( )

i

h VH ,D, we have

P(h |D) =P Dh( )P h( )

P D( )=

P Xi( )i

H

H

VH ,D P Xi( )i

=

1VH ,D

and h VH ,D, we have P(h |D) = 0

H D

20




H




21




D

hi

h h

hi




H

22




H

H




23







24







25







26







27







28







•

•

•

29







30







31




EntropyRatio(S | A)Entropy(S | A)

SplitEntropy(S | A)

SplitEntropy(S | A)| Si || S |i=1

Values(A )log2

| Si || S |




Entropy Impurity(Z) =ZiZi=1

Classeslog2

ZiZ

Gini Impurity(Z) =ZiZ

i j

Z j

Z

=1

ZiZ

2

i=1

Classes

32




Impurity(S | A) = Min

i Values(A )Impurity Si( ){ }




•

•

•

•

•

w

ASGain

ACost 1

12

+

-

ACost

ASGain2

33




Impurity(S) = ijij

SiS

S jS




•

•

•

•

34




•

•




•

•

•

35




•

•

•




36







37




iowa state universitycs573x/notes/cs573week3... · 2010-02-07 · 2 ! # department of computer...

Documents