Download - ID3 Presentation
-
8/18/2019 ID3 Presentation
1/28
ID3 Algorithm
Michael Crawford
-
8/18/2019 ID3 Presentation
2/28
Overview
ID3 Background Entropy Shannon Entropy Information ain ID3 Algorithm ID3 E!ample Clo"ing #ote"
-
8/18/2019 ID3 Presentation
3/28
ID3 Background
$Iterative Dichotomi%er 3&'
Invented (y )o"" *uinlan in +,-,'
enerate" Deci"ion .ree" u"ing Shannon
Entropy'
Succeeded (y *uinlan/" C0'1 and C1'2
algorithm"'
-
8/18/2019 ID3 Presentation
4/28
Entropy
In thermodynamic" entropy i" a mea"ureof how ordered or di"ordered a "y"tem i"'
In information theory entropy i" amea"ure of how certain or uncertain thevalue of a random varia(le i" 4or will (e5'
6arying degree" of randomne""depending on the num(er of po""i(levalue" and the total "i%e of the "et'
-
8/18/2019 ID3 Presentation
5/28
Shannon Entropy
Introduced (y Claude Shannon in +,07
*uantifie" $randomne""&
8ower value implie" le"" uncertainty 9igher value implie" more uncertainty
-
8/18/2019 ID3 Presentation
6/28
Information ain
:"e" Shannon Entropy I calculate" effective change in entropy
after making a deci"ion (a"ed on thevalue of an attri(ute'
;or deci"ion tree" it/" ideal to (a"e
deci"ion" on the attri(ute that provide" thelarge"t change in entropy the attri(utewith the highe"t gain'
-
8/18/2019 ID3 Presentation
7/28
Information ain
-
8/18/2019 ID3 Presentation
8/28
-
8/18/2019 ID3 Presentation
9/28
ID3 Algorithm
+5 E"ta(li"h Cla""ification Attri(ute 4in .a(le )5
1 until all attri(ute" have (een u"ed or the "ame
cla""ification value remain" for all row" in the reduced ta(le'
-
8/18/2019 ID3 Presentation
10/28
E!ample
-
8/18/2019 ID3 Presentation
11/28
E!ample
Model Attri(ute can (e to""ed out "inceit" alway" uni?ue and it doe"n/t help our
re"ult'
-
8/18/2019 ID3 Presentation
12/28
E!ample
E"ta(li"h a target cla""ification I" the car fa"t@ =+1 ye" ,+1 no
-
8/18/2019 ID3 Presentation
13/28
E!ample Cla""ification Entropy
Calculating for the Cla""ification Entropy
IE= -(6/15)log2(6/15)-(9/15)log2(9/15) = ~0.971
Mu"t calculate Information ain of remainingattri(ute" to determine the root node'
-
8/18/2019 ID3 Presentation
14/28
E!ample Information ain
Engine = "mall 1 medium 0 large 3 value" for attri(ute engine "o we need 3 entropy calculation"
"mall 1 no + ye" Ismall = -(5/6)log2(5/6)-(1/6)log2(1/6) =
~0.65
medium 3 no < ye" Imedium = -(3/5)log2(3/5)-(2/5)log2(2/5) =~0.97
large < no < ye" Ilarge = 1 (evenly disri!ued su!se)
IGEngine
= IE(S) – [(6/15)*Ismall
+ (5/15)*Imedium
+ (4/15)*Ilarge
]
IEngine 0.971 – 0.85 = 0.121
-
8/18/2019 ID3 Presentation
15/28
E!ample Information ain
SC.ur(o 0 ye" ++ no < value" for attri(ute SC.ur(o "o we need < entropy calculation"
ye" < ye" < no Iur!o = 1 (evenly disri!ued su!se)
no 3 ye" 7 no Inour!o = -(3/11)log2(3/11)-("/11)log2("/11) =~0."#
IGturb
= IE(S) – [(4/15)*Iturb
+ (11/15)*Inturb
]
Itur(o 0.971 – 0.886 = 0.085
-
8/18/2019 ID3 Presentation
16/28
E!ample Information ain
eight = Average 0 8ight 1 9eavy 3 value" for attri(ute weight "o we need 3 entropy calculation"
average 3 no 3 ye" Iaverage = 1 (evenly disri!ued su!se)
light 3 no + ye" Ilig$ = -(3/#)log2(3/#)-(1/#)log2(1/#) =~0."1
heavy 0 no + ye" I$eavy = -(#/5)log2(#/5)-(1/5)log2(1/5) =~0.72
IG!eig"t
= IE(S) – [(6/15)*Ia#erage
+ (4/15)*Ilig"t
+ (5/15)*I"ea#$
]
Ieight 0.971 – 0.856 = 0.115
-
8/18/2019 ID3 Presentation
17/28
E!ample Information ain
;uel Economy < good 3 average +2 (ad 3 value" for attri(ute ;uel Eco "o we need 3 entropy calculation"
good 2 ye" < no Igood = 0 (no varia!iliy)
average 2 ye" 3 no Iaverage = 0 (no varia!iliy)
(ad 1 ye" 1 no I!ad = 1 (evenly disri!ued su!se)
!e %an mit %al%ulatins &r gd and a#erage sin%e t"e$ al'a$send u nt &ast
IGuelE% = IE(S) – [(1/15)*Ibad]
I;uelEco 0.971 – 0.667 = 0.304
-
8/18/2019 ID3 Presentation
18/28
E!ample Choo"ing the )oot #ode
)ecap
IGEngine 1,1
IGturb -5
IG!eig"t 115
IGuelE%
.4
%ur !es &i' is uel *'o+ and ,e 'an immediaely &redi'$e 'ar is no as ,$en uel e'onomy is good or average.
-
8/18/2019 ID3 Presentation
19/28
E!ample )oot of Deci"ion .ree
-
8/18/2019 ID3 Presentation
20/28
-
8/18/2019 ID3 Presentation
21/28
E!ample Information ain
Engine + "mall 1 medium 0 large 3 value" for attri(ute engine "o we need 3 entropy calculation"
"mall + ye" 2 no Ismall = 0 (no varia!iliy)
medium < ye" 3 no Imedium = -(2/5)log2(2/5)-(3/5)log2(3/5) =~0.97
large < no < ye" Ilarge = 1 (evenly disri!ued su!se)
IGEngine
= IE(SuelE%
) – (5/1)*Imedium
+ (4/1)*Ilarge
]
IEngine 1 – 0.885 = 0.115
-
8/18/2019 ID3 Presentation
22/28
E!ample Information ain
SC.ur(o 3 ye" - no < value" for attri(ute SC.ur(o "o we need < entropy calculation"
ye" < ye" + no Iur!o = -(2/3)log2(2/3)-(1/3)log2(1/3) = ~0."#
no 3 ye" 0 no Inour!o = -(3/7)log2(3/7)-(#/7)log2(#/7) = ~0."#
IGturb
= IE(SuelE%
) – [(./1)*Iturb
+ (/1)*Inturb
]
Itur(o 1 – 0.965 = 0.035
-
8/18/2019 ID3 Presentation
23/28
E!ample Information ain
eight 3 average 1 heavy < light 3 value" for attri(ute weight "o we need 3 entropy calculation"
average 3 ye" 2 no Iaverage = 0 (no varia!iliy)
heavy + ye" 0 no I$eavy = -(1/5)log2(1/5)-(#/5)log2(#/5) =~0.72
light + ye" + no Ilig$ = 1 (evenly disri!ued su!se)
IGEngine = IE(Suel E%) – [(5/1)*I"ea#$+(,/1)*Ilig"t]
IEngine 1 – 0.561 = 0.439
-
8/18/2019 ID3 Presentation
24/28
E!ample Choo"ing the 8evel < #ode
)ecap
IGEngine 115
IGturb .5
IG!eig"t 4.0
eig$ $as $e $ig$es gain+ and is $us $e !es '$oi'e.
-
8/18/2019 ID3 Presentation
25/28
E!ample Deci"ion .ree
in'e $ere are only ,oiems or /ur!o ,$ere
eig$ = ig$+ and $eresul is 'onsisen+ ,e 'ansim&liy $e ,eig$ = ig$&a$.
-
8/18/2019 ID3 Presentation
26/28
E!ample :pdated .a(le
ll 'ars ,i$ large engines in $is a!le are no as.
4ue o in'onsisen &aerns in $e daa+ $ere isno ,ay o &ro'eed sin'e medium sie engines maylead o ei$er as or no as.
-
8/18/2019 ID3 Presentation
27/28
Clo"ing #ote"
ID3 attempt" to make the "horte"t
deci"ion tree out of a "et of learning data
"horte"t i" not alway" the (e"tcla""ification'
)e?uire" learning data to have completelycon"i"tent pattern" with no uncertainty'
-
8/18/2019 ID3 Presentation
28/28
)eference"
*uinlan F' ) 4+,715' Induction of Deci"ion .ree" Machine 8earning + 7+>+2= +,7='
)o"" Geter 4+232,--'
Shannon Claude E' Grediction and Entropy of Grinted Engli"h' 4)etrieved
20