© tan,steinbach, kumar introduction to data mining 4/18/2004 1 support vector machines l find a...
TRANSCRIPT
![Page 1: © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 Support Vector Machines l Find a linear hyperplane (decision boundary) that will separate](https://reader030.vdocuments.us/reader030/viewer/2022032701/56649ca25503460f94961667/html5/thumbnails/1.jpg)
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1
Support Vector Machines
Find a linear hyperplane (decision boundary) that will separate the data
![Page 2: © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 Support Vector Machines l Find a linear hyperplane (decision boundary) that will separate](https://reader030.vdocuments.us/reader030/viewer/2022032701/56649ca25503460f94961667/html5/thumbnails/2.jpg)
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 2
Support Vector Machines
One Possible Solution
B1
![Page 3: © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 Support Vector Machines l Find a linear hyperplane (decision boundary) that will separate](https://reader030.vdocuments.us/reader030/viewer/2022032701/56649ca25503460f94961667/html5/thumbnails/3.jpg)
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 3
Support Vector Machines
Another possible solution
B2
![Page 4: © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 Support Vector Machines l Find a linear hyperplane (decision boundary) that will separate](https://reader030.vdocuments.us/reader030/viewer/2022032701/56649ca25503460f94961667/html5/thumbnails/4.jpg)
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 4
Support Vector Machines
Other possible solutions
B2
![Page 5: © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 Support Vector Machines l Find a linear hyperplane (decision boundary) that will separate](https://reader030.vdocuments.us/reader030/viewer/2022032701/56649ca25503460f94961667/html5/thumbnails/5.jpg)
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 5
Support Vector Machines
Which one is better? B1 or B2? How do you define better?
B1
B2
![Page 6: © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 Support Vector Machines l Find a linear hyperplane (decision boundary) that will separate](https://reader030.vdocuments.us/reader030/viewer/2022032701/56649ca25503460f94961667/html5/thumbnails/6.jpg)
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 6
Support Vector Machines
Find hyperplane maximizes the margin => B1 is better than B2
B1
B2
b11
b12
b21
b22
margin
![Page 7: © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 Support Vector Machines l Find a linear hyperplane (decision boundary) that will separate](https://reader030.vdocuments.us/reader030/viewer/2022032701/56649ca25503460f94961667/html5/thumbnails/7.jpg)
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 7
| |
b
w
X1
X2
x
w
0w x b
| || | cos 0
| | cos| |
x w b
bx
w
| |
b
w
此條線為由原點沿 w
方向走 與 垂直的線w
![Page 8: © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 Support Vector Machines l Find a linear hyperplane (decision boundary) that will separate](https://reader030.vdocuments.us/reader030/viewer/2022032701/56649ca25503460f94961667/html5/thumbnails/8.jpg)
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 8
Support Vector Machines
B1
b11
b12
0 bxw
1 bxw 1 bxw
1bxw if1
1bxw if1)(
xf
2Margin
|| ||w
![Page 9: © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 Support Vector Machines l Find a linear hyperplane (decision boundary) that will separate](https://reader030.vdocuments.us/reader030/viewer/2022032701/56649ca25503460f94961667/html5/thumbnails/9.jpg)
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 9
w
dx1
x2
x1-x2
w•x+b = -1
w•x+b=1
W•X +b =0
w•x+b=0
0
此線在沿法向量走 -b/|w| 的距離,與法向量垂直
![Page 10: © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 Support Vector Machines l Find a linear hyperplane (decision boundary) that will separate](https://reader030.vdocuments.us/reader030/viewer/2022032701/56649ca25503460f94961667/html5/thumbnails/10.jpg)
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 10
開始推導 w•x+b=0 這是一條怎樣的線呢? xa , xb 為線上兩點 w•xa+b=0,
w•xb+b=0
兩式相減 w •(xa-xb)=0 線上兩點所形成的向量與 w 垂直 所以 w 是法向量
w •xs+b=k , xs 在上面的線 k > 0
w •xc+b=k’ , xc 在下面的線 k’ < 0
方塊分類為 y=1 , 圓圈分類為 y=-1 分類公式 z 是 y=1 類或 y=-1 類依據:
1,~ ~ 0
1,~ ~ 0
if w z by
if w z b
![Page 11: © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 Support Vector Machines l Find a linear hyperplane (decision boundary) that will separate](https://reader030.vdocuments.us/reader030/viewer/2022032701/56649ca25503460f94961667/html5/thumbnails/11.jpg)
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 11
調整 w 與 b 得到
w•(x1-x2)=2 |w|*d=2 ∴
1
2
: 1,
: 1i
i
b w x b
b w x b
2
| |d w
![Page 12: © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 Support Vector Machines l Find a linear hyperplane (decision boundary) that will separate](https://reader030.vdocuments.us/reader030/viewer/2022032701/56649ca25503460f94961667/html5/thumbnails/12.jpg)
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 12
Support Vector Machines
We want to maximize:
– Which is equivalent to minimizing:
– But subjected to the following constraints:
This is a constrained optimization problem– Numerical approaches to solve it (e.g., quadratic programming)
2
2Margin
| |w
1bxw if1
1bxw if1)(
i
i
ixf
2| |( )
2
wL w
![Page 13: © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 Support Vector Machines l Find a linear hyperplane (decision boundary) that will separate](https://reader030.vdocuments.us/reader030/viewer/2022032701/56649ca25503460f94961667/html5/thumbnails/13.jpg)
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 13
Min f(w)=
Subject to
Change to Min f(w)=
Subject to
2| |
2
w
( ) 1, 1,2,..., .i iy w x b i N
( ( ) 1) 0, 1,2,...,i iy w x b i N
2| |
2
w
![Page 14: © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 Support Vector Machines l Find a linear hyperplane (decision boundary) that will separate](https://reader030.vdocuments.us/reader030/viewer/2022032701/56649ca25503460f94961667/html5/thumbnails/14.jpg)
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 14
2
1
1| | ( 1)
2
N
P i i ii
L w y w x b
1
0N
Pi i i
i
Lw y x
w
1
0 0N
Pi i
i
Ly
b
![Page 15: © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 Support Vector Machines l Find a linear hyperplane (decision boundary) that will separate](https://reader030.vdocuments.us/reader030/viewer/2022032701/56649ca25503460f94961667/html5/thumbnails/15.jpg)
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 15
Nonlinear Support Vector Machines
What if decision boundary is not linear?
![Page 16: © Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 Support Vector Machines l Find a linear hyperplane (decision boundary) that will separate](https://reader030.vdocuments.us/reader030/viewer/2022032701/56649ca25503460f94961667/html5/thumbnails/16.jpg)
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 16
Nonlinear Support Vector Machines
Transform data into higher dimensional space