support vector machin, an excellent tool

8/18/2019 Support vector machin, an excellent tool

1/36


2/36


3/36

08/11/05CSE 802. Prepared ! Martin

Law 4

$istor! o% SVM

SVM is a c#assier deri&ed %ro' statistica##earnin- theor! ! Vapni and Cher&onenis

SVM was rst introduced in C"L*62

SVM eco'es %a'ous when+ usin- pi7e# 'apsas input+ it -i&es accurac! co'para#e tosophisticated neura# networs with e#aorated%eatures in a handwritin- reco-nition tas

Current#!+ SVM is c#ose#! re#ated toerne# 'ethods+ #ar-e 'ar-in c#assiers+reproducin- erne# $i#ert space+ 9aussian process


4/36


Law :

*wo C#ass Pro#e' LinearSepara#e Case

C#ass 1

C#ass 2

Man! decisionoundaries canseparate thesetwo c#asses

,hich one shou#d

we choose


5/36


Law 5

E7a'p#e o% ;ad ecision;oundaries

C#ass 1

C#ass 2

C#ass 1

C#ass 2


6/36


Law <

9ood ecision ;oundar! Mar-inShou#d ;e Lar-e

*he decision oundar! shou#d e as %arawa! %ro' the data o% oth c#asses aspossi#e

,e shou#d 'a7i'i=e the 'ar-in+ m

C#ass 1

C#ass 2

m


7/3608/11/05CSE 802. Prepared ! Martin

Law >

*he "pti'i=ation Pro#e'

Let ? x 1+ ...+ x n@ e our data set and #et y i ∈ ?1+1@ e the c#ass #ae# o% x i

*he decision oundar! shou#d c#assi%! a##

points correct#! ⇒A constrained opti'i=ation pro#e'


8/3608/11/05

CSE 802. Prepared ! MartinLaw 8


,e can trans%or' the pro#e' to its dua#

*his is a uadratic pro-ra''in- (BP)

pro#e' 9#oa# 'a7i'u' o% αi can a#wa!s e %ound

w can e reco&ered !


9/3608/11/05


Characteristics o% the So#ution

Man! o% the αi are =erow is a #inear co'ination o% a s'a## nu'er o% dataSparse representation

xi with non=ero αi are ca##ed support &ectors (SV) *he decision oundar! is deter'ined on#! ! the SV Let t ( jD1+ ...+ s) e the indices o% the s support

&ectors. ,e can write

or testin- with a new data z

Co'pute and

c#assi%! z as c#ass 1 i% the su' is positi&e+ and c#ass 2

otherwise


10/3608/11/05


α


11/3608/11/05


So'e Fotes

*here are theoretica# upper ounds on theerror on unseen data %or SVM *he #ar-er the 'ar-in+ the s'a##er the ound

*he s'a##er the nu'er o% SV+ the s'a##er theound

Fote that in oth trainin- and testin-+ thedata are re%erenced on#! as inner product+

x *y *his is i'portant %or -enera#i=in- to the non#inear case


12/3608/11/05


$ow Aout Fot Linear#!Separa#e

,e a##ow GerrorH ξi in c#assication

C#ass 1

C#ass 2


13/36


Law 14

So%t Mar-in $!perp#ane

ene ξiD0 i% there is no error %or 7i ξi are ust Gs#ac &aria#esH in opti'i=ation

theor!

,e want to 'ini'i=eC tradeo para'eter etween error and'ar-in

*he opti'i=ation pro#e' eco'es


14/36


Law 1:


*he dua# o% the pro#e' is

w is a#so reco&ered as

*he on#! dierence with the #inear separa#ecase is that there is an upper ound C on αi"nce a-ain+ a BP so#&er can e used to ndαi


15/36


Law 15

E7tension to Fon#inear ecision;oundar!

e! idea trans%or' xi to a hi-herdi'ensiona# space to G'ae #i%e easierH Input space the space xi are in

eature space the space o% φ(xi) a%tertrans%or'ation

,h! trans%or' Linear operation in the %eature space is

eui&a#ent to non#inear operation in inputspace

*he c#assication tas can e GeasierH with aproper trans%or'ation. E7a'p#e J"3


16/36


Law 1<

E7tension to Fon#inear ecision;oundar!

Possi#e pro#e' o% the trans%or'ation$i-h co'putation urden and hard to -et a-ood esti'ate

SVM so#&es these two issuessi'u#taneous#!erne# trics %or eKcient co'putationMini'i=e w2 can #ead to a G-oodH c#assier

φ( )

φ( )

φ( )φ( )φ( )

φ( )

φ( )φ( )

φ(.) φ( )

φ( )

φ( )φ( )

φ( )

φ( )

φ( )

φ( )φ( )

φ( )

Feature spaceInput space


17/36


Law 1>

E7a'p#e *rans%or'ation

ene the erne# %unction K (x+y) as

Consider the %o##owin- trans%or'ation

*he inner product can e co'puted ! K without -oin- throu-h the 'ap φ(.)


18/36


Law 18

erne# *ric

*he re#ationship etween the erne# %unctionK and the 'appin- φ(.) is

*his is nown as the erne# tric

In practice+ we speci%! K + there! speci%!in-φ(.) indirect#!+ instead o% choosin- φ(.)

Intuiti&e#!+ K (x+y) represents our desired

notion o% si'i#arit! etween data x and y andthis is %ro' our prior now#ed-e

K (x+y) needs to satis%! a technica# condition(Mercer condition) in order %or φ(.) to e7ist


19/36


Law 16

E7a'p#es o% erne# unctions

Po#!no'ia# erne# with de-ree d

3adia# asis %unction erne# with width σ

C#ose#! re#ated to radia# asis %unction neura#networs

Si-'oid with para'eter κ and θ

It does not satis%! the Mercer condition on a## κ and θ

3esearch on dierent erne# %unctions indierent app#ications is &er! acti&e


20/36


Law 20

E7a'p#e o% SVM App#ications$andwritin- 3eco-nition


21/36


Law 21

Modication ue to erne#unction

Chan-e a## inner products to erne#%unctions

or trainin-+

"ri-ina#

,itherne#%unction


22/36


Law 22

Modication ue to erne#unction

or testin-+ the new data z is c#assied asc#ass 1 i% f ≥0+ and as c#ass 2 i% f 0

"ri-ina#

,itherne#%unction


23/36


Law 24

E7a'p#e

Suppose we ha&e 5 1 data points71D1+ 72D2+ 74D:+ 7:D5+ 75D


24/36


Law 2:

E7a'p#e

;! usin- a BP so#&er+ we -etα1D0+ α2D2.5+ α4D0+ α:D>.444+ α5D:.844

Fote that the constraints are indeed satised

*he support &ectors are ?72D2+ 7:D5+ 75D


25/36


Law 25

E7a'p#e

Va#ue o% discri'inant %unction

1 2 : 5 <

c#ass 2 c#ass 1c#ass 1


26/36


Law 2<

Mu#tic#ass C#assication

SVM is asica##! a twoc#ass c#assier"ne can chan-e the BP %or'u#ation to a##ow'u#tic#ass c#assication

More co''on#!+ the data set is di&ided intotwo parts Ginte##i-ent#!H in dierent wa!s anda separate SVM is trained %or each wa! o%di&ision

Mu#tic#ass c#assication is done ! co'inin-the output o% a## the SVM c#assiersMaorit! ru#eError correctin- codeirected ac!c#ic -raph


27/36


Law 2>

So%tware

A #ist o% SVM i'p#e'entation can e %oundat http//www.erne#'achines.or-/so%tware.ht'#

So'e i'p#e'entation (such as LI;SVM)can hand#e 'u#tic#ass c#assicationSVMLi-ht is a'on- one o% the ear#iesti'p#e'entation o% SVM

Se&era# Mat#a too#o7es %or SVM are a#soa&ai#a#e


28/36


Law 28

Su''ar! Steps %or C#assication

Prepare the pattern 'atri7Se#ect the erne# %unction to useSe#ect the para'eter o% the erne# %unction

and the &a#ue o% C ou can use the &a#ues su--ested ! the SVMso%tware+ or !ou can set apart a &a#idation set todeter'ine the &a#ues o% the para'eter

E7ecute the trainin- a#-orith' and otainthe αiQnseen data can e c#assied usin- the αiand the support &ectors


29/36

08/11/05 CSE 802. Prepared ! MartinLaw 26

e'onstration

Iris data setC#ass 1 and c#ass 4 are G'er-edH in this de'o


30/36


Stren-ths and ,eanesses o%SVM

Stren-ths *rainin- is re#ati&e#! eas!

Fo #oca# opti'a#+ un#ie in neura# networs

It sca#es re#ati&e#! we## to hi-h di'ensiona# data *radeo etween c#assier co'p#e7it! and errorcan e contro##ed e7p#icit#!

Fontraditiona# data #ie strin-s and trees can eused as input to SVM+ instead o% %eature &ectors

,eanessesFeed a G-oodH erne# %unction

E i# S t V t 3 i


31/36


Epsi#on Support Vector 3e-ression

(εSV3)Linear re-ression in %eature spaceQn#ie in #east suare re-ression+ the error%unction is εinsensiti&e #oss %unction

Intuiti&e#!+ 'istae #ess than ε is i-nored *his #eads to sparsit! si'i#ar to SVM

ε−ε

Va#ue o tar-et

Pena#t!

Va#ue o tar-et

Pena#t!

Square loss functionε-insensitive loss function

E i# S t V t 3 i


32/36



(εSV3)9i&en a data set ?71+ ...+ 7n@ with tar-et&a#ues ?u1+ ...+ un@+ we want to do εSV3

*he opti'i=ation pro#e' is

Si'i#ar to SVM+ this can e so#&ed as auadratic pro-ra''in- pro#e'

E i# S t V t 3 i


33/36



(εSV3)C is a para'eter to contro# the a'ount o%inRuence o% the error

*he w2 ter' ser&es as contro##in- the

co'p#e7it! o% the re-ression %unction *his is si'i#ar to rid-e re-ression

A%ter trainin- (so#&in- the BP)+ we -et&a#ues o% αi and αiT+ which are oth =ero i%

xi does not contriute to the error %unctionor a new data z+


34/36

08/11/05 CSE 802. Prepared ! MartinLaw 4:

"ther *!pes o% erne# Methods

A #esson #earnt in SVM a #inear a#-orith'in the %eature space is eui&a#ent to anon#inear a#-orith' in the input space

C#assic #inear a#-orith's can e-enera#i=ed to its non#inear &ersion !-oin- to the %eature spaceerne# principa# co'ponent ana#!sis+ erne#

independent co'ponent ana#!sis+ erne#canonica# corre#ation ana#!sis+ erne# 'eans+1c#ass SVM are so'e e7a'p#es


35/36


Conc#usion

SVM is a use%u# a#ternati&e to neura#networs

*wo e! concepts o% SVM 'a7i'i=e the

'ar-in and the erne# tricMan! acti&e research is tain- p#ace onareas re#ated to SVM

Man! SVM i'p#e'entations are a&ai#a#e

on the we %or !ou to tr! on !our data setU


36/36

CSE 802 Prepared ! Martin

3esources

http//www.erne#'achines.or-/http//www.support&ector.net/http//www.support&ector.net/ic'#tutoria#

.pd% http//www.erne#'achines.or-/papers/tutoria#nips.ps.

-=http//www.c#opinet.co'/isae##e/Proects/SVM/app#ist.ht'#
http://www.kernel-machines.org/http://www.support-vector.net/http://www.support-vector.net/icml-tutorial.pdfhttp://www.support-vector.net/icml-tutorial.pdfhttp://www.support-vector.net/icml-tutorial.pdfhttp://www.kernel-machines.org/papers/tutorial-nips.ps.gzhttp://www.kernel-machines.org/papers/tutorial-nips.ps.gzhttp://www.kernel-machines.org/papers/tutorial-nips.ps.gzhttp://www.clopinet.com/isabelle/Projects/SVM/applist.htmlhttp://www.clopinet.com/isabelle/Projects/SVM/applist.htmlhttp://www.clopinet.com/isabelle/Projects/SVM/applist.htmlhttp://www.clopinet.com/isabelle/Projects/SVM/applist.htmlhttp://www.kernel-machines.org/papers/tutorial-nips.ps.gzhttp://www.kernel-machines.org/papers/tutorial-nips.ps.gzhttp://www.kernel-machines.org/papers/tutorial-nips.ps.gzhttp://www.support-vector.net/icml-tutorial.pdfhttp://www.support-vector.net/icml-tutorial.pdfhttp://www.support-vector.net/icml-tutorial.pdfhttp://www.support-vector.net/http://www.kernel-machines.org/

support vector machin, an excellent tool

Documents