svm and kernel machines: linear and non-linear classification · tuesday, september 9, 2014, 9:00...

Tuesday, September 9, 2014, 9:00 am - 10:30 am

SVM and kernel machines: linearand non-linear classification

Prof. Stéphane Canu

Kernel methods are a class of learning machine that has become an increasingly populartool for learning tasks such as pattern recognition, classification or novelty detection. Thispopularity is mainly due to the success of the support vector machines (SVM), probablythe most popular kernel method, and to the fact that kernel machines can be used in manyapplications as they provide a bridge from linearity to non-linearity. This allows thegeneralization of many well known methods such as PCA or LDA to name a few. Other keypoints related with kernel machines are convex optimization, duality and related sparcity.The Objective of this course is to provide an overview of all these issues related withkernels machines. To do so, we will introduce kernel machines and associatedmathematical foundations through practical implementation. All lectures will be devoted tothe writing of some Matlab functions that, putting all together, will provide a toolbox forlearning with kernels.

About Stéphane Canu

Stéphane Canu is a Professor of the LITIS research laboratory and of theinformation technology department, at the National institute of appliedscience in Rouen (INSA). He has been the former executive director of theLITIS, an information technology research laboratory in Normandy (150researcher) form 2005 to 2012. He received a Ph.D. degree in SystemCommand from Comiègne University of Technology in 1986. He joined the

faculty department of Computer Science at Compiegne University of Technology in 1987.He received the French habilitation degree from Paris 6 University. In 1997, he joined theRouen Applied Sciences National Institute (INSA) as a full professor, where he created theinformation engineering department. He has been the dean of this department until 2002when he was named director of the computing service and facilities unit. In 2004 he join forone sabbatical year the machine learning group at ANU/NICTA (Canberra) with Alex Smola

Ocean's Big Data Mining, 2014(Data mining in large sets of complex oceanic data: new challenges and solutions)

8-9 Sep 2014 Brest (France)

SUMMER SCHOOL #OBIDAM14 / 8-9 Sep 2014 Brest (France) oceandatamining.sciencesconf.org

and Bob Williamson. In the last five years, he has published approximately thirty papers inrefereed conference proceedings or journals in the areas of theory, algorithms andapplications using kernel machines learning algorithm and other flexible regressionmethods. His research interests includes kernels and frames machines, regularization,machine learning applied to signal processing, pattern classification, matrix factorizationfor recommender systems and learning for context aware applications.

SVM and Kernel machinelinear and non-linear classification

Stéphane Canu

stephane.canu@litislab.eu

Ocean’s Big Data Mining, 2014

September 9, 2014

Road map

1 Supervised classification and prediction

2 Linear SVMSeparating hyperplanesLinear SVM: the problemOptimization in 5 slidesDual formulation of the linear SVMThe non separable case

3 Kernels

4 Kernelized support vector machine

Supervised classification as Learning from examples

The task, use longitude and latitude to predict: is it a boat or a house?

credit: A Gentle Introduction to Support Vector Machines in Biomedicine A. Statnikov, D. Hardin, I. Guyon and C. F. Aliferiswww.nyuinformatics.org/downloads/supplements/SVM_Tutorial_2010/Final_WB.pdf

Using (red and green) labelled examples learn a (yellow) decision rule

Using (red and green) labelled examples...

Using (red and green) labelled examples... learn a (yellow) decision rule

Use the decision border to predict unseen objects label

Suppervised classification: the 2 steps

}i = 1, n

A the learning algorithmf the decision frontier

= f (x)

1 the border Learn(xi , yi , n training data) % A is SVM_learn2

Predict(unseen x , the border) % f is SVM_val

Unavaliable speakers (more qualified in Environmental Data Learning ;)

Mikhail Kanevski S. Thiria & F. Badran .UNIL geostat UPMC Locean .

less "ocean", but...

more maths, more optimization, more matlab...

Unavaliable speakers (more qualified in Environmental Data Learning ;)

Mikhail Kanevski S. Thiria & F. Badran .UNIL geostat UPMC Locean .

less "ocean", but...

more maths, more optimization, more matlab...

Road map

3 Kernels

margin

"The algorithms for constructing the separating hyperplane considered above willbe utilized for developing a battery of programs for pattern recognition." inLearning with kernels, 2002 - from V .Vapnik, 1982

Separating hyperplanes

Find a line to separate (classify) blue from red

D(x) = sign�v

>x + a

the decision border:

>x + a = 0

there are many solutions...The problem is ill posed

How to choose a solution?

D(x) = sign�v

>x + a

>x + a = 0

D(x) = sign�v

>x + a

>x + a = 0

Maximize our confidence = maximize the margin

the decision border: �(v, a) = {x 2 IRd

��v

>x + a = 0}

margin

maximize the margin

maxv,a

mini2[1,n]

dist(xi

,�(v, a))

| {z }margin: m

Maximize the confidence

maxv,a

with mini=1,n

+ a|kvk � m

the problem is still ill posed

if (v, a) is a solution, 8 0 < k (kv, ka) is also a solution. . .

From the geometrical to the numerical margin

�1/|w|

{x | wTx = 0}

marge<− −>

Valeur de la marge dans le cas monodimensionnel

Maximize the (geometrical) margin8><

maxv,a

with mini=1,n

+ a|kvk � m

if the min is greater, everybody is greater(y

2 {�1, 1})8><

maxv,a

withyi

+ a)kvk � m, i = 1, n

change variable: w = v

mkvk and b = a

mkvk =) kwk = 1

maxw,b

with y

+ b) � 1 ; i = 1, nand m = 1

minw,b

kwk2with y

+ b) � 1i = 1, n

From the geometrical to the numerical margin

�1/|w|

{x | wTx = 0}

marge<− −>

Valeur de la marge dans le cas monodimensionnel

Maximize the (geometrical) margin8><

maxv,a

with mini=1,n

+ a|kvk � m

if the min is greater, everybody is greater(y

2 {�1, 1})8><

maxv,a

withyi

+ a)kvk � m, i = 1, n

change variable: w = v

mkvk and b = a

mkvk =) kwk = 1

maxw,b

with y

+ b) � 1 ; i = 1, nand m = 1

minw,b

kwk2with y

+ b) � 1i = 1, n

Road map

3 Kernels

margin

"The algorithms for constructing the separating hyperplane considered above willbe utilized for developing a battery of programs for pattern recognition." inLearning with kernels, 2002 - from V .Vapnik, 1982

Linear SVM: the problem

The maximal margin (=minimal norm)canonical hyperplane

margin

Linear SVMs are the solution of the following problem (called primal)

Let {(xi

); i = 1 : n} be a set of labelled data with x 2 IRd , yi

2 {1,�1}A support vector machine (SVM) is a linear classifier associated with thefollowing decision function: D(x) = sign

>x + b

�where w 2 IRd and

b 2 IR a given thought the solution of the following problem:(

minw2IRd , b2IR

with y

+ b) � 1 , i = 1, n

This is a quadratic program (QP):

>Az� d

with Bz e

Support vector machines as a QPThe Standart QP formulation(

minw,b

kwk2with y

+ b) � 1, i = 1, n,

z2IRd+1

>Az� d

with Bz e

z = (w, b)>, d = (0, . . . , 0)>, A =

I 00 0

�, B = �[diag(y)X , y] and

e = �(1, . . . , 1)>

Solve it using a standard QP solver such as (for instance)% QUADPROG Quadratic programming.% X = QUADPROG(H,f,A,b) attempts to solve the quadratic programming problem:%% min 0.5*x’*H*x + f’*x subject to: A*x <= b% x% so that the solution is in the range LB <= X <= UB

For more solvers (just to name a few) have a look at:plato.asu.edu/sub/nlores.html#QP-problem

www.numerical.rl.ac.uk/people/nimg/qp/qp.html

Road map

3 Kernels

First order optimality condition (1)

problem P =

minx2IRn

with hj

(x) = 0 j = 1, . . . , pand g

(x) 0 i = 1, . . . , q

Definition: Karush, Kuhn and Tucker (KKT) conditions

stationarity rJ(x?) +pX

(x?) +qX

(x?) = 0

primal admissibility h

(x?) = 0 j = 1, . . . , pg

(x?) 0 i = 1, . . . , qdual admissibility µ

� 0 i = 1, . . . , qcomplementarity µ

(x?) = 0 i = 1, . . . , q

and µi

are called the Lagrange multipliers of problem P

First order optimality condition (2)

Theorem (12.1 Nocedal & Wright pp 321)

If a vector x? is a stationary point of problem PThen there existsa Lagrange multipliers such that

�x?, {�

}j=1:p, {µi

}i=1:q

fulfill KKT conditionsaunder some conditions e.g. linear independence constraint qualification

If the problem is convex, then a stationary point is the solution of theproblem

A quadratic program (QP) is convex when. . .

>Az� d

with Bz e

. . . when matrix A is positive definite

KKT condition - Lagrangian (3)

problem P =

minx2IRn

with hj

(x) = 0 j = 1, . . . , pand g

(x) 0 i = 1, . . . , q

Definition: Lagrangian

The lagrangian of problem P is the following function:

L(x,�, µ) = J(x) +pX

(x) +qX

The importance of being a lagrangian

the stationarity condition can be written: rL(x?,�, µ) = 0

the lagrangian saddle point max�,µ

L(x,�, µ)

Primal variables: x and dual variables �, µ (the Lagrange multipliers)

Duality – definitions (1)

Primal and (Lagrange) dual problems

minx2IRn

with hj

(x) = 0 j = 1, pand g

(x) 0 i = 1, qD =

�2IRp,µ2IRqQ(�, µ)

with µj

� 0 j = 1, q

Dual objective function:

Q(�, µ) = infx

L(x,�, µ)

= infx

J(x) +pX

(x) +qX

Wolf dual problem

8>>>><

maxx,�2IRp,µ2IRq

L(x,�, µ)with µ

� 0 j = 1, q

and rJ(x?) +pX

(x?) +qX

(x?) = 0

Duality – theorems (2)

Theorem (12.12, 12.13 and 12.14 Nocedal & Wright pp 346)

If f , g and h are convex and continuously differentiablea, then the solutionof the dual problem is the same as the solution of the primal

aunder some conditions e.g. linear independence constraint qualification

(�?, µ?) = solution of problem Dx

? = arg minx

L(x,�?, µ?)

Q(�?, µ?) = arg minx

L(x,�?, µ?) = L(x?,�?, µ?)

= J(x?) + �?H(x?) + µ?G (x?) = J(x?)

and for any feasible point x

Q(�, µ) J(x) ! 0 J(x)� Q(�, µ)

The duality gap is the difference between the primal and dual cost functions

Road map

3 Kernels

Figure from L. Bottou & C.J. Lin, Support vector machine solvers, in Large scale kernel machines, 2007.

Linear SVM dual formulation - The lagrangian

(minw,b

kwk2with y

+ b) � 1 i = 1, n

Looking for the lagrangian saddle point max↵

minw,b

L(w, b,↵) with so called

lagrange multipliers ↵i

L(w, b,↵) =12kwk2 �

+ b)� 1�

represents the influence of constraint thus the influence of the trainingexample (x

Stationarity conditions

L(w, b,↵) =12kwk2 �

+ b)� 1�

Computing the gradients:

L(w, b,↵) = w �nX

@L(w, b,↵)@b

i=1 ↵i

we have the following optimality conditions8>>>><

L(w, b,↵) = 0 ) w =nX

@L(w, b,↵)

= 0 )nX

KKT conditions for SVM

stationarity w �nX

= 0 andnX

primal admissibility yi

+ b) � 1 i = 1, . . . , n

dual admissibility ↵i

� 0 i = 1, . . . , n

complementarity ↵i

+ b)� 1⌘= 0 i = 1, . . . , n

The complementary condition split the data into two sets

A be the set of active constraints: usefull points

A = {i 2 [1, n]�� y

(w⇤>x

+ b⇤) = 1}

its complementary A useless points

if i /2 A,↵i

The KKT conditions for SVM

The same KKT but using matrix notations and the active set Astationarity w � X>D

↵ = 0

↵>y = 0

primal admissibility Dy

(Xw + b I1) � I1

dual admissibility ↵ � 0

complementarity Dy

(XAw + b I1A) = I1A↵A = 0

Knowing A, the solution verifies the following linear system:8<

w �X>AD

↵A = 0�D

XAw �byA = �eA�y

>A↵A = 0

with Dy

= diag(yA), ↵A = ↵(A) , yA = y(A) et XA = X (XA; :).

The KKT conditions as a linear system8<

w �X>AD

↵A = 0�D

XAw �byA = �eA�y

>A↵A = 0

with Dy

= diag(yA), ↵A = ↵(A) , yA = y(A) et XA = X (XA; :).

I �X>AD

XA 0 �yA

0 �y

we can work on it to separate w from (↵A, b)

The SVM dual formulationThe SVM Wolfe dual

8>>>>>><

>>>>>>:

maxw,b,↵

12kwk2 �

+ b)� 1�

with ↵i

� 0 i = 1, . . . , n

and w �nX

= 0 andnX

using the fact: w =nX

The SVM Wolfe dual without w and b8>>>>>><

>>>>>>:

max↵

� 12

with ↵i

� 0 i = 1, . . . , n

Linear SVM dual formulationL(w, b,↵) =

12kwk2 �

+ b)� 1�

Optimality: w =nX

L(↵) = 12

| {z }w

i=1 ↵i

| {z }w

� bnX

| {z }=0

i=1 ↵i

= �12

Dual linear SVM is also a quadratic program

problem D

min↵2IRn

>G↵� e

with y

>↵ = 0and 0 ↵

i = 1, n

with G a symmetric matrix n ⇥ n such that Gij

SVM primal vs. dual

Primal

minw2IRd ,b2IR

with y

+ b) � 1i = 1, n

d + 1 unknownn constraintsclassical QPperfect when d << n

min↵2IRn

↵>G↵� e

with y

>↵ = 0and 0 ↵

i = 1, n

n unknownG Gram matrix (pairwiseinfluence matrix)n box constraintseasy to solveto be used when d > n

f (x) =dX

+ b =nX

SVM primal vs. dual

Primal

minw2IRd ,b2IR

with y

+ b) � 1i = 1, n

d + 1 unknownn constraintsclassical QPperfect when d << n

min↵2IRn

↵>G↵� e

with y

>↵ = 0and 0 ↵

i = 1, n

n unknownG Gram matrix (pairwiseinfluence matrix)n box constraintseasy to solveto be used when d > n

f (x) =dX

+ b =nX

Road map

3 Kernels

4 Kernelized support vector machine 0

Slack j

The non separable case: a bi criteria optimization problem

Modeling potential errors: introducing slack variables ⇠i

⇢no error: y

+ b) � 1) ⇠i

= 0error: ⇠

= 1� yi

+ b) > 00

Slack j

8>>>>>>><

>>>>>>>:

minw,b,⇠

12kwk2

minw,b,⇠

with y

+ b) � 1� ⇠i

� 0 i = 1, n

Our hope: almost all ⇠i

The non separable caseModeling potential errors: introducing slack variables ⇠

⇢no error: y

+ b) � 1) ⇠i

= 0error: ⇠

= 1� yi

+ b) > 0

Minimizing also the slack (the error), for a given C > 08>>><

minw,b,⇠

12kwk2 + C

with y

+ b) � 1� ⇠i

i = 1, n⇠i

� 0 i = 1, n

Looking for the saddle point of the lagrangian with the Lagrangemultipliers ↵

� 0 and �i

L(w, b,↵,�) =12kwk2 + C

+ b)� 1 + ⇠i

��nX

The KKT

L(w, b,↵,�) =12kwk2 + C

+ b)� 1 + ⇠i

��nX

= 0 andnX

C � ↵i

� �i

= 0 i = 1, . . . , n

+ b) � 1 i = 1, . . . , n

� 0 i = 1, . . . , n

+ b)� 1 + ⇠i

⌘= 0 i = 1, . . . , n

= 0 i = 1, . . . , n

Let’s eliminate �!

= 0 andnX

+ b) � 1 i = 1, . . . , n⇠i

� 0 i = 1, . . . , n;

� 0 i = 1, . . . , nC � ↵

� 0 i = 1, . . . , n;

+ b)� 1 + ⇠i

⌘= 0 i = 1, . . . , n

(C � ↵i

) ⇠i

= 0 i = 1, . . . , n

sets I0 IA IC

0 0 < ↵ < C C�

C C � ↵ 0⇠i

0 0 1� yi

+ b)yi

+ b) > 1 yi

+ b) = 1 yi

+ b) < 1useless usefull (support vec) suspicious

The importance of being support

−2 −1 0 1 2 3 4−2

datapoint ↵

constraintvalue set

useless ↵i

�> 1 I

support 0 < ↵i

�= 1 I↵

suspicious ↵i

�< 1 I

Table : When a data point is « support » it lies exactly on the margin.

here lies the efficiency of the algorithm (and its complexity)!

sparsity: ↵i

Optimality conditions (p = 1)

L(w, b,↵,�) =12kwk2 + C

+ b)� 1 + ⇠i

��nX

Computing the gradients:

8>>>>>><

>>>>>>:

L(w, b,↵) = w �nX

@L(w, b,↵)@b

r⇠iL(w, b,↵) = C � ↵i

� �i

no change for w and b

� 0 and C � ↵i

� �i

= 0 ) ↵i

The dual formulation:8><

min↵2IRn

>G↵� e

with y

>↵ = 0and 0 ↵

C i = 1, n

SVM primal vs. dual

Primal

minw,b,⇠2IRn

kwk2 + C

with y

+ b) � 1� ⇠i

� 0 i = 1, n

d + n + 1 unknown2n constraintsclassical QPto be used when n is toolarge to build G

min↵2IRn

↵>G↵� e

with y

>↵ = 0and 0 ↵

C i = 1, n

n unknownG Gram matrix (pairwiseinfluence matrix)2n box constraintseasy to solveto be used when n is not toolarge

Eliminating the slack but not the possible mistakes8>>><

minw,b,⇠2IRn

12kwk2 + C

with yi

+ b) � 1� ⇠i

� 0 i = 1, n

Introducing the hinge loss

= max�1� y

+ b), 0�

minw,b

12 kwk2 + C

max�0, 1� y

+ b)�

Back to d + 1 variables, but this is no longer an explicit QP

The hinge and other loss

Square hinge: (huber/hinge) and Lasso SVM

minw,b

kwk1 + CnX

max�1� y

+ b), 0�p

Penalized Logistic regression (Maxent)

minw,b

kwk22 � CnX

log�1 + exp�2yi (w

>xi+b)

The exponential loss (commonly used in boosting)

minw,b

kwk22 + CnX

exp�yi (w>

The sigmoid loss

minw,b

kwk22 � CnX

tanh�yi

+ b)�

−1 0 10

0/1 losshingehinge2

logisticexponentialsigmoid

Roadmap

3 Kernels

Introducing non linearities through the feature mapSVM Val

f (x) =dX

+ b =nX

x) + b

◆2 IR2

�(t) =

linear in x 2 IR5

quadratic in t 2 IR2

The feature map

� : IR2 �! IR5

t 7�! �(t) = x

x = �(ti

)>�(t)

Stéphane Canu (INSA Rouen - LITIS) September 9, 2014 39 / 62

Introducing non linearities through the feature mapSVM Val

f (x) =dX

+ b =nX

x) + b

◆2 IR2 �(t) =

linear in x 2 IR5

quadratic in t 2 IR2

The feature map

� : IR2 �! IR5

t 7�! �(t) = x

x = �(ti

)>�(t)Stéphane Canu (INSA Rouen - LITIS) September 9, 2014 39 / 62

Introducing non linearities through the feature map

A. Lorena & A. de Carvalho, Uma Introducão às Support Vector Machines, 2007

Non linear case: dictionary vs. kernel

in the non linear case: use a dictionary of functions

(x), j = 1, p with possibly p =1

for instance polynomials, wavelets...

f (x) =pX

(x) with wj

so that

f (x) =nX

| {z }k(xi ,x)

p � n so what since k(xi

, x) =P

j=1 �j

Non linear case: dictionary vs. kernel

in the non linear case: use a dictionary of functions

(x), j = 1, p with possibly p =1

for instance polynomials, wavelets...

f (x) =pX

(x) with wj

so that

f (x) =nX

| {z }k(xi ,x)

p � n so what since k(xi

, x) =P

j=1 �j

closed form kernel: the quadratic kernelThe quadratic dictionary in IRd :

� : IRd ! IRp=1+d+ d (d+1)2

s 7! � =�1, s1, s2, ..., sd , s2

1 , s22 , ..., s

, ..., si

, ...�

in this case�(s)>�(t) = 1 + s1t1 + s2t2 + ...+ s

+ s21 t2

1 + ...+ s2d

+ ...+ si

The quadratic kenrel: s, t 2 IRd , k(s, t) =�s

>t + 1�2

= 1 + 2s>t +�s

>t�2 computes

the dot product of the reweighted dictionary:

� : IRd ! IRp=1+d+ d (d+1)2

s 7! � =�1,p

2s2, ...,p

, s21 , s

22 , ..., s

, ...,p

, ...�

p = 1 + d + d(d+1)2

multiplications vs. d + 1use kernel to save computration

� : IRd ! IRp=1+d+ d (d+1)2

s 7! � =�1, s1, s2, ..., sd , s2

1 , s22 , ..., s

, ..., si

, ...�

+ s21 t2

1 + ...+ s2d

+ ...+ si

>t + 1�2

= 1 + 2s>t +�s

>t�2 computes

� : IRd ! IRp=1+d+ d (d+1)2

s 7! � =�1,p

2s2, ...,p

, s21 , s

22 , ..., s

, ...,p

, ...�

p = 1 + d + d(d+1)2

� : IRd ! IRp=1+d+ d (d+1)2

s 7! � =�1, s1, s2, ..., sd , s2

1 , s22 , ..., s

, ..., si

, ...�

+ s21 t2

1 + ...+ s2d

+ ...+ si

>t + 1�2

= 1 + 2s>t +�s

>t�2 computes

� : IRd ! IRp=1+d+ d (d+1)2

s 7! � =�1,p

2s2, ...,p

, s21 , s

22 , ..., s

, ...,p

, ...�

p = 1 + d + d(d+1)2

kernel: features through pairwise comparisons

x �(x)

e.g. a text e.g. BOW

n examples

p features

K The matrix of pairwise comparizons (O(n2))

Kenrel machinekernel as a dictionary

f (x) =nX

k(x, xi

influence of example i depends on y

k(x, xi

) the kernel do NOT depend on y

Definition (Kernel)

Let ⌦ be a non empty set (the input space).

A kernel is a function k from ⌦⇥ ⌦ onto IR. k : ⌦⇥ ⌦ 7�! IRs, t �! k(s, t)

semi-parametric version: given the family q

(x), j = 1, p

f (x) =nX

k(x, xi

Kenrel machinekernel as a dictionary

f (x) =nX

k(x, xi

influence of example i depends on y

k(x, xi

) the kernel do NOT depend on y

Definition (Kernel)

Let ⌦ be a non empty set (the input space).

A kernel is a function k from ⌦⇥ ⌦ onto IR. k : ⌦⇥ ⌦ 7�! IRs, t �! k(s, t)

semi-parametric version: given the family q

(x), j = 1, p

f (x) =nX

k(x, xi

In the beginning was the kernel...

Definition (Kernel)

a function of two variable k from ⌦⇥ ⌦ to IR

Definition (Positive kernel)

A kernel k(s, t) on ⌦ is said to be positiveif it is symetric: k(s, t) = k(t, s)

an if for any finite positive interger n:

8{↵i

}i=1,n 2 IR, 8{x

}i=1,n 2 ⌦,

) � 0

it is strictly positive if for ↵i

6= 0nX

Examples of positive kernelsthe linear kernel: s, t 2 IRd , k(s, t) = s

symetric: s

>t = t>s

positive:nX

↵i↵jk(xi , xj ) =nX

↵i↵jx>i xj

↵ixi

↵jxj

��

↵ixi

��

the product kernel: k(s, t) = g(s)g(t) for some g : IRd ! IR,

symetric by constructionpositive:

↵i↵jk(xi , xj ) =nX

↵i↵jg(xi )g(xj )

↵ig(xi )

↵jg(xj )

↵ig(xi )

k is positive , (its square root exists) , k(s, t) = h�s

,�ti

J.P. Vert, 2006

Positive definite Kernel (PDK) algebra (closure)

(s, t) and k

(s, t) are two positive kernels

DPK are a convex cone: 8a1 2 IR+ a1k1(s, t) + k2(s, t)

product kernel k1(s, t)k2(s, t)

proofs

by linearity:nX

↵i↵j�a

(i , j) + k

(i , j)�= a

↵i↵jk1

(i , j) +nX

↵i↵jk2

(i , j)

assuming 9 ` s.t. k

(s, t) =X

`(s) `(t)

↵i↵j k

(xi , xj )k2

(xi , xj ) =nX

↵i↵j�X

`(xi ) `(xj )k2

(xi , xj )�

�↵i `(xi )

� �↵j `(xj )

(xi , xj )

N. Cristianini and J. Shawe Taylor, kernel methods for pattern analysis, 2004

Kernel engineering: building PDKfor any polynomial with positive coef. � from IR to IR

��k(s, t)

if is a function from IRd to IRd

� (s), (t)

if ' from IRd to IR+, is minimum in 0k(s, t) = '(s + t)� '(s� t)

convolution of two positive kernels is a positive kernelK

Example : the Gaussian kernel is a PDK

exp(�ks� tk2) = exp(�ksk2 � ktk2 + 2s>t)= exp(�ksk2) exp(�ktk2) exp(2s>t)

>t is a PDK and function exp as the limit of positive series expansion, soexp(2s>t) is a PDK

exp(�ksk2) exp(�ktk2) is a PDK as a product kernel

the product of two PDK is a PDKO. Catoni, master lecture, 2005

some examples of PD kernels...

type name k(s, t)

radial gaussian exp⇣� r

⌘, r = ks � tk

radial laplacian exp(�r/b)

radial rationnal 1� r

radial loc. gauss. max�0, 1� r

�d exp(� r

non stat. �2 exp(�r/b), r =P

(sk�tk )2

projective polynomial (s>t)p

projective affine (s>t + b)p

projective cosine s

>t/kskktk

projective correlation exp⇣

kskktk � b

Most of the kernels depends on a quantity b called the bandwidth

Roadmap

3 Kernels

4 Kernelized support vector machine −1

using relevant features...

a data point becomes a function x �! k(x, •)

Representer theorem for SVM(

minf ,b

12kf k2H

with yi

�f (x

) + b� � 1

Lagrangian

L(f , b,↵) =12kf k2H �

(f (xi

) + b)� 1�

↵ � 0

optimility condition: rf

L(f , b,↵) = 0, f (x) =nX

Eliminate f from L:

8>>>><

kf k2H =nX

Q(b,↵) = �12

)�nX

b � 1�

Dual formulation for SVMthe intermediate function

Q(b,↵) = �12

)� b� nX

max↵

Q(b,↵)

b can be seen as the Lagrange multiplier of the following (balanced)constaint

i=1 ↵i

= 0 which is also the optimality KKT condition on b

Dual formulation8>>>>>><

>>>>>>:

max↵2IRn

� 12

such thatnX

and 0 ↵i

, i = 1, n

SVM dual formulation

Dual formulation8>>>><

max↵2IRn

� 12

withnX

= 0 and 0 ↵i

, i = 1, n

The dual formulation gives a quadratic program (QP)(min↵2IRn

>G↵� I1>↵with ↵>

y = 0 and 0 ↵

with Gij

with the linear kernel f (x) =P

i=1 ↵i

j=1 �j

when d is small wrt. n primal may be interesting.

the general case: C -SVMPrimal formulation

f 2H,b,⇠2IRn12kf k2 + C

such that yi

�f (x

) + b� � 1� ⇠

, ⇠i

� 0, i = 1, n

C is the regularization path parameter (to be tuned)

p = 1 , L

SVM(max↵2IRn

� 12↵

>G↵+ ↵>I1such that ↵>

y = 0 and 0 ↵i

C i = 1, n

p = 2, L

SVM (max↵2IRn

� 12↵

> �G + 1

I�↵+ ↵>I1

such that ↵>y = 0 and 0 ↵

i = 1, n

the regularization path: is the set of solutions ↵(C ) when C varies

Data groups: illustrationf (x) =

↵ik(x, xi )

D(x) = sign

�f (x) + b

useless data important data suspicious datawell classified support

↵ = 0 0 < ↵ < C ↵ = C

the regularization path: is the set of solutions ↵(C ) when C varies

The importance of being support

f (x) =nX

datapoint ↵

constraintvalue set

useless ↵i

�f (x

�> 1 I

support 0 < ↵i

�f (x

�= 1 I↵

suspicious ↵i

�f (x

�< 1 I

Table : When a data point is « support » it lies exactly on the margin.

here lies the efficiency of the algorithm (and its complexity)!

sparsity: ↵i

checker board

2 classes500 examplesseparable

a separable case

n = 500 data points

n = 5000 data points

Tuning C and � (the kernel width) : grid search

Empirical complexity

103 10410−1

Training size (log)

103 104101

Training size (log)

103 1040

Training size (log)

103 10410−1

Training size (log)

103 104101

Training size (log)

103 1040

Training size (log)

103 10410−1

Training size (log)

103 104101

Training size (log)

103 1040

Training size (log)

103 10410−1

Training size (log)

103 104101

Training size (log)

103 1040

Training size (log)

103 10410−1

Training size (log)

103 104101

Training size (log)

103 1040

Training size (log)

103 10410−1

Training size (log)

103 104101

Training size (log)

103 1040

Training size (log)

LibSVM

SimpleSVM

Results for C = 1Left : γ = 1 Right γ = 0.3

Trainingtimeresultsin cpuseconds(log scale)

Number ofSupportVectors

(log scale)

Errorrate (%)

(over 2000unseenpoints)

G. Loosli et al JMLR, 2007

Conclusion

Learning as an optimization problemI use CVX to prototypeI MonQPI specific parallel and distributed solvers

Universal through Kernelization (dual trick)

ScalabilityI Sparsity provides scalabilityI Kernel implies "locality"I Big data limitations: back to primal (an linear)

svm and kernel machines: linear and non-linear classification · tuesday, september 9, 2014, 9:00...

Documents

linear classification and svm - hcii lab...

infinite svm: a dp mixture of large-margin kernel...

analysis of eeg signals and facial expressions to detect...

efficient multiple kernel learning algorithms using...

linear svm

ece595 / stat598: machine learning i lecture 20 support...

kernels and the kernel trick -...

support vector machines (svm) and kernel methods

svm and kernel machine lecture 1: linear svm...a support...

introduction to object recognition & svm · •linear svm...

single to multiple kernel learning with four popular svm...

lecture5 kernel svm

linear classification: the kernel method

support vector machines - mit opencourseware · svm and...

kernel svm

support vector classi cation and...

nearest neighbors. kernel functions, svm. decision trees

support vector machines - svm &...

anomaly detection using single class svm with gaussian...

sergey maidanov engineering manager for intel® distribution...