Download - Support Vector Machines (part 1)
Face Recognition & Biometric Systsems
Support Vector Machines (part 1)
Face Recognition & Biometric Systsems
Plan of the lecture
Problem of classificationSVM for solving linear problems training classification
Application of convolution kernels
Face Recognition & Biometric Systsems
Bibliography
Corrina Cortez, Vladimir VapnikSupport-Vector Networks
Face Recognition & Biometric Systsems
Classification problem
Aim: classification of an element to one of defined classesTwo stages: training classification of samples
Available solutions: Artificial Neural Networks Support Vector Machines other classifiers
Face Recognition & Biometric Systsems
Classification problem
Training set - requirements: classified representative
Training process: aims at finding general rules a risk of overfitting to the training
set (especially when it is not representative)
Face Recognition & Biometric Systsems
Classification problem
Classification of samples: must be preceded by the training stage applies rules derived from the training
Number of classes: SVM solves two-class problems it is possible to solve multi-class
problems basing on two-class problems
Face Recognition & Biometric Systsems
Classification problem
Linearly separable
Face Recognition & Biometric Systsems
Classification problem
Non-linearly separable
Face Recognition & Biometric Systsems
Classification problem
Training with error (soft margin)
Face Recognition & Biometric Systsems
Classification problem
Margin maximisation
Face Recognition & Biometric Systsems
Linear separability
Data set: (y1,x1),...,(yl,xl), yi{-1,1}
Vector w, scalar value b:w • xi + b 1 for yi = 1
w • xi + b -1 for yi = -1
henceyi (w • xi + b) 1
The condition must be fulfilled for the whole data set
Face Recognition & Biometric Systsems
SVM – training
SVM solves linear separable two-class problems other cases transformed to the
basic problem
Optimal hyperplane margin between samples of two
classes margin maximisation
Face Recognition & Biometric Systsems
SVM – training
Optimal hyperplane:w0 • x + b0 = 0
2D example – hyperplane is a line
Margin width (without b):
||
max||
min}1:{}1:{ w
wx
w
wxw
yxyx,bρ
Face Recognition & Biometric Systsems
SVM – training
Optimal width:
Maximisation of , minimisation of w0 • w0
Limitation: yi (w • xi + b) 1
00000
2
||
2
wwww
),bρ(
Face Recognition & Biometric Systsems
SVM – training
Margin:Optimal hyperplane:
yi – class identifier i – Lagrange multipliers
A problem: how to find i?
1)( by ii xw
l
iiiiy
1
00 xw
Face Recognition & Biometric Systsems
SVM – training
Function maximisation:
1 – unitary vector (l – dimensional)D – l x l matrix:
DΛΛ1ΛΛ TTW2
1)(
),...,( 1 lT Λ
jijiij yyD xx
Face Recognition & Biometric Systsems
SVM – training
Optimisation limits:
Optimisation based on the gradient method
0Λ0YΛT
),...,( 1 lT yyY
Face Recognition & Biometric Systsems
SVM – training
Lagrange multipliers : non-zero values for support vectors equal zero for other vectors
(majority)
Training set after the training: support vectors (a small subset of
the training set) coefficients for every vector
Face Recognition & Biometric Systsems
SVM – classification
Calculate y for a vector which is to be classified:
xr, xs – support vectors from both classes
Classification decision
byfl
iiii
1)( xxx
l
isiriii yb
1)(
2
1xxxx
Face Recognition & Biometric Systsems
SVM – limitations
SVM conditions: solves two-class problem linear separability of data
A XOR problem:
Face Recognition & Biometric Systsems
SVM – limitations
Possibilities of enhancement: SVM for non-linear data – too
complicated calculations transformation of the data, so that
they are linearly separable
Mapping into higher dimension example of XOR in 2D mapped into
3D
Face Recognition & Biometric Systsems
Convolution kernelsFunction:Mapping into higher dimension: x (x)Calculations use scalar product of vectors, not the vectors themselvesKernels of convolution may be used instead of scalar products
No need to find function
Nn RR :
)()(),( vuvu K
Face Recognition & Biometric Systsems
Convolution kernels
),( jijiij KyyD xx
Training with convolution kernels
Face Recognition & Biometric Systsems
Convolution kernels
bKyfl
iiii
1),()( xxx
l
isiriii KKyb
1)],(),([
2
1xxxx
xr, xs – support vectors from both classes
Classification with convolution kernels
Face Recognition & Biometric Systsems
Convolution kernels
Linear
Polynomial
RBF (radial basis functions)
2
2||
),( vu
vu
eK
vuvu ),(K
dK )1(),( vuvu
Face Recognition & Biometric Systsems
Summary
ClassifiersBasic problem: two-class linear separable data set solved by the SVM
Enhancement convolution kernels – SVM for non-
linear separable data
Face Recognition & Biometric Systsems
Thank you for your attention!
Next week
Support Vector Machines – continued... multi-class cases soft margin training applications to face recognition