What is Learning All about ?What is Learning All about ?

Get knowledge of by study, experience, or being taught

Become aware by information or fromobservation

Commit to memory

Be informed of or receive instruction

A Possible Definition of LearningA Possible Definition of Learning

Things learn when they change their behavior in a way that makes them perform better in the future.

Have your shoes learned the shape of your foot ?

In learning the purpose is the learner’s, whereas

in training it is the teacher’s

Our Learning Tasks in the ClassOur Learning Tasks in the Class

Classification (Supervised learning) binary classification problem multi-class classification problem

Regression (Supervised learning)

Does your machine learn anything from you?

Who/which is the better teacher/algorithm?

The Mathematical Background The Mathematical Background Material in the ClassMaterial in the Class

Calculus (Multi-variable)

f (x1;x2;x3) = x21 + x2

2 + x23

What is the gradient of function

Linear Algebra

How to compute the distance between two parallel hyperplanes in ? Rn

eigenvalue, positive definite matrix, inner product, projection matrix etc.

Basic Concepts of Probability and Basic Concepts of Probability and StatisticsStatistics

Probability:

Statistics:

Random variables, probability distribution,expected value (mean), variance …

Confidence interval, testing hypothesis …

Classification ProblemClassification Problem2-Category Linearly Separable Case2-Category Linearly Separable Case

A-

A+

wx0w = í + 1

x0w = í à 1

x0w = í

Malignant

Benign

Support Vector MachinesSupport Vector MachinesMaximizing the Margin between Bounding Maximizing the Margin between Bounding

PlanesPlanes

x0w = í + 1

x0w = í à 1

A+

A-

jjwjj22

w

= Margin

Why Use Support Vector Machines (SVMs)?Why Use Support Vector Machines (SVMs)?Powerful tools for Data MiningPowerful tools for Data Mining

SVM classifier is an optimally defined surface

SVMs have a good geometric interpretation SVMs can be generated very efficiently Can be extended from linear to nonlinear case

Typically nonlinear in the input space Linear in a higher dimensional “feature” space Implicitly defined by a kernel function

Have a sound theoretical foundation Based on Statistical Learning Theory

Why We Maximize the Margin?(Based on Statistical Learning Theory)

The Structural Risk Minimization (SRM):

The expected risk will be less than or equal to

empirical risk (training error)+ VC (error) bound

íí w

íí

2 / VC bound

minVC bound , min21íí w

íí 2

2 , maxMargin

Two-spiral Dataset(94 White Dots & 94 Red Dots)