Pattern Classification

All materials in these slides* were taken from

Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John Wiley

& Sons, 2000 with the permission of the authors and

the publisher

* Some slides are modified

1

Maximum-Likelihood And Bayesian Parameter Estimation

(2)

2

Reading Assignment

R. O.Duda,P. E. Hart, and D.G.Stork, Pattern Classification. John Wiley & Sons, 2nd ed., 2001. Chapter 3, Sections 4,5 and 7

3

Chapter 3:Maximum-Likelihood & Bayesian Parameter Estimation (part 2)

Bayesian Parameter Estimation: Gaussian Case Bayesian Parameter Estimation: General

Estimation Problems of Dimensionality Computational Complexity

Pattern Classification, Chapter 1 5

Goal: Estimate p(x) by computing p(x|D)

This formula links the desired class conditional density p(x|D) to the posterior density p( |D) .

3

The Parameter Distribution

θθθxxθ

dDppDp )|()|()|(

Pattern Classification, Chapter 1 6

Bayesian Parameter Estimation: Gaussian Case

The univariate case:

( is the only unknown parameter)

(0 and 0 are known!)

Therefore, we need to find:

),N( ~ )P(),N( ~ ) | P(x

200

2

4

θθθxxθ

dDppDp )|()|()|(

θxxθ

dDppDp )|()|()|(

Pattern Classification, Chapter 1 7

Let D = {x1, x2, …, xn}

nk

kk PxP

dPPPPP

1

)().|(

(1) )().|()().|()|(

DDD

4

Pattern Classification, Chapter 1 8

Reproducing density

220

2202

0220

2

2200

20

.ˆ

nand

nnn

n

nn

4

Pattern Classification, Chapter 1 94

Pattern Classification, Chapter 1 10

The univariate case P(x | D) P( | D) computed P(x | D) remains to be computed!

Which is equal to

Where

Gaussian is d)|(P).|x(P)|x(P DD

4

Pattern Classification, Chapter 1 11

It provides:

(Desired class-conditional density P(x | Dj, j))

Therefore: P(x | Dj, j) together with P(j)And using Bayes formula, we obtain theBayesian classification rule:

),(N~)|x(P 2n

2n D

)(P).,|x(PMax,x|(PMax jjjj

jj

DD

4

Pattern Classification, Chapter 1 12

The Multivariate case:

4

Pattern Classification, Chapter 1 13

Bayesian Parameter Estimation: General Theory

P(x | D) computation can be applied to any situation in which the unknown density can be parameterized: the basic assumptions are:

The form of P(x | ) is assumed known, but the value of is not known exactly

Our knowledge about is assumed to be contained in a known prior density P()

The rest of our knowledge is contained in a set D of n random variables x1, x2, …, xn that follows P(x)

5

Pattern Classification, Chapter 1 14

The basic problem is:“Compute the posterior density P( | D)”then “Derive P(x | D)”

Using Bayes formula, we have:

And by independence assumption:

)|x(P)|(P knk

1k

D

,d)(P).|(P)(P).|(P)|(P

D

DD

5

Pattern Classification, Chapter 1 15

Bayesian Parameter Estimation: Incremental Learning

To indicate explicitly the number of samples in a set for a single category, we shall write Dn = {x1, ..., xn}. Then, if n > 1

Using

we get

5

Pattern Classification, Chapter 1 16

Problems of Dimensionality Problems involving 50 or 100 features (binary valued)

Classification accuracy depends upon the dimensionality and the amount of training data

Case of two classes multivariate normal with the same covariance

0)error(Plim

)()(r :where

due21)error(P

r

211t

212

2

2u

2/r

7

Pattern Classification, Chapter 1 17

If features are independent then:

Most useful features are the ones for which the difference between the means is large relative to the standard deviation

It has frequently been observed in practice that, beyond a certain point, the inclusion of additional features leads to worse rather than better performance: we have the wrong model !

2di

1i i

2i1i2

2d

22

21

r

),...,,(diag

7

Pattern Classification, Chapter 1 18

77

7

Pattern Classification, Chapter 1 19

Computational Complexity

Our design methodology is affected by the computational difficulty

“big oh” notationf(x) = O(h(x)) “big oh of h(x)”If:

(An upper bound on f(x) grows no worse than h(x) for sufficiently large x!)

f(x) = 2+3x+4x2

g(x) = x2

f(x) = O(x2)

)x(hc)x(f ;)x,c( 02

00

7

Pattern Classification, Chapter 1 20

“big oh” is not unique!

f(x) = O(x2); f(x) = O(x3); f(x) = O(x4)

“big theta” notationf(x) = (h(x))If:

f(x) = (x2) but f(x) (x3)

)x(gc)x(f)x(gc0xx ;)c,c,x(

21

03

210

7

Pattern Classification, Chapter 1 21

Complexity of the ML Estimation

Gaussian priors in d dimensions classifier with n training samples for each of c classes

For each category, we have to compute the discriminant function

Total = O(d2..n)Total for c classes = O(cd2.n) O(d2.n)

Cost increase when d and n are large!

)n(O)n.2d(O

)1(O)2d.n(O1t

)n.d(O

)(Plnˆln212ln

2d)ˆx()ˆx(

21)x(g

7

Pattern Classification, Chapter 1 22

Overfitting

Frequently, no of available samples usually inadequate, how to proceed?

Solutions: Reduce dimensionality: redesign feature extractor, or

use a subset of features Assume all c classes share same covariance matrix:

pool available data Assume statistical independence: all off-diagonal

elements of covariance matrix are zero

7

Pattern Classification, Chapter 1 23

Overfitting

Classifier performs better than if we overfit the data, why?

The “training data” (black dots) were selected from a quadratic function plus Gaussian noise. The 10th degree polynomial shown fits the data perfectly, but we desire instead the second-order function f(x), since it would lead to better predictions for new samples.

7