1

Pattern recognition

"To understand is to perceive patterns" – Sir Isaiah Berlin, Russian philosopher

“The more relevant patterns at your disposal, the better your decisions will be. This is hopeful news to proponents of artificial intelligence, since computers can surely be taught to recognize patterns” – Herbert Simon, Nobel laureate, 1978

2

Outline

Introduction of conceptSyntactical pattern recognitionStatistical pattern recognitionReading Sonka 9.2, 9.4 and Gonzalez and Woods 12 (selected subsections)

Building simple classifiersSupervised classification

Minimum distance classifier Bayesian classifiers

Unsupervised classificationK-means algorithm

3

What is Pattern Recognition?

Visual pattern recognition tries to answer a basic question:What is in the image (video)? (where, when)Examples:

What object is this based on shape, position? (geometry-based object recognition)What kind of pixel is this based on local image properties? (appearance-based pattern recognition)

4

Example: face detection

5

Finding faces

Faces “look like”templates (at least when they’re frontal).General strategy:

search image windows at a range of scalesCorrect for illuminationPresent corrected window to classifier

IssuesHow corrected?What features?What classifier?what about lateral views?

6

Challenge 1: viewpoint variation

Michelangelo 1475-1564

7

Challenge 2: illumination

slide credit: S. Ullman

8

Challenge 3: occlusion

Magritte, 1957

9

Challenges 4: scale

10

Challenge 5: deformation

Xu, Beihong 1943

11

Challenge 6: background clutter

Klimt, 1913

12

Approaches in pattern recognition

StatisticalOperates in the feature spaceClassifies objects based on their featuresThe features of an object provide a quantitative description of the object

SyntacticalBased on a qualitative description of an object

13

Approaches in pattern recognition: structural (syntactic)

Section 9.4 (only pp. 410-411-412)hierarchic perspective suitable for recognizing complex patternsThe simplest sub-pattern to be recognized is called a primitive. The complex pattern is represented in terms of interrelationships between primitives

14

Approaches in pattern recognition: statistical

Terminology: features, feature vectors, classes, classifiersEach pattern is represented in terms of d features and is viewed as a point in a d-dimensional spaceIssue: how do we choose the features? The effectiveness of the chosen representation (feature set) is determined by how well patterns from different classes are separated.Given a set of training samples for each class, the objective is to establish decision boundaries in the feature space, and thus to separate patterns belonging to different classes

15

Statistical pattern recognition: Building classifiers

Each pattern is a point in feature space

Three types of flowers described by two measurements (features)

-Petal length

-Petal width

16

Features and patterns

Can’t really classify thingsHave to classify their perceived patternTerminology:

Properties we can measure: featuresCollections of features: feature vector, which is a

description of a pattern

Features have to be quantitative measures(scalar or vector-valued)

Examples: area, average curvature on the outer boundary, average intensity, etc.Questions:

How many features?Which ones?Relative importance?

17

Classes

Pattern recognition doesn’t produce general, open-ended object descriptions“which of these possibilities is it?”Finite sets of possibilities are called classesMight be a specific “yes/no” (two-class problem)Might be multiple options (e.g., OCR)

18

General classification process

Extract featuresAssemble features into a feature vector or patternAssign to a class

Might be a single classificationMight rank possible classificationsMight produce (relative) probabilities

19

Building classifiers (cont’d)

Key ideas:Patterns from the same class should cluster together in feature spaceSupervised training: learn the properties of the cluster (distribution) for each class

Example: minimum distance classifier

Unsupervised training: find the clusters from scratch; no information about the class structure is provided

Example: k-means classifier

20

Minimum-distance classifier

Reading Gonzalez and Woods excerpt pp. 698-701Idea: Use a single prototype for each class ωi (usually the class’s mean mi )

Training:compute each class’s prototype (mean)

Classification:Assign unlabeled patterns to the nearest prototype in the feature space

21

Decision boundaries

If we partition the feature space according to the nearest prototype, we create decision boundaries.

Decision boundary between two classes; means are represented by the dark dot and square

22

Minimum distance classifier (cont’d)

Simpler calculation: The distances to the prototypes don’t need to be computedWhat we need is finding the distance to the nearest prototype (minimal)

23

Linear decision boundariesLinearity:

24

Discriminants

A function used to test the class membership is called a discriminantThe two-class result can be extended to multiple cases Construct a single discriminant gi(x) for each class ωi , and assign x to class ωi if gi(x) > gj (x) for all other classes ωj .

25

Performance of the minimum distance classifier: discussion

26

Unsupervised training: clustering

The goal is to find natural groupings of patternsIt is useful if the training set is not pre-labeledUsually some a priori information is available ( for instance number of classes)

27

The k-means algorithm

is an unsupervised version of minimum distance classification; requires number of classes k.

1. Place k points into the space represented by the feature vectors that are being clustered. These points represent initial group centroids (prototypes).

2. Assign each object to the group that has the closest centroid.

28

The k-means algorithm (cont’d)3. When all vectors have been assigned, recalculate the positions of the K centroids.4. Repeat Steps 2 and 3 until the centroids no longer move.

29

The k-means algorithm (cont’d)

Trajectory of the means m1 and m2

30

Outline

Introduction of conceptSyntactical pattern recognitionStatistical pattern recognitionReading Sonka 9.2, 9.4 and Gonzalez and Woods 12 (selected subsections)

Building simple classifiersSupervised classification

Minimum distance classifierBayesian classifiers

Unsupervised classificationK-means algorithm

31

Main concept

Perform classification using a probabilistic frameworkWhen we classify, we measure the feature vector x“Given that this pattern has features x, what is the probability that it belongs to class ωi ?”

32

Notation: probabilities

The probability of discrete event A occurring is P(A)The continuous random variable x has a probability density function (pdf) is p(x)For vector-valued random variables x, we write this as p(x)

33

Conditional probabilities

We write the conditional probability of A given B as P(A/B)

This means “the probability of A given B” or “given B, what is the probability of A?”

Or for random variables, p(x/A) and p(x/A)

34

Training of Bayesian Classifiers

Suppose that we measure features for a large training set taken from class ωi .

Each of these training patterns has a different value x for the features. This can be written as the class-conditional probability: p(x/ωi )

In other words, How often do things in class ωi exhibit features x?

35

Bayesian Classification

When we classify, we measure the feature vector x“Given that this pattern has features x, what is the probability that it belongs to class ωi ?”Mathematically, this is written as

P(ωi / x)

36

How are P(ωi/ x) and p(x/ωi ) related?

Through the Bayes Theorem…

37

General structure of a Bayesian classifier

Measure for each class p(x/ωi )Prior Knowledge:

Measure or estimate P(ωi ) in the general population.(Can sometimes aggregate the training set if it is a reasonable sampling of the population)

Classification:1. Measure feature (x) for new pattern.2. Calculate posterior probabilities P(ωi/ x)

for each class with Bayes formula3. Choose the one with the largest posterior P(ωi/ x)

38

Normally-distributed class-conditional probabilities

( )( )

2

2

21

21/ i

ix

ii exp σ

μ

σπω

−−

=

39

From probabilities to discriminants

We want to compute the discriminant for class ωi assuming that the likelihood has a normal distribution…

40

Extending to multiple features

The key term in the discriminant for a 1-D normal distribution is:

Natural extension to multiple features: -normalizing each feature’s “distance from the mean” by the respective standard deviation-then use minimum distance classification(by considering the use of prior class probabilities as well)

Normalizing each feature by its variance is called naïve Bayes because it does not take into account the relationships between features

41

The multivariate normal distribution

In multiple dimensions, the normal distribution takes on the following form:

The covariance matrix consists of the variances of the variables (features) along the main diagonal and the covariances between each pair of variables (features) in the other matrix positions.

42

Multivariate normal Bayesian classification

For multiple classes in a p-dimensional feature space, each class ωi has its own mean vector mi and covariance matrix Ci

The class-conditional probabilities are:

43

Mahalonobis distanceThe expression

can be considered as a distance between feature vector x and class i. C is the covariance matrix computed for class i.It can be proven that the Maholonobis distance satisfies all properties of a distance functionMahalonobis distance is more useful than Euclidian distance when features have different scales; the importance of each feature in classification is lost with Euclidian distance.

44

Equivalence between classifiers

Pattern recognition using multivariate normal distributions and equal priors is simply a minimum Mahalonobis distance classifier.