face detection - matematikcentrum...face detection skin color • example of skin pixel classifier...

FACE DETECTIONMikael Nilsson

2

FACE DETECTION

Outline

• Introduction

• Skin Color

• Appearance Based Learning

– Timeline, ~1994 ->

• Non Maximum Suppression

• Bicycle Counting Project

3

FACE DETECTION

Introduction

• Goal of Face Detection

– Given an arbitrary image, the goal of face detection is to determine whether or not there are any faces in the image and, if present, return the image location and extent of each face.

[Yang et al, Detecting Faces in Images: a Survey, Pami 2002]

4

FACE DETECTION

Introduction

• Goal of Face Detection version 2

– Given an image, tell whether there is any human face, if there is, where is it (or where they are).

5

FACE DETECTION

Introduction

• Goal of Face Detection version 3

– Identify and locate human faces in an image regardless of their • position

• scale

• in-plane rotation

• orientation

• pose (out-of-plane rotation)

• and illumination

6

FACE DETECTION

Introduction

• Why is it hard?– Pose Variation

– Beards, Glasses

– Facial Expression

– Occlusion

– Image Orientation

– Lighting, Noise

– Resolution

7

FACE DETECTION

Introduction

• Related problems– Facial Feature Extraction (like eyes, nose, mouth localization)

– Face Recognition (who is the person)

– Facial Expression Analysis (example: only “smile-photos”)

– Head Pose Estimation

– Gender/Age/Race Classification

– Face Tracking

8

FACE DETECTION

Introduction

• Two methods discussed

– Skin Color

– Appearance Based Learning

9

FACE DETECTION

Skin Color

• Skin Color

– General Idea• Find skin color pixels in image

• Use found pixels and do blob analysis

• Skin color methods has been proposed as preprocessing before other face detection methods (all ”orange” objects detected)

10

FACE DETECTION

Skin Color

• A skin pixel classifier using likelihood ratio approach

)|(

)|(skinrgbP

skinrgbP

rgb

P(rgb)

skin non-skin

)|( skinrgbP

)|( skinrgbP

11

FACE DETECTION

Skin Color

• Example of skin pixel classifier

– Model andusing Gaussian Mixture Model (GMM)

– Train two GMMs, one with skin pixels and another with nonskin pixels

– Find likelihood ratio using GMMs and set threshold

)|( skinrgbP )|( skinrgbP

12

FACE DETECTION

Demo

• Skin Color– Michael J. Jones and James M. Rehg, ”Statistical Color Models with Application to Skin Detection”, International Journal of Computer Vision, 1999, pp.274—280.

13

FACE DETECTION

Appearance Based Learning

• Apperance Based Learning

– General Idea• Collect a large set of resized (e.g. 19x19, 20x20, 22x22 or 32x32) face and nonface images and learn a classifier to discriminate them.

• Given a test image, detect faces by applying the classifier at each position and scale of the image. 22x22

nonface

face

14

FACE DETECTION

Appearance Based Learning

• Handling position and scale

– Search all positions and relevant scales

15

FACE DETECTION

Appearance Based Learning

• Timeline

– Kah Kay Sung and Tomaso Poggio, ”Example-based Learning for View-based Human Face Detection”, MIT A.I. Memo No. 1521, 1994, C.B.C.L. Paper No. 112

– H. Rowley, S. Baluja, and T. Kanade, “Neural network-based face detection”, CVPR, 1996, pp. 203–208.

– H. Schneiderman, T. Kanade. "Probabilistic Modeling of Local Appearance and Spatial Relationships for Object Recognition”, CVPR,

1998, pp. 45-51.

16

FACE DETECTION

Appearance Based Learning

• Timeline– D. Roth, M. Yang, and N. Ahuja, “A snow-based face detector”, in NIPS 12, 2000, pp. 855–861.

– P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features”, CVPR, 2001, vol. 1, pp. 511–518.

17

FACE DETECTION

Appearance Based Learning

• Sung & Poggio 1994– They collected 1067 face patterns and for each one of these training images they did:

• Resizing to 19x19 pixels

• Masking to avoid background pixels

• Lighting correction / Histogram Equalization

19x19

18

FACE DETECTION

Appearance Based Learning

• Sung & Poggio 1994– In addition, for each one of the 1067face patterns, new virtual samples were generated by rotation and mirror operations, totalizing 4150 face patterns.

19

FACE DETECTION

Appearance Based Learning

• Sung & Poggio 1994– Each 19x19 pattern can be seen as a point in a high dimensional space.

– The K-means algorithm is used to cluster face and non-face samples into 6 (six) clusters.

– Each cluster is described by a multi-dimensional Gaussian with a centroid and covariance matrix.

20

FACE DETECTION

Appearance Based Learning

• Sung & Poggio 1994

21

FACE DETECTION

Appearance Based Learning

• Sung & Poggio 1994– Compute the distance of a sample to all face and non-face clusters

– Final classifier use a Multi-Layer Perceptron (MLP) trained using backpropagation on the distance vector

22

FACE DETECTION

Appearance Based Learning

• Sung & Poggio 1994

– Bootstrapping• 1. Start with a small set of

nonface examples in the training set

• 2. Train a MLP classifier with the current training set

• 3. Run the learned face detector on a sequence of random images (not containing any faces).

• 4. Collect all the nonface patterns that the current system wrongly classifies as faces (i.e., false positives)

• 5. Add these nonface patterns to the training set

• 6. Got to Step 2 or stop if satisfied

Test Image

False Detections

23

FACE DETECTION

Appearance Based Learning

• Sung & Poggio 1994

– Result• Able to detect upright frontal faces in complex scenes.

• Computationally

expensive

24

FACE DETECTION

Appearance Based Learning

• Rowley et al. 1996– Similar to Sung and Poggio

– 20x20 instead of 19x19, no mask used

– Same techniques for bootstrap, preprocessing, etc.

– Neural Network applied directly into the image.

– Different heuristics

– Faster than Sung & Poggio (but still far from real-time)

25

FACE DETECTION

Appearance Based Learning

• Rowley et al. 1996

26

FACE DETECTION

Appearance Based Learning

• Rowley et al. 1997

– Rowley extended his initial work in 1997 by adressing rotations also

27

FACE DETECTION

Appearance Based Learning

• Rowley et al. 1997

– Router network, input and output

28

FACE DETECTION

Appearance Based Learning

• Rowley et al. 1997

29

FACE DETECTION

Appearance Based Learning

• Schneiderman et al. 1998– Used parts of image

– Three different classifiers are created: frontal face detector, left profile, and right profile. The right profile classifier is the same as the left profile, but take as input mirrored images.

30

FACE DETECTION

Appearance Based Learning

• Schneiderman et al. 1998

31

FACE DETECTION

Appearance Based Learning

• Schneiderman et al. 1998

– Left-, right profile and frontal face

– Also for cars

32

FACE DETECTION

Appearance Based Learning

• Yang et al.

– Created sparse boolean primitive input space (intensity x position) from 20x20 patches

– Also evaluated to extend input space with (position x intensity mean x intensity variance) with some improvements

33

FACE DETECTION

Appearance Based Learning

• Yang et al.

– Example sparse input, 2 bits per value and 5 values => (2^2)*5=20 boolean vector

34

FACE DETECTION

Appearance Based Learning

• Utilized Winnow update rule

• Winnow

– Promotion: if

– Demotion: if

• Perceptron

– Promotion: if

– Demotion: if

thresholdactivation

35

FACE DETECTION

Appearance Based Learning

• Sparse Input and Winnow update rule

– Sparse Network of Winnows (SNoW)

+

= SNoW

36

FACE DETECTION

Appearance Based Learning

• Main conclusion of…

– Sung & Poggio

– Rowley et al.

– Schneiderman et al.

– Yang et al.

• …is that they works quite well in detecting faces but are too slowfor real-time operation (at leastseveral seconds per image)

37

FACE DETECTION

Appearance Based Learning

• Viola and Jones 2001

– Breakthourgh regarding detection speed ~15 FPS on 384x288 size image

– Rectangle features (“Haar-like features”)

– Adaboost learning

– Classifier cascade

38

FACE DETECTION

Appearance Based Learning

• Rectangle features– Uses the integral image

– The integral image (ii) computes a value at each pixel (x,y) that is the sum of the pixel values above and to the left of (x,y) including the point.

(x,y)

39

FACE DETECTION

• Integral image

Appearance Based Learning

40

FACE DETECTION

Appearance Based Learning

• Integral image

ii(x, y-1)

s(x-1, y)

i(x, y)

41

FACE DETECTION

Appearance Based Learning

• Integral image

– Cumulative row sum: s(x, y) = s(x–1, y) + i(x, y)

– Integral image: ii(x, y) = ii(x, y−1) + s(x, y)

• Integral image in MATLAB – ii = cumsum(cumsum(double(i)), 2);

42

FACE DETECTION

Appearance Based Learning

• Let a,b,c,d be corner coordinates

• Then the sum of original image values within the rectangle can be computed as:

– D=ii(d)+ii(a)–ii(b)–ii(c)

• Only 3 additions (2-, 1+) and 4 table lookups are required to find sum over area for any size of rectangle!

43

FACE DETECTION

Appearance Based Learning

• Rectangle features

• Value=sum(pixels in white area)-sum(pixels in black area)

• sums done with integral image

44

FACE DETECTION

Appearance Based Learning

• Rectangle features

45

FACE DETECTION

Appearance Based Learning

• Which rectangle features to use

• Design of classifier

• Adaboost

Strong classifier

Weak classifier

WeightFeaturesvector

46

FACE DETECTION

• Given a nested set of classifier hypothesis classes

• Computational Risk Minimization

Appearance Based Learning

FACEIMAGESUB-WINDOW

Classifier 1

F

T

NON-FACE

Classifier 3T

F

NON-FACE

F

T

NON-FACE

Classifier 2T

F

NON-FACE

vs false neg determined by

% False Pos

% D

etec

tion

0 50

0

100

47

FACE DETECTION

Appearance Based Learning

• Classifier cascade

FACEIMAGESUB-WINDOW

Classifier 1

F

T

NON-FACE

Classifier 3T

F

NON-FACE

F

T

NON-FACE

Classifier 2T

F

NON-FACE

Classifier 1 Classifier 2 Classifier 3

48

FACE DETECTION

Non Maximum Supression

• Non Maximum Supression

– Note that there will typically be several detections on a face• This is true for all appearance based methods discussed

49

FACE DETECTION

Non Maximum Supression

• Merge overlapping detection boxes using non maximum supression

– Set a threshold on overlap ratio (e.g. 0.2)

– Use the rectangle with the highest (max) score from the classifier

50

FACE DETECTION

Non Maximum Supression

• Non Maximum Supression (NMS)

NMS

51

FACE DETECTION

Demo

• Face Detection– M. Nilsson, J. Nordberg and I. Claesson, ”Face Detection using Local SMQT Features and Split Up SNoW Classifier”, ICASSP, 2007, pp. II-589 -II-592.

52

FACE DETECTION

Bicycle Counting Project

• Bicycle Counting Project

– Note that other objects than faces can be used• For example Schneiderman tested on cars also

– In the Bicycle Counting project bicycles are considered• in low resolution to keep sensor cost low

53

FACE DETECTION

Bicycle Counting Project

• Design system that can classify bicycles…

54

FACE DETECTION

Bicycle Counting Project

• …in videos like this

55

FACE DETECTION

Bicycle Counting Project

• Start making positive samples

– Example of workflow

1

2

3

Zoom

Mark three points

Warp to patch

32x32

56

FACE DETECTION

Bicycle Counting Project

• Negatvie samples

– Start with some random images

– Utilize Bootstrapping (see Sung & Poggio) once classifier method is in place and build up database with more samples

57

FACE DETECTION

Bicycle Counting Project

• Design method to find bicycles

– Suggested method• Histogram of Oriented Gradients (HOG)

• Support Vector Machine (SVM)

– Aim:• Calculate bicycles going in both directions

58

FACE DETECTION

Bicycle Counting Project

• Example bicycle couting final result

Time <<< >>>

09:00 1 0

09:30 1 6

10:00 0 3

10:30 2 2

11:00 2 2

11:30 2 2

12:00 1 3

12:30 1 2

13:00 3 1

13:30 2 6

14:00 3 1

14:30 5 1

15:00 3 2

15:30 10 3

16:00 12 3