machine learning & computer vision

RECOGNISING FACESMACHINE LEARNING & COMPUTER VISION: 101

SAKSHAM GAUTAM

08.06.2016 Saksham Gautam

LET’S START WITH A QUICK SHOW OF HANDS

HOW MANY OF YOU…

▸ have heard about machine learning?

▸ have used machine learning in your projects?

▸ have implemented any ML algorithm from scratch?

▸ have done Andrew Ng’s (or other) courses on ML?

▸ understand that deep learning uses neural network?

▸ still remember what the kernel trick is?

08.06.2016 Saksham Gautam

SHOW OF HANDS ON YOUR FAMILIARITY WITH COMPUTER VISION

HOW ABOUT…

▸ know how an image can be represented as a matrix?

▸ have used openCV or MATLAB?

▸ understand how convolution can be used to detect edges?

▸ know the role of scale space in computer vision?

▸ remember how eigenvectors can be used for face recognition?

08.06.2016 Saksham Gautam

WHAT DO WE WANT TO ACHIEVE?

FACE DETECTION & RECOGNITION

http://docs.opencv.org/master/d7/d8b/tutorial_py_face_detection.html#gsc.tab=0

FACE

MONA LISA

NOT A FACE!

08.06.2016 Saksham Gautam

TO RECOGNISE AND LABEL OBJECTS IN AN IMAGE

OBJECT RECOGNITION

08.06.2016 Saksham Gautam

BASIC STEPS FOR FACE RECOGNITION

BUT HOW EXACTLY?

1. Capture image

2. Filter out noise

3. Find face in the image

4. Create a similarity metric and a model (Training)

5. Match any given face to one from the database

6. Return the closest match with the probability

08.06.2016 Saksham Gautam

FACE RECOGNITION CAN BE BROKEN DOWN INTO SIMPLE STEPS

BUILDING BLOCKS

RAW IMAGE PROCESSED IMAGE FEATURES

MACHINE LEARNING ALGORITHM

Training

Validation

MODEL

RAW IMAGE

DECISION

How can I capture image?

Remove any noise?

What’s the information in the image?

Can we match patterns?

08.06.2016 Saksham Gautam

HUMAN SEES BEAUTY BUT COMPUTER SEES NUMBERS

WHAT IS THIS?

=

08.06.2016 Saksham Gautam

PIXEL VALUES CAN BE THOUGHT OF AS ‘ELEVATION’ IN IMAGE PLANE

MONALISA IN 3D?

08.06.2016 Saksham Gautam

FILTER - I, MEAN FILTER CAN REMOVE RANDOM NOISE

MEAN FILTER

08.06.2016 Saksham Gautam

FILTERS - II, MEDIAN FILTER CAN REMOVE SALT & PEPPER NOISE

MEDIAN FILTER

08.06.2016 Saksham Gautam

FILTERS-III, ADVANCED FILTERS CAN REMOVE DIFFERENT TYPES OF NOISE

ADVANCED FILTERS

▸ Bilateral filters

▸ Adaptive bilateral filters

▸ …

08.06.2016 Saksham Gautam

38 53 38 70 84

5 56 69 82 98

34 87 85 5 40

25 99 43 69 76

11 61 86 94 59

-1 0 1

-1 0 1

-1 0 1

CONVOLUTION CAN BE APPLIED VERY QUICKLY ON AN IMAGE

CONVOLUTION

*

38 0 -38

5 0 -69

34 0 -85

-115

-115 56 16

-69 112 12

-93 3 6

08.06.2016 Saksham Gautam

CONVOLUTION CAN BE USED FOR COMPUTING IMAGE GRADIENT

IMAGE GRADIENT

0 0 0

100 100 100

0 50 100

100 50 0

-1 0 1*0

0

100

-100

=

08.06.2016 Saksham Gautam

EDGES AND CORNERS ARE FEATURES IN AN IMAGE

SOBEL FILTER FOR DETECTING EDGES

HARRIS CORNER DETECTOR

-1 0 1

-2 0 2

-1 0 1

Gx =-1 -2 -1

0 0 0

1 2 1

Gy =

$python sobel-filter.py

$python harris-corner.py

08.06.2016 Saksham Gautam

MORE ROBUST FEATURES CAN BE USED FOR OBJECT RECOGNITION

SIFT, SURF, HOG

▸ More advanced features can be used for scale invariance

▸ Some are robust even under varying lighting conditions

▸ These serve as the starting point for the ML part

08.06.2016 Saksham Gautam

CASCADES OF FILTERS ON AN IMAGE CAN BE USED FOR DETECTING FACES

DETECTING FACES

http://siret.ms.mff.cuni.cz/facereco/method/

$python viola-jones.py

08.06.2016 Saksham Gautam

FEATURES FRO THE FACE CAN BE FED TO AN ML ALGORITHM

BUILDING BLOCKS

RAW IMAGE PROCESSED IMAGE FEATURES

MACHINE LEARNING ALGORITHM

Training

Validation

MODEL

RAW IMAGE

DECISION

PERFORMANCE (P) OF A METHOD FOR A TASK (T) INCREASES WITH EXPERIENCE (E)

Tom Mitchell

BTW, WHO LEARNS? THE MACHINE, REALLY?

08.06.2016 Saksham Gautam

PROBABILITY AND STATISTICS CAN HELP ANSWER MANY QUESTIONS

T-SHIRT SIZE FOR THE SUMMITMAYBE I SHOULD HAVE PICKED ‘M’ INSTEAD OF ’S'

08.06.2016 Saksham Gautam

EXACT SOLUTION IS NOT ALWAYS POSSIBLE

T-SHIRT SIZE ~ MY BODY’S MEASUREMENTS

08.06.2016 Saksham Gautam

MAXIMUM LIKELIHOOD ESTIMATE HELPS ON THE FACE OF UNCERTAINTY

CLASSIFICATION PROBLEM?

Length #2

Length #1

S

MLength #1

Length #2

S

08.06.2016 Saksham Gautam

MEMBERSHIP TO A CLUSTER CAN BE USED FOR CLASSIFICATION

CLUSTERING

Length #2

Length #1

S

M

08.06.2016 Saksham Gautam

MARGINS CAN BE USED FOR SEPARATING CLASSES

LARGE MARGIN CLASSIFIER

Length #2

Length #1

08.06.2016 Saksham Gautam

LOG(DISTANCE FROM THE HYPERPLANE) = PROBABILITY

LOGISTIC REGRESSION

Length #2

Length #1

08.06.2016 Saksham Gautam

LARGE MARGIN CLASSIFIER CAN BE USED FOR DETECTING FACES

FEEDING VALUES FROM VIOLA JONES FILTER

X1

X2FACE

NOT A FACE

08.06.2016 Saksham Gautam

FEATURES SHOULD MAXIMISE VARIANCE (SCATTER)

EIGENVECTORS

V1

V2

X1

X2

08.06.2016 Saksham Gautam

MATRIX DECOMPOSITION ON FACES CAN BE USED TO FIND EIGENFACES

FACIAL FEATURES AS EIGENVECTORS

V1

V2

Projection

08.06.2016 Saksham Gautam

ACCURACY ITSELF IS NOT ENOUGH

CROSS VALIDATION AND F SCORE

DUMMY CANCER DETECTOR

Data

No

ACCURACY = 98%

08.06.2016 Saksham Gautam

IMAGE CAN BE DIRECTLY FED INTO NEURAL NETWORK

NEURAL NETWORK

08.06.2016 Saksham Gautam

GO WITH THE HYPE, BUT WITH CARE

DEEP LEARNING ~ MASSIVE NEURAL NETWORK

▸ Learning algorithm is the same, i.e. back propagation

▸ Has the same problem with overfitting

▸ Can be used for feature extraction and selection

▸ Mathematical foundations for neural network still not “perfect”

▸ Pointer: https://www.tensorflow.org from Google

08.06.2016 Saksham Gautam

MACHINE LEARNING PIPELINE

SUMMARY

RAW IMAGE PROCESSED IMAGE FEATURES

MACHINE LEARNING ALGORITHM

Training

Validation

MODEL

RAW IMAGE

DECISION

REFERENCES

• OpenCV Documentation. http://docs.opencv.org/3.1.0/#gsc.tab=0

• Andrew Ng. Machine Learning Courser on Coursera. http://www.coursera.org/learn/machine-learning

• Christopher Bishop. Machines that Learn. https://www.youtube.com/watch?v=icaA7gVxqSs

• Video Lecture on Face Detection and Tracking. https://www.youtube.com/watch?v=WfdYYNamHZ8

• Adam Harvey explains Viola-Jones Face Detection. http://www.makematics.com/research/viola-jones/

• Christopher, M. Bishop. "Pattern recognition and machine learning." Company New York 16.4 (2006): 049901.

• Bradski, Gary, and Adrian Kaehler. Learning OpenCV: Computer vision with the OpenCV library. " O'Reilly Media, Inc .", 2008

• Solem, Jan Erik. Programming Computer Vision with Python: Tools and algorithms for analyzing images. " O'Reilly Me dia, Inc.", 2012.

• Hartley, Richard, and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.

BACKUP SLIDES

08.06.2016 Saksham Gautam

IMAGE GRADIENT IS THE BASIS OF FEATURE EXTRACTION

GRADIENT IMAGE

-1 0 1gx =-1

0

1

gy =* A * A

08.06.2016 Saksham Gautam

EVERY SIGNAL CAN BE DECOMPOSED TO SINES AND COSINES

FOURIER TRANSFORM

▸ Frequency can be thought of as information in the image

▸ Fourier Transform can be used to decompose a signal into these components

▸ Signal can be multiplied with filter in frequency domain

▸ Multiplication in frequency domain is convolution in time domain