Feature Detection and Feature Detection and Emotion RecognitionEmotion Recognition

Chris MatthewsChris Matthews

Advisor: Prof. CotterAdvisor: Prof. Cotter

Motivation #1: Attempt to Motivation #1: Attempt to Answer a Long-Existing Answer a Long-Existing

QuestionQuestion Used to definitively Used to definitively

characterize what characterize what expressions the expressions the Mona Lisa is Mona Lisa is displaying (F.Y.I., displaying (F.Y.I., she is 83% happy, she is 83% happy, 9% disgusted, 6% 9% disgusted, 6% fearful and 2% fearful and 2% angry, according to angry, according to BBC News)BBC News)

Motivation #2: Create “Life-Motivation #2: Create “Life-Like” RobotsLike” Robots

Create Create convincingconvincing artificial artificial intelligence.intelligence.

Motivation #3: Enhance Motivation #3: Enhance Society!Society!

Currently being used to teach autistic Currently being used to teach autistic children to pick out facial subtleties children to pick out facial subtleties and their corresponding emotionsand their corresponding emotions

MethodologyMethodology

FEATURE DETECTIONFEATURE DETECTION Isolate and crop particular areas of the faceIsolate and crop particular areas of the face

EMOTION RECOGNITIONEMOTION RECOGNITIONTrainingTraining

Train neural networks for each areaTrain neural networks for each area Combine the resultants from each and come out with a Combine the resultants from each and come out with a

definitive resultdefinitive result Alter variables of the networks by trial-and-error until Alter variables of the networks by trial-and-error until

the desired results are achievedthe desired results are achieved

TestingTesting Input new photos into the trained network and check Input new photos into the trained network and check

resultsresults

Feature Detection:Feature Detection:SUSAN filtering for Edge SUSAN filtering for Edge

Detection Detection Because no derivatives are Because no derivatives are

implemented in SUSAN, the implemented in SUSAN, the algorithm excels in noisy imagesalgorithm excels in noisy images

Massive Problem: Boolean images don’t necessarily Massive Problem: Boolean images don’t necessarily make Computer Vision problems easier!make Computer Vision problems easier!

Edge Detection with brightness threshold = 33

Mouth not fully Mouth not fully enclosedenclosed

Only the pupil of the Only the pupil of the left eye is enclosedleft eye is enclosed

Even if everything Even if everything was perfectly was perfectly encapsulated, how encapsulated, how would one make would one make sense of the sense of the detected objects?detected objects?

Original

Lesson Learned: Complete Lesson Learned: Complete Automation is difficult!Automation is difficult!

New methodology for isolating parts New methodology for isolating parts of the face – manual labor.of the face – manual labor.

Draw matrices over the approximate Draw matrices over the approximate area of interestarea of interest

Apply filters to detect the actual Apply filters to detect the actual object of interestobject of interest

Crop again based on those findingsCrop again based on those findings

Example: The Uncentered Example: The Uncentered EyeEye

Uncentered Eye

10 20 30 40 50 60

10

20

30

40

50

60

The neural network The neural network will perform poorly will perform poorly if there is variance if there is variance in either the x or y in either the x or y directions, from directions, from photo to photophoto to photo

Circular Objects Located

0.25

0.27

Voila!Voila!

Centered Eye

5 10 15 20 25 30

5

10

15

20

25

30

35

40

On to the Emotion On to the Emotion Training…Training…

Once the areas have been defined Once the areas have been defined and scaled, they can be used as and scaled, they can be used as inputs to neural networksinputs to neural networks

Introduction to Neural Introduction to Neural Networks: The PerceptronNetworks: The Perceptron

Perceptron ImplementationPerceptron Implementation

• Initialize weight matrix and bias Initialize weight matrix and bias array to small, random values. array to small, random values.

• Feed an image through the networkFeed an image through the network• Calculate the errorCalculate the error• Readjust the weight matrix and bias Readjust the weight matrix and bias

array based on the errorarray based on the error• Iteratively train the network using a Iteratively train the network using a

dictionary of photos.dictionary of photos.

Yet another problem!Yet another problem!

Each neuron has one weight value Each neuron has one weight value for each pixelfor each pixel

Weight matrix is too large to train!Weight matrix is too large to train!

Solution: PCASolution: PCA

Principle Component Analysis Principle Component Analysis generates a set of eigenvectors. generates a set of eigenvectors.

Each picture can be reconstructed Each picture can be reconstructed using a weighted sum of these using a weighted sum of these eigenvectors.eigenvectors.

Original Image

5 10 15 20 25 30

5

10

15

20

25

30

35

40

Reconstruction

5 10 15 20 25 30

5

10

15

20

25

30

35

40

Eigenvector #1

5 10 15 20 25 30

5

10

15

20

25

30

35

40

Eigenvector #2

5 10 15 20 25 30

5

10

15

20

25

30

35

40

Eigenvector #3

5 10 15 20 25 30

5

10

15

20

25

30

35

40

Eigenvector #4

5 10 15 20 25 30

5

10

15

20

25

30

35

40

Eigenvector #5

5 10 15 20 25 30

5

10

15

20

25

30

35

40

Eigenvector #6

5 10 15 20 25 30

5

10

15

20

25

30

35

40

Eigenvector #7

5 10 15 20 25 30

5

10

15

20

25

30

35

40

Average

5 10 15 20 25 30

5

10

15

20

25

30

35

40

Eigenvector #8

5 10 15 20 25 30

5

10

15

20

25

30

35

40

Final ArchitectureFinal Architecture

Use a set of adaptive Use a set of adaptive backpropagation networks, training backpropagation networks, training on PCA coefficients. on PCA coefficients.

Use majority rules to determine the Use majority rules to determine the emotion. emotion.

ResultsResults

Training with 60 photos yielded Training with 60 photos yielded 100% accuracy mapping to only two 100% accuracy mapping to only two targets: happy and sadtargets: happy and sad

Training with 112 photos yielded Training with 112 photos yielded 60% accuracy mapping to four 60% accuracy mapping to four targets: angry, fearful, happy, and targets: angry, fearful, happy, and sad.sad.

Future WorkFuture Work

Find larger and more diverse image Find larger and more diverse image dictionariesdictionaries

Improve Feature DetectionImprove Feature Detection Read Psychological Journals and Read Psychological Journals and

apply their findings into the apply their findings into the algorithmsalgorithms

Questions?Questions?

A gross simplification of how A gross simplification of how SUSAN worksSUSAN works

2

21 rn

Smallest Univalue Segment Smallest Univalue Segment Assimilating NucleusAssimilating Nucleus

Edge if n = (½)*pi*r^2Edge if n = (½)*pi*r^2

Corner if n << (½)*pi*r^2Corner if n << (½)*pi*r^2