“now! – that should clear up a few things around here!”

81
“Now! – That should clear up a few things around here!”

Upload: dulcie-russell

Post on 13-Jan-2016

212 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: “Now! – That should clear up a few things around here!”

“Now! – That should clear upa few things around here!”

Page 2: “Now! – That should clear up a few things around here!”

The Challenge of Recognition

Page 3: “Now! – That should clear up a few things around here!”

The Challenge of Recognition

Page 4: “Now! – That should clear up a few things around here!”

The Importance of Recognition

Navigation

Social interactions

Sexualselection

Foraging

Offspringcare

Dangeravoidance

Pavlovianconditioning

Objectrecognition

Objectrecognition

Page 5: “Now! – That should clear up a few things around here!”

The Importance of Recognition

“Not only did Dr. P fail to see faces, but he saw faces when there wereno faces to see. In the street he might pat the heads of water hydrantsand parking meters, taking these to be the heads of children; he would amiably address carved knobs on the furniture and be astounded whenthey did not reply. Such incidents multiplied, causing embarrassment, perplexity and fear.”

From ‘The man who mistook his wife for a hat’by Dr. Oliver Sacks

Page 6: “Now! – That should clear up a few things around here!”

Brain Mechanisms of Recognition

Pawan SinhaDepartment of Brain and Cognitive Sciences

MIT

• Current understanding• Ongoing research• Real-world applications• What the future holds

Page 7: “Now! – That should clear up a few things around here!”
Page 8: “Now! – That should clear up a few things around here!”

Current Understanding – Lesion Studies

Which parts of the brain are involved in visual recognition?

Initial clues:Kluver-Bucy syndrome (1939)

Temporal lobe lesions in humansand monkeys lead to:1. Visual agnosia, and2. Hypersexuality

Page 9: “Now! – That should clear up a few things around here!”

Responses of a Patient with Temporal Lobe Damage

(Farah, 1995)

Page 10: “Now! – That should clear up a few things around here!”

Current Understanding – Electrophysiology

Face-cell 1

Face-cell 2

Desimone, 1984

Are there specific neurons involved in visual recognition?

Page 11: “Now! – That should clear up a few things around here!”

Current Understanding – Brain Imaging

Kanwisher et al, 1997

Are there specific brain regions involved in visual recognition?

“Face area”

Page 12: “Now! – That should clear up a few things around here!”

Current Understanding – Summary

The temporal lobe is involved in visual recognition.

Page 13: “Now! – That should clear up a few things around here!”

Current Understanding – Summary

The temporal lobe is involved in visual recognition.

So what?

Page 14: “Now! – That should clear up a few things around here!”

Current Understanding – Summary

The temporal lobe is involved in visual recognition.

So what?

This doesn’t tell us how the brain recognizes objects.

Page 15: “Now! – That should clear up a few things around here!”

How Does the Brain Recognize Objects?

PrimaryVisualCortex

Oriented bar and edgedetector neurons

Hubel and Wiesel (1977)

Page 16: “Now! – That should clear up a few things around here!”

A Proposal for How the Brain Recognizes Objects

Edges!

Further processing

Marr (1979)

Page 17: “Now! – That should clear up a few things around here!”

A Proposal for How the Brain Recognizes Objects

Image

Edge map

Binocularprocessing

3D estimate Recognition!

Page 18: “Now! – That should clear up a few things around here!”

A Proposal for How the Brain Recognizes Objects

What underlies the researchers’ fascination for edges?

Fine edges, but not the coarse structure, are expected to be invariant to imaging variations.

In principle, this can make the recognition task very easy.

Page 19: “Now! – That should clear up a few things around here!”

The Problem with Edges…

In practice, fine edges turn out to be highly unstable.

Even after more than two decades of research, we havebeen unable to create a robust recognition system based onthe proposed model.

Page 20: “Now! – That should clear up a few things around here!”

The Problem with the Rest of the Model…

Image

Edge map

Binocularprocessing

3D estimate Recognition!

Recent experimental results suggest that recognitionmay precede the intermediate steps.

Sinha & Poggio, Nature, 1996Jones, Sinha, Poggio & Vetter, Current Biology, 1997Bulthoff, Bulthoff & Sinha, Nature Neuroscience, 1998

Page 21: “Now! – That should clear up a few things around here!”

The Million Dollar Question…

If recognition has to happen before the image can be finely analyzed, what then is the minimum image information that suffices for recognition?

Image

Edge map

Binocularprocessing

3D estimate Recognition!

?

Page 22: “Now! – That should clear up a few things around here!”

Reducing Image Information

An ecologically sound method: Progressive blurring(Equivalent to recognition at increasing distances)

RecognitionPerformance

Amount of Blur

Criterion

Page 23: “Now! – That should clear up a few things around here!”

Face Detection – Experimental Protocol

Subject’s task: Given an image, to determine whether it is a face.

Random patterns

Symmetric patterns

False-alarms from an artificial detectionsystem

Targets Distractors

All stimuli are presented at several blur levels.

Page 24: “Now! – That should clear up a few things around here!”

Face Detection – Results

Amount of Blur(radius of Gaussian)

Face Detection Performance

0

20

40

60

80

100

120

0 2 4 6 8 10 12 14 16

Blur Amount

Per

form

ance

(%)

Face Hit Rate

FP FA

Symm FA

Random FA

Hit Rate Criterion

FA Criterion

Page 25: “Now! – That should clear up a few things around here!”

Face Detection – Analysis

Are there any useful invariants at such high levels of blur?

Yes! Sinha, 1994, 1995; Lipson, Grimson, Sinha, 1997

Conceptlearning

algorithmsRatio-template: A stable

face-signature that comprises pairwise ordinal brightness

relationships

http://www.ai.mit.edu/projects/cbcl/web-pawan/cartoon/cartoon.html

Page 26: “Now! – That should clear up a few things around here!”

Face Detection – Analysis (contd.)

Does the brain use a ‘ratio-template’ like invariant fordetecting faces?

There is no direct evidence yet. However, there is someindirect evidence.

1. Neurons in the visual cortex have the required propertiesneeded to implement this model.

2. Computer implementations of this model yield goodperformance.

Page 27: “Now! – That should clear up a few things around here!”

Performance of Ratio-templates

Page 28: “Now! – That should clear up a few things around here!”

Benefits of Low-Resolution Approach

1. Permits face detection at a distance2. Is robust to image degradations3. Can generalize across facial variations4. Is computationally simple

Page 29: “Now! – That should clear up a few things around here!”

An Application of Our Face-detection System

The Nielsen People Meter

Page 30: “Now! – That should clear up a few things around here!”

Beyond Mere Detection – Face Recognition

Low-resolution images may suffice for detection, but, surely,we must have fine detailed information for recognition…

A popular approach to face recognition – feature matching

Page 31: “Now! – That should clear up a few things around here!”

Beyond Mere Detection – Face Recognition

Prediction of such an approach:

Amount of Blur

RecognitionPerformance

Page 32: “Now! – That should clear up a few things around here!”

Face Recognition – Experimental Protocol

Subject’s task: To recognize celebrity images subjected todifferent levels of blur.

Stimuli: Blur series for 36 celebrity faces.

Page 33: “Now! – That should clear up a few things around here!”

Face Recognition – Experimental Protocol

Page 34: “Now! – That should clear up a few things around here!”

29.3%

37.9%

56.1%

66.7%

82.1%87.4%91.2%

91.4%

0%0%0%2.4%

7.1%

29.8%

65.9%

79%

0%0%0%0%0%2%

28.6%

52.4%

94.1%

85.3%

0

10

20

30

40

50

60

70

80

90

100

REF 0 2 4 6 8 10 12 14

Blur Level

Pe

rce

nt

Co

rre

ct

Full_Face_NC_11

Internal_Intact_7

Internal_Broken_7

REF_Intact

REF_Broken

Face Recognition - Results

10 x 12 pixels/face

Page 35: “Now! – That should clear up a few things around here!”

29.3%

37.9%

56.1%

66.7%

82.1%87.4%91.2%

91.4%

0%0%0%2.4%

7.1%

29.8%

65.9%

79%

0%0%0%0%0%2%

28.6%

52.4%

94.1%

85.3%

0

10

20

30

40

50

60

70

80

90

100

REF 0 2 4 6 8 10 12 14

Blur Level

Pe

rce

nt

Co

rre

ct

Full_Face_NC_11

Internal_Intact_7

Internal_Broken_7

REF_Intact

REF_Broken

Face Recognition - Results

10 x 12 pixels/face70 x 70 pixels/face

Page 36: “Now! – That should clear up a few things around here!”

Face Recognition - inference

Overall face configuration supports much more robustrecognition as compared to individual features.

Page 37: “Now! – That should clear up a few things around here!”

Face Recognition - inference

Overall face configuration supports much more robustrecognition as compared to individual features.

“By and large, Dr. P recognized nobody: neither his family, norHis colleagues, nor his pupils. He recognized Einstein becauseHe picked up the characteristic moustache, and the same thingHappened with one or two other people. ‘Ach, Paul!’ he said,When shown a portrait of his brother. ‘That square jaw, thoseBig teeth –I would know Paul anywhere!’

From ‘The man who mistook his wife for a hat’By Dr. Oliver Sacks

Page 38: “Now! – That should clear up a few things around here!”

Face Recognition - inference

Overall face configuration supports much more robustrecognition as compared to individual features.

Sinha & Poggio, Nature, 1996

Page 39: “Now! – That should clear up a few things around here!”

The Two Million Dollar Question

Which aspects of facial configuration are importantand which are not?

Page 40: “Now! – That should clear up a few things around here!”

The Two Million Dollar Question

Which aspects of facial configuration are importantand which are not?

Caricaturists probably know the answer to this question…

Page 41: “Now! – That should clear up a few things around here!”

The Two Million Dollar Question

Which aspects of facial configuration are importantand which are not?

Caricaturists probably know the answer to this question…

…but it is difficult for them to articulate this intuitive knowledge, sometimes even to themselves.

Page 42: “Now! – That should clear up a few things around here!”

Caricaturists in their own words…

“When I’m having difficulty caricaturing someone, I just keep drawing by doing ten, twenty, thirty, forty sketches of the subject…”

- Bill Plympton

“I once spent about eighteen hours trying to caricatureBarry Manilow. It was frustrating not being able to draw someone who is so funny looking in the first place.”

- Taylor Jones

Page 43: “Now! – That should clear up a few things around here!”

The Hirschfeld Project

An attempt to make explicit the intuitive knowledgethat caricaturists possess and, in the process, togain insights into the brain’s face recognitionstrategies.

Page 44: “Now! – That should clear up a few things around here!”

The Hirschfeld Project - Goal

Given multiple caricatures corresponding to everyface image in a large set…

m f

ace

imag

esn caricatures

…the goal is to determine which facial measurements Caricaturists consistently emphasize (or de-emphasize) and how the extent of distortion relates to the deviation of a given face from the population average.

Page 45: “Now! – That should clear up a few things around here!”

The Hirschfeld Project - Caveats

“To effectively caricature a subject, I must feel that person’spersonality. A caricature to me is not just a big nose, big earsAnd a big head on a little body – it is much more.”

-Gerald ScarfeCaricaturist

Page 46: “Now! – That should clear up a few things around here!”

The Hirschfeld Project - Caveats

“I wouldn’t be surprised if some day they would create a computer program to do caricature, but it would be terrible.what a caricaturist does involves his whole life experience,education and some indefinable thing that makes it all work.”

-Robert GrossmanCaricaturist

“To effectively caricature a subject, I must feel that person’spersonality. A caricature to me is not just a big nose, big earsAnd a big head on a little body – it is much more.”

-Gerald ScarfeCaricaturist

Page 47: “Now! – That should clear up a few things around here!”

The Hirschfeld Project – in 10 Steps

Step 1:Caricature database creation(~50 caricaturists; ~100 faces of celebrities and others)

Step 2:Assessment of recognizability of caricatures

Step 3: Database digitization

Page 48: “Now! – That should clear up a few things around here!”

The Hirschfeld Project – in 10 Steps

Step 4:Measurements for each database entry

Page 49: “Now! – That should clear up a few things around here!”

The Hirschfeld Project – in 10 Steps

Step 4:Measurements for each database entry

Page 50: “Now! – That should clear up a few things around here!”

The Hirschfeld Project – in 10 Steps

Step 5:Average face construction and measurement

Page 51: “Now! – That should clear up a few things around here!”

The Hirschfeld Project – in 10 Steps

Step 6:Over-complete attribute set creation for each database entry

Step 7:Determination of ‘deviations’ of input face attributes w.r.t. average face

Point coordinates (a0, a1, a2, …, an)Lengths, ratios of lengths, angles,Areas, ratios of areas

dk = a k/a k - 1 inp avg

Dk = 0 -> Face attribute same as averageDk < 0 -> Face attribute smaller than averageDk > 0 -> Face attribute larger than average

Step 8:Determination of attribute exaggeration in caricatures

ek = a k/a k - 1 caric avg

Page 52: “Now! – That should clear up a few things around here!”

The Hirschfeld Project – in 10 Steps

Step 9:Plot e vs d for each attribute across all inputs and perform linear regression.

d

eBest linear fit

Slope of regression line provides a measure of theemphasis assigned to anattribute.

d

e Emphasis

De-emphasisAnti-emphasis

Page 53: “Now! – That should clear up a few things around here!”

The Hirschfeld Project – in 10 Steps

Step 10:Rank order attributes using regression line slopes.This provides an estimate of their relative salienceto the caricaturist and, perhaps, to the visual system.

Page 54: “Now! – That should clear up a few things around here!”

The Hirschfeld Project – Early Results

Prediction:The width and height dimensions of a face can be independently scaledwithout adversely affecting recognizability.

ab

cde

f

g

h

i

Important attributes:

A /Aa/bc/de/df/gh/gi/(g+h)

hair face

Page 55: “Now! – That should clear up a few things around here!”

Testing the Prediction of Independent Scalability in x & y

Page 56: “Now! – That should clear up a few things around here!”

Artificially Created Caricatures

We can also specialize this approach to generate caricatures in the style of a particular caricaturist.

One day, the Hirschfeld Project may allow us to create an artificial Mr. Hirschfeld.

Besides yielding clues about the brain’s recognition strategies, these results also provide a prescription for automatically creating caricatures.

Page 57: “Now! – That should clear up a few things around here!”

The Hirschfeld Project - Hurdles

Getting good caricaturists to help populate the database.

Page 58: “Now! – That should clear up a few things around here!”

The Hirschfeld Project - Hurdles

Getting good caricaturists to help populate the database.

[email protected]

Page 59: “Now! – That should clear up a few things around here!”

Interim Summary

1. The human brain can recognize objects well even invery low-resolution images.2. In contrast to previous proposals, we are formulatinga recognition scheme designed to utilize low-resolutionimage information. We call this the ‘Configural Recognition Scheme’3. We have made some headway on the task of face-detectionand are currently exploring face-recognition.

Page 60: “Now! – That should clear up a few things around here!”

Interim Summary

1. The human brain can recognize objects well even invery low-resolution images.2. In contrast to previous proposals, we are formulatinga recognition scheme designed to utilize low-resolutionimage information. We call this the ‘Configural Recognition Scheme’.3. We have made some headway on the task of face-detectionand are currently exploring face-recognition.

Understanding the brain’s strategies for recognition mayallow us to create useful artificial vision systems…

Page 61: “Now! – That should clear up a few things around here!”

Pedestrian Detection

Collaboration with T. PoggioPartly funded by Daimler-Benz

Driver’s attentionalwindow

Achtung!

Page 62: “Now! – That should clear up a few things around here!”

Pedestrian Detection – Early Results

Page 63: “Now! – That should clear up a few things around here!”

Logo Search

The US-PTO has more than 2 million logos on file.New logo registration requires a search to prevent design infringement.

Existing logo search method:

Retrieval via numerical design annotation system (USPTO)

Five pointed star: 01-01-03

Page 64: “Now! – That should clear up a few things around here!”

Logo Search - Challenges

Problem: Many logos do not have simple annotations!

Page 65: “Now! – That should clear up a few things around here!”

Logo Search - Results

Query

Query

Page 66: “Now! – That should clear up a few things around here!”

Industrial Inspection

3000 components/PCboard

60 seconds for inspection

Yield rate: 20%

Page 67: “Now! – That should clear up a few things around here!”

Industrial Inspection - Results

Page 68: “Now! – That should clear up a few things around here!”

Venturing into the Real World…

1st

MIT $50KEntrepreneurshipCompetition

Page 69: “Now! – That should clear up a few things around here!”

What Does the Future Hold?

- A great deal more research- More sophisticated artificial systems

The COG project:Brooks, Scassellati et al.MIT AI Lab

Page 70: “Now! – That should clear up a few things around here!”

What Does the Future Hold?

Face detection and tracking by Cog using Ratio-templates

Scassellati, 1998

Page 71: “Now! – That should clear up a few things around here!”

What Does the Future Hold?

Face motion imitation by Cog

Page 72: “Now! – That should clear up a few things around here!”

What Does the Future Hold?

- Better psycho-forensic systems

Current IdentiKit systems use a piecemeal approach:

Page 73: “Now! – That should clear up a few things around here!”

What Does the Future Hold?

- Better psycho-forensic systems

Some IdentiKit composites generated by a police operator:

Page 74: “Now! – That should clear up a few things around here!”

What Does the Future Hold?

- Better tools for visual information management

100 million

1997 199919961995199419931992

# of digital imageson the web

Page 75: “Now! – That should clear up a few things around here!”

What Does the Future Hold?

Current tools for visual information management

If an image is worth a thousand words, how canwe hope to describe it with just a few?

Textual annotations

Page 76: “Now! – That should clear up a few things around here!”

What Does the Future Hold? (contd.)

Desirable features for an image retriever

Graphical queries (“bring me images like this one”) No annotation required (content-based search) Quantitative measure of perceptual similarity

Page 77: “Now! – That should clear up a few things around here!”

What Does the Future Hold? (contd.)

A content-based image retriever

Page 78: “Now! – That should clear up a few things around here!”

What Does the Future Hold? (contd.)

Non-configural image retrievers

Page 79: “Now! – That should clear up a few things around here!”

What Does the Future Hold?

- Better tools for visual information filtering(the flip side of searching)

xxx xxx

xxxTrainingData

Content-basedImage filter

xxx xxx

xxxWebimages

xxxFilteredcontent

Page 80: “Now! – That should clear up a few things around here!”

What Does the Future Hold?

- Understanding recognition in the other sensory modalities

Page 81: “Now! – That should clear up a few things around here!”

Summary and Conclusion

• Current understanding• Ongoing research

•Face detection•Face recognition

• Real-world applications

• The future holds very exciting prospects