CVPR SLAM 2007

Using Group Prior to Identify People In Consumer Images

Andrew C. GallagherTsuhan Chen

Carnegie Mellon University Eastman Kodak Company

June 18, 2007

CVPR SLAM 2007

The ProblemConsumer image collections are growing exponentially each year.

Consumers want to search for images based on whom the image contains. And they don’t like to label images!

This is more than a face recognition problem. To best understand the semantics of who is in the images, we need to understand the people in the images.

CVPR SLAM 2007

Traditional Face Recognition

Determines the assignment of each person independently

Extract facial features Build a classifier that finds the most likely name, given the features.

But this method does not take full advantage of the available information!

CVPR SLAM 2007

The Group Prior for Learning The Semantics of People in

ImagesDetermine the joint assignment of all people in the image to names, using the group prior.

By the unique object constraint (UOC), an individual can appear only once in the image. The group prior characterizes the prior probability of certain groups of people appearing together in an image.

CVPR SLAM 2007

System Diagram

Ambiguous LabelResolution

Images (Faces)

Ambiguous Labels

Classifier Training

Labeled Faces

RecognizePeople

Annotated Image

HannahJonahHolly

AndyJonah

HollyJonah

Jonah Holly Jonah

Holly JonahAndyHannah

Unlabeled Image

Group Prior

Hannah

Holly

CVPR SLAM 2007

Recognizing a Person

When a single person is in the image:

: the set of all unique names

: a member of the set

: the features from person image

Posterior Probability

Individual PriorLikelihood

CVPR SLAM 2007

Recognizing Multiple People

The graph model represents the features and people in an image.

The graph encodes the independence assumptions of our model.

E.g. given the identity of a person, their features are independent of others in the image.

p1

f1

p2

f2

pM

fM

…

…

The Group Prior

CVPR SLAM 2007

Recognizing Multiple People

The joint probability function:

p1

f1

p2

f2

pM

fM

…

…

The Group Prior

: an index over the people in the image

: the set of all features for all people

: the set of people in the image

: a subset of ; a particular assignmentof a name to each person in .

The Group PriorLikelihood

CVPR SLAM 2007

Estimating the Group Prior

For pairs of names, the group prior is estimated by counting the number of images the pair appears, then normalizing.

The group prior for 3 or more people is estimated according to the group prior pairwise graphical model.

The Group Prior

The Individual Prior

CVPR SLAM 2007

System Diagram

Ambiguous LabelResolution

Images (Faces)

Ambiguous Labels

Classifier Training

Labeled Faces

RecognizePeople

Annotated Image

HannahJonahHolly

AndyJonah

HollyJonah

Jonah Holly Jonah

Holly JonahAndyHannah

Unlabeled Image

Group Prior

Hannah

Holly

CVPR SLAM 2007

Ambiguous Labels

Ambiguous labels indicate who is in the image, but not which person is which name.

A constrained clustering algorithm is used to ‘resolve’ the labels.

The resolved labels are used to learn the feature distribution for each name.

Andy Hannah Andy HannahJonah HannahJonah Hannah Jonah Holly

Hannah

Jonah Holly Jonah AndyJonah Andy

HannahJonah Andy

Hannah

Jonah Holly

Hannah

Hannah AndyHannah HollyJonah Andy

Hannah JonahHolly

Hannah AndyHannah Andy

-1.5 -1 -0.5 0 0.5 1-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5Feature Space

Feature 1

Fea

ture

2

Andy Hannah Andy HannahJonah HannahJonah Hannah Jonah Holly

Hannah

Jonah Holly Jonah AndyJonah Andy

HannahJonah Andy

Hannah

Jonah Holly

Hannah

Hannah AndyHannah HollyJonah Andy

Hannah JonahHolly

Hannah AndyJonah Andy

-1.5 -1 -0.5 0 0.5 1-0.5

0

0.5Feature Space

Feature 1

Fe

atu

re 2

Andy Hannah Andy HannahJonah HannahJonah Hannah Jonah Holly

Hannah

Jonah Holly Jonah AndyJonah Andy

HannahJonah Andy

Hannah

Jonah Holly

Hannah

Hannah AndyHannah HollyJonah Andy

Hannah JonahHolly

Hannah AndyJonah Andy

-1.5 -1 -0.5 0 0.5 1-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5Feature Space

Feature 1

Fe

atu

re 2

Andy Hannah Andy HannahJonah HannahJonah Hannah Jonah Holly

Hannah

Jonah Holly Jonah AndyJonah Andy

HannahJonah Andy

Hannah

Jonah Holly

Hannah

Hannah AndyHannah HollyJonah Andy

Hannah JonahHolly

Hannah AndyJonah Andy

-1.5 -1 -0.5 0 0.5 1-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5Feature Space

Feature 1

Fe

atu

re 2

Andy Hannah Andy HannahJonah HannahJonah Hannah Jonah Holly

Hannah

Jonah Holly Jonah AndyJonah Andy

HannahJonah Andy

Hannah

Jonah Holly

Hannah

Hannah AndyHannah HollyJonah Andy

Hannah JonahHolly

Hannah AndyJonah Andy

-1.5 -1 -0.5 0 0.5 1-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5Feature Space

Feature 1

Fe

atu

re 2

CVPR SLAM 2007

Classification with Group Prior

From the joint pdf, inference questions can be answered:Most Probable Explanation MAP

MAP- Most probable assignment of a particular person .

CVPR SLAM 2007

Experiment

The image collection:

Facial Features: Active Shape Model [Cootes95] based features, then PCA reduces to 5D.

Images 1197

Images with multiple people

188

No. Faces in these images

420

Individuals 5

• Ambiguously label a portion of the image collection, classify the identities of all the rest.

• Compare 4 Priors:• Group Prior (GP)• UOC Prior A binary

version of the GP that respects the UOC.

• The individual prior. • No Prior

• The performance is quantified for:• MAP • MPE

CVPR SLAM 2007

ResultsGroup Prior produces a large benefit.

Note: All images were ambiguously labeled; no people were explicitly labeled.

Example Classification

(from 10 labeled images)

Individual Prior / no Prior

Ethan Ethan

Unique Object Constraint

Hannah

Ethan

Group Prior Holly Ethan

0 0.2 0.4 0.6 0.8 10.2

0.4

0.6

0.8

1

Portion of Images Labeled

Cla

ssifi

catio

n R

ate

MAP

GroupUOCIndivnone

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

Portion of Images Labeled

Cla

ssifi

catio

n R

ate

MPE

GroupUOCIndivnone

CVPR SLAM 2007

Prior Work

Many face recognition methods- most ignore the issue of prior probabilities. [Zhao03]

Face recognition methods have been used to assist the labeling of image collections. [Zhang04]

In news photos, names from captions have been assigned to faces. [Berg04]

The co-occurrence of people in images has been studied, but not combined with image features. [Naaman05]

CVPR SLAM 2007

Conclusions

The group prior models the social relationships between individuals.

We learn feature distributions and relationships between the labels (people).

By using the group prior, recognition accuracy is significantly improved!