why categorize in computer vision?

Why Categorize

in Computer

Vision?

Why Use Categories?

People love categories!

Why Use Categories?

What if we didn’t have categories?

Humuhumunukunukuapua'a – “fish that grunts like a pig”

Why Use Categories?

Our minds work very intimately with categories– Every common noun in English is a category– Proper nouns name object instances– “this,” “that,” “the,” “my,” “yours,” etc. refer to

object instances anonymously

The Categorization Problem

The Categorization Problem

Categorization/Classification:Given a set of pre-defined categories, “bin” this imageDoes not necessarily require object detection

Vertical Dimension:1. General: “Animal”2. Basic: “Bird”3. Specific: “Robin”

The Categorization Problem

What kinds of categorization are computers good at? • Basic -- especially when using context clues• Specific -- due to low intra-class variation

The Categorization Problem

Bad at?• General, due to high intra-class variation and a lack of visual

cues

The Categorization Problem

Bad at?• Categories defined by non-visual characteristics

(like chairs)

Summary

• Semantic categories allow humans to convey a large amount of information concisely

• We want computers to be able to do the same• What work has been done on this problem?

Has it been successful?

Uses of Categorizati

on

Two Examples1. Using Context in Categorization2. Fine-Grain Object Classification

Caltech 101 (2003)

• Dataset for basic-level categorization• Objects from 101 classes• Famously difficult

Categorization with Context

Goal: Resolve ambiguity between similar-looking objects of different classes using the semantic context of an object

Rabinovich et al. (UC San Diego): Objects in ContextFirst paper to attempt to use context at the object levelPASCAL 2007 dataset

Categorization with Context

Categorization with Context

Approach1. Segment image to preserve some spatial data2. Perform Bag-of-Features to give an initial

ranked list of labels for each segment3. Use a Conditional Random Field (CRF)

framework to find agreement between segment labels

Categorization with Context

Bag-of-Features with Segmentation

Labeling Segments:

Confidence:

Conditional Random Field

Way to assign joint probabilities to elements without considering every possible combination in the training set

Conditional Random Field

Idea • Given set of segments S, set of labels C • Want to find p(C | S) without knowing p(S) • Associate a special graph with C that obeys the

“Markov Property” (uses S)• The ordered pair (S, C) is a CRF conditioned on

S

Conditional Random Field

Results

Results

False correction

Fine-Grain Classification

Fine-Grain Image Categorization

Challenge: need good classifiers that capture detail well

Fine-Grain Image Categorization

Yao et al. (Stanford): Combining Randomization and Discrimination for Fine-Grained Image Categorization

ApproachRandom forest with discriminative classifiersThis is a kind of machine learning framework that allows us to handle the fine detail in this problem.

Fine-Grain Image Categorization

Random Discriminative Tree

Approach• For each tree node, train an SVM classifier for a

randomly sampled image region• At each node, make a yes-or-no decision• Uses grayscale SIFT descriptors

Random Discriminative Tree

Results

Conclusion

• Semantic categories allow humans to convey a large amount of information concisely

• Categorization has been used for basic-level object detection and scene recognition

• Fine-grain categorization can provide us with expert-level classification of objects

• Not all categories are defined by visual characteristics!

Questions?