the problem with the blank slate approach derek hoiem beckman institute, uiuc * unpublished *
TRANSCRIPT
![Page 1: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/1.jpg)
The Problem with the Blank Slate Approach
Derek HoiemBeckman Institute, UIUC
* unpublished *
![Page 2: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/2.jpg)
Scene Understanding: What We Have
DOG
GROUND
TENT7
![Page 3: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/3.jpg)
Scene Understanding: What We Have (More Realistically)
CAR
GROUND
7
![Page 4: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/4.jpg)
Scene Understanding: What We Want
• What/where: Large, furry dog lying in opening of tent, leashed to tent
• Why: The tent provides shelter from the bright sun.
• What if: What would happen if I tried to pet the dog?
![Page 5: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/5.jpg)
The Blank Slate Approach to Recognition
LAB Histogram
Textons
Bag of SIFT
HOG
xxx x
x
x
x
xx
oo
oo
o
= Object Definition
Examples Image Features Classifier+ +_____
The Assumption: It is possible to learn to recognize an object from positive and negative examples (with the right features, classifier, etc)
![Page 6: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/6.jpg)
Problems with the Blank Slate Approach
1. Is inefficient
![Page 7: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/7.jpg)
Inefficient: Objects Defined in Isolation
Person
+/- Person Examples
Feature Space
Classifier
= Patch “Person”?
Car
+/- Car Examples
Feature Space
Classifier
= Patch “Car”?
Road
+/- Road Examples
Feature Space
Classifier
= Patch “Road”?
Dog
+/- Dog Examples
Feature Space
Classifier
= Patch “Dog”?
Building
+/- Building Examples
Feature Space
Classifier
= Patch “Building”?
![Page 8: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/8.jpg)
Problems with the Blank Slate Approach
1. Is inefficient.
2. Does not allow the underlying concept to be learned.
![Page 9: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/9.jpg)
translation
Signifiers
dog
Saussure 1916
barks
reproduces
four-legged
furry
face-licker
squirrel-chaser
Signified (Mental Concept)
recognition
A View from Semiotics (study of signs)
pet
![Page 10: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/10.jpg)
Magritte’s Warning
The Treachery of Images –Magritte 1929
pipe
Ceci aussi n’est pas une pipe .
![Page 11: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/11.jpg)
Problems with the Blank Slate Approach
1. Is inefficient.
2. Does not allow the underlying concept to be learned.
3. Might not allow accurate naming.
![Page 12: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/12.jpg)
A Flawed Argument for Blank Slate Approach
• People can identify objects in isolation• But inference ≠ learning - it might not be possible to
learn to identify objects from +/- examples
![Page 13: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/13.jpg)
Appearances are Just Symptoms
![Page 14: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/14.jpg)
Quantitative and Qualitative Gap
• Quantitative– CalTech 101: ~85% accuracy– CalTech 256: ~65% accuracy– MSRC 21: ~75% accuracy– PASCAL 2007: 0.1 – 0.45 average precision
• Qualitative
![Page 15: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/15.jpg)
Moving Beyond the Blank Slate
![Page 16: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/16.jpg)
Major Shifts in Approach 1. Progressive vision: from starting from scratch to
accumulating knowledge
![Page 17: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/17.jpg)
Major Shifts in Approach1. Progressive vision: from starting from scratch to
accumulating knowledge2. From isolated to relational models of objects
LAB Histogram
Textons
Bag of SIFT
HOG
xxx x
x
x
x
xx
oo
oo
o+
“dog” = “dog” =
barks1-4 ft high
four-legged
furry
face-licker
squirrel-chaser
pethas tail
OLD NEW
![Page 18: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/18.jpg)
Major Shifts in Approach1. Progressive vision: from starting from scratch to
accumulating knowledge2. From isolated to relational models of objects3. Beyond naming: infer properties and explain cause
and predict effect
What/where: Large, furry dog lying in opening of tent, leashed to tentWhy: The tent provides shelter from the bright sun.What if: What would happen if I tried to pet the dog?
![Page 19: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/19.jpg)
Existing Solutions are not Enough• More data
– May never have enough– No understanding
• Feature sharing or “deep learning”– May not be possible to discover underlying concepts from visual data– “Hidden” features cannot be communicated (or independently improved)– Still no understanding
• Compositional models– Only handles “part of” relation
• Context with graphical models– Hand-designed relational structure: brittle, not scalable
![Page 20: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/20.jpg)
Key Step: Incorporate Background Knowledge
LAB Histogram
Textons
Bag of SIFT
HOG
xxx x
x
x
x
xx
oo
oo
o
Examples Image Features Classifier+ +Background
Things
Attributes
Relations
Describe in Terms of Background Knowledge
Bias Feature Relevance
![Page 21: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/21.jpg)
Uses of Background Knowledge
Background knowledge (things that can be identified and relational structures) can
• Serves as building blocks for new concepts
• Helps determine what is likely to be important
• Allows properties to be inferred through class hierarchies
Early Category and Concept Development – Eds. Rakison and Oakes 2003
![Page 22: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/22.jpg)
Uses of Background Knowledge: Examples
• Using background knowledge as building blocks
• Explaining away large “stuff”
![Page 23: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/23.jpg)
Background Knowledge as Building Blocks
…
Object Appearance
Model
Boundary Appearance
Model
Surface Appearance
Model…
Independently Trained Appearance Models
Input ImageObject
Concept Model
Final Object Estimates
Part ofLooks likeNear …
Set of Possible Relations
+/- Training Examples
![Page 24: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/24.jpg)
Results on MSRC 21
Input Image Appearance Only Conceptual Model
Method Average Pixel Accuracy
Multisegs (baseline) 72%
Final Result (with knowledge) 80%
Best Published ~75%
![Page 25: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/25.jpg)
Most image area belongs to a few categories
Plot here
Percent of Fully Labeled Image Area
![Page 26: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/26.jpg)
Many of these can be recognized with high precision
Plot here
Precision Recall for 60 Classes (total area = 0.95)
![Page 27: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/27.jpg)
Being able to recognize the big stuff is useful
1. Recognize big stuff and ignore it– Efficient search strategies– Improve object discovery
2. Big stuff provides context
![Page 28: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/28.jpg)
Explaining Away the Big Stuff
Alpha mask = 1 - P(background)
Images
![Page 29: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/29.jpg)
Explaining Away the Big Stuff
Images
Alpha mask = 1 - P(background)
![Page 30: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/30.jpg)
Knowledge-Based Regularization
• Add penalties for assigning weights to features or relations that are unlikely to be relevant, inferred by– Superclasses (soft inheritance)– Guidance from semantic webs, text mining, or instruction– Similar objects– Functional attributes
![Page 31: The Problem with the Blank Slate Approach Derek Hoiem Beckman Institute, UIUC * unpublished *](https://reader036.vdocuments.us/reader036/viewer/2022062404/551b556e550346d41a8b6217/html5/thumbnails/31.jpg)
Conclusions
• The blank slate approach is not sufficient for scene understanding
• We need an approach that progressively builds on existing knowledge– Broader recognition (beyond nouns and verbs)– New representations to support relational models– New mechanisms for relational learning and knowledge
transfer– New types of evaluation– Integration with linguists, psychologists, etc.