are categories necessary in a data-rich world? alexei (alyosha) efros cmu joint work with tomasz...
TRANSCRIPT
![Page 1: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/1.jpg)
Are Categories Necessaryin a Data-Rich World?
Alexei (Alyosha) EfrosCMU
Joint work with Tomasz Malisiewicz
![Page 2: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/2.jpg)
Acknowledgements
Talks by Moshe Bar; writings of Shimon Edelman
Murphy Big Book of Concepts
Weinberger Everything is Miscellaneous
Many great discussions with many colleagues, especially Tomasz Malisiewicz, James Hays, and Derek Hoiem
![Page 3: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/3.jpg)
“Unreasonable Effectiveness of Data”
• Parts of our world can be explained by elegant mathematics:– physics, chemistry, astronomy, etc.
• But much cannot:– psychology, genetics, economics, etc.
• Enter: The Magic of Big Data– Great advances in several fields:
• e.g. speech recognition, machine translation, search engines
[Halevy, Norvig, Pereira 2009]
![Page 4: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/4.jpg)
Categorization vs. The Data
4
Philosophy and PsychologyPhilosophy and Psychology
LanguageLanguageArts and recreationArts and recreation
LiteratureLiterature
TechnologyTechnology ReligionReligion
Weinberger, Everything is Miscellaneous
![Page 5: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/5.jpg)
categorization is losing…
vs.
![Page 6: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/6.jpg)
What’s in a name?
![Page 7: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/7.jpg)
“…That which we call a roseBy any other name would smell as sweet.”
“chair” category (PASCAL VOC)
“train” category (PASCAL VOC)
![Page 8: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/8.jpg)
Why Categorize?
1. Knowledge Transfer
2. Communication
Tigercat
dog
Leopard
![Page 9: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/9.jpg)
Classical View of Categories
•Dates back to Plato & Aristotle 1. Categories are defined by a
list of properties shared by all elements in a category
2. Category membership is binary
3. Every member in the category is equal
![Page 10: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/10.jpg)
Problems with Classical View• Humans don’t do this!
– People don’t rely on abstract definitions / lists of shared properties (Wittgenstein 1953, Rosch 1973)• e.g. define the properties shared by all “games” • e.g. are curtains furniture? Are olives fruit?
– Typicality• e.g. Chicken -> bird, but bird -> eagle, pigeon, etc.
– Language-dependent• e.g. “Women, Fire, and Dangerous Things” category is
Australian aboriginal language (Lakoff 1987)
– Doesn’t work even in human-defined domains• e.g. Is Pluto a planet?
![Page 11: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/11.jpg)
Visual Problems with Categories• A lot of categories are
functional
• Categories are 3D, but images are 2D
• World is too varied
Chair
car
train
![Page 12: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/12.jpg)
Typical HOG car detector
Felzenszwalb et al, PASCAL 2007
![Page 13: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/13.jpg)
Why not?
+
submitted to CVPR 2011
![Page 14: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/14.jpg)
Semantic -> Visual Categories
Aspect ratiosplits
“parts” Poselets
All use a priori domain information
![Page 15: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/15.jpg)
Fundamental Problem with Categorization
Making decisions too early!Like Amazinn.com, can we just categorize at
run-time, once we know the task!
![Page 16: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/16.jpg)
On-the-fly Categorization?
1. Knowledge Transfer
2. Communication
![Page 17: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/17.jpg)
Association instead of categorization
Ask not “what is this?”, ask “what is this like”
– Moshe Bar
• Exemplar Theory (Medin & Schaffer 1978, Nosofsky 1986, Krushke 1992)–categories represented in terms of remembered objects
(exemplars)–Similarity is measured between input and all exemplars–think non-parametric density estimation
![Page 18: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/18.jpg)
“What is this?”
CarCarCar
Road
Building
Input Image
He 2004, Tu 2004, Shotton 2006, Galleguillos 2008, Fei-Fei 2009, Gould 2009, etc.
![Page 19: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/19.jpg)
“What is this like?”
Malisiewicz & Efros, CVPR’08
![Page 20: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/20.jpg)
20
What is the ultimate goal?
• Understanding / Parsing Images
• A “what is it like?” machine
![Page 21: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/21.jpg)
21
Our Contributions
• Posing Recognition as Association–Use large number of object exemplars
21
•Learning Object Similarity–Different distance function per exemplar
•Recognition-Based Object Segmentation– Use multiple segmentation approach
![Page 22: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/22.jpg)
Visual Associations
• How are objects similar?
Shape
Shape
Color
Color
![Page 23: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/23.jpg)
Distance “Similarity” Functions
• Positive Linear Combinations of Elementary Distances Computed Over 14 Features
Building e Distance Function
Building e
![Page 24: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/24.jpg)
Learning Object Similarity
• Learn a different distance function for each exemplar in training set
• Formulation is similar to Frome et al [1,2][1] Andrea Frome, Yoram Singer, Jitendra Malik. "Image Retrieval and Recognition Using Local Distance Functions." In NIPS, 2006.
[2] Andrea Frome, Yoram Singer, Fei Sha, Jitendra Malik. "Learning Globally-Consistent Local Distance Functions for Shape-Based Image Retrieval and Classification." In ICCV, 2007.
![Page 25: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/25.jpg)
25
Non-parametric density estimation
Color Dimension
Sh
ap
e D
imen
sio
n
Class 1Class 2
Class 3
![Page 26: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/26.jpg)
26
Non-parametric density estimation
Color Dimension
Sh
ap
e D
imen
sio
n
Class 1Class 2
Class 3
![Page 27: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/27.jpg)
27
Non-parametric density estimation
Color Dimension
Sh
ap
e D
imen
sio
n
Class 1Class 2
Class 3
![Page 28: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/28.jpg)
Learning Distance Functions
28Dshape
Dcolor
Focal Exemplar
“similar” side
DecisionDecisionBoundaryBoundary
“dissimilar” side
Don’t Care
![Page 29: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/29.jpg)
Visualizing Distance Functions (Training Set)
Query
Query
Top Neighbors with Tex-Hist Dist
Top Neighbors with Learned Dist
![Page 30: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/30.jpg)
Visualizing Distance Functions (Training Set)
![Page 31: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/31.jpg)
Labels Crossing Boundary
![Page 32: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/32.jpg)
Object Segmentation via Recognition
• Generate Multiple Segmentations (Hoiem 2005, Russell 2006, Malisiewicz 2007)
– Mean-Shift and Normalized Cuts
– Use pairs and triplets of adjacent segments
– Generate about 10,000 segments per image
• Enhance training with bad segments
• Apply learned distance functions to bottom-up segments
![Page 33: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/33.jpg)
33
Example AssociationsBottom-Up Segments
![Page 34: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/34.jpg)
34
Quantitative Evaluation
34
Object hypothesis is correct if labels match and OS > .5
*We do not penalize for multiple correct overlapping associations
OS(A,B) = Overlap Score = intersection(A,B) / union(A,B)
![Page 35: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/35.jpg)
35
Towards Image Parsing
Need for Context 35
![Page 36: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/36.jpg)
Image Parsing with Context
![Page 37: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/37.jpg)
Bush’s Memex (1945)• Store publications, correspondence, personal work, on
microfilm• Items retrieved rapidly using index codes
–Builds on “rapid selector”• Can annotate text with margin notes, comments• Can construct a trail through the material and save it
–Roots of hypertext• Acts as an external memory
![Page 38: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/38.jpg)
Visual Memex, a proposal[Malisiewicz & Efros]
Nodes = instancesEdges = associations
types of edges:• visual similarity• spatial, temporal co-occurrence• geometric structure• language• geography•..
New object
![Page 39: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/39.jpg)
Torralba’s Context Challenge
![Page 40: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/40.jpg)
2
1
Torralba’s Context Challenge
Slide by Antonio Torralba
![Page 41: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/41.jpg)
Torralba’s Context Challenge
Chance ~ 1/30000 Slide by Antonio Torralba
![Page 42: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/42.jpg)
Our Challenge Setup
Malisiewicz & Efros, NIPS’09
![Page 43: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/43.jpg)
3 models
• Visual Memex: exemplars, non-parametric object-object relationships– Recurse through the graph
• Baseline: CoLA: categories, parametric object-object relationships
• Reduced Memex: categories, non-parametric relationships
![Page 44: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/44.jpg)
Qual. results
![Page 45: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/45.jpg)
![Page 46: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/46.jpg)
Quant. results
![Page 47: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/47.jpg)
Next Step: top-down segmentation
Visual Memex
A
B
C
B’
C’
![Page 48: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/48.jpg)
Take Home Message
• Categorization is not a goal in itself–Rather, it is a means for transferring knowledge onto a new instance
• Skipping explicit categorization might make things easier, not harder–The “harder intermediate problem” syndrome
• Keeping around all your data isn’t so bad… –you never know when you will need it
![Page 49: Are Categories Necessary in a Data-Rich World? Alexei (Alyosha) Efros CMU Joint work with Tomasz Malisiewicz](https://reader037.vdocuments.us/reader037/viewer/2022110303/551476b3550346b0158b532b/html5/thumbnails/49.jpg)
Questions?
+