Slide 1

Large-Scale Object Recognition using Label Relation Graphs Jia Deng 1,2, Nan Ding 2, Yangqing Jia 2, Andrea Frome 2, Kevin Murphy 2, Samy Bengio 2, Yuan Li 2, Hartmut Neven 2, Hartwig Adam 2 University of Michigan 1, Google 2 Slide 2 Object Classification Assign semantic labels to objects Corgi Puppy Dog Cat Slide 3 Object Classification Assign semantic labels to objects Probabilities 0.9 0.8 0.9 0.1 Corgi Puppy Dog Cat Slide 4 Object Classification Assign semantic labels to objects Feature Extractor FeaturesClassifier Probabilities 0.9 0.8 0.9 0.1 Corgi Puppy Dog Cat Slide 5 Object Classification Multiclass classifier: Softmax Corgi Puppy Dog Cat / / / / + Assumes mutual exclusive labels. 0.2 0.4 0.3 0.1 Independent binary classifiers: Logistic Regression Corgi Puppy Dog Cat 0.2 0.8 0.6 0.4 No assumptions about relations. Slide 6 Object labels have rich relations Corgi Puppy Dog Cat Exclusion Hierarchical Dog Cat Corg i Puppy Overlap Softmax: all labels are mutually exclusive Logistic Regression: all labels overlap Slide 7 Goal: A new classification model Respects real world label relations Corgi Puppy Dog Cat 0.9 0.8 0.9 0.1 Corgi Puppy Dog Cat Slide 8 Visual Model + Knowledge Graph Corgi Puppy Dog Cat Visual Model 0.9 0.8 0.9 0.1 Knowledge Graph Joint Inference Assumption in this work: Knowledge graph is given and fixed. Slide 9 Agenda Encoding prior knowledge (HEX graph) Classification model Efficient Exact Inference Experiments Conclusion and Future Work Slide 10 Agenda Encoding prior knowledge (HEX graph) Classification model Efficient Exact Inference Experiments Conclusion and Future Work Slide 11 Hierarchy and Exclusion (HEX) Graph Corgi Puppy Dog Cat Exclusion Hierarchical Hierarchical edges (directed) Exclusion edges (undirected) Slide 12 Examples of HEX graphs Car Bird Dog Cat Male Female Person Child Boy Round Red Shiny Thick Mutually exclusiveAll overlapping Combination Girl Slide 13 State Space: Legal label configurations DogCatCorgiPuppy 0000 0001 0010 0011 1000 1100 1101 Corgi Puppy Dog Cat Each edge defines a constraint. Slide 14 State Space: Legal label configurations Corgi Puppy Dog Cat DogCatCorgiPuppy 0000 0001 0010 0011 1000 1100 1101 Hierarchy: (dog, corgi) cant be (0,1) Each edge defines a constraint. Slide 15 State Space: Legal label configurations Corgi Puppy Dog Cat DogCatCorgiPuppy 0000 0001 0010 0011 1000 1100 1101 Exclusion: (dog, cat) cant be (1,1) Hierarchy: (dog, corgi) cant be (0,1) Each edge defines a constraint. Slide 16 Agenda Encoding prior knowledge (HEX graph) Classification model Efficient Exact Inference Experiments Conclusion and Future Work Slide 17 HEX Classification Model Pairwise Conditional Random Field (CRF) Input scores Binary Label vector Slide 18 HEX Classification Model Pairwise Conditional Random Field (CRF) Binary Label vector Unary: same as logistic regression Input scores Slide 19 HEX Classification Model Pairwise Conditional Random Field (CRF) Binary Label vector All illegal configurations have probability zero. Unary: same as logistic regression If violates constraints Otherwise 0 Pairwise: set illegal configuration to zero Input scores Slide 20 HEX Classification Model Pairwise Conditional Random Field (CRF) Binary Label vector Partition function: Sum over all (legal) configurations Input scores Slide 21 HEX Classification Model Pairwise Conditional Random Field (CRF) Binary Label vector Probability of a single label: marginalize all other labels. Input scores Slide 22 Special Case of HEX Model Softmax Car Bird Dog Cat Round Red Shiny Mutually exclusive All overlapping Thick Logistic Regressions Slide 23 Learning Corgi Puppy Dog Cat DNN Label: Dog Maximize marginal probability of observed labels Back Propagation Dog Corgi Puppy Cat 1 ? ? ? Slide 24 Agenda Encoding prior knowledge (HEX graph) Classification model Efficient Exact Inference Experiments Conclusion Slide 25 Nave Exact Inference is Intractable Inference: Computing partition function Perform marginalization HEX-CRF can be densely connected (large treewidth) Slide 26 Observation 1: Exclusions are good Car Bird Dog Cat Lots of exclusions Small state space Efficient inference Realistic graphs have lots of exclusions. Rigorous analysis in paper. Number of legal states is O(n), not O(2 n ). Slide 27 Observation 2: Equivalent graphs Dog Cat Corgi Puppy Pembroke Welsh Corgi Cardigan Welsh Corgi Dog Cat Corgi Puppy Pembroke Welsh Corgi Cardigan Welsh Corgi Slide 28 Observation 2: Equivalent graphs Sparse equivalent Small Treewidth Dynamic programming Dog Cat Corgi Puppy Pembroke Welsh Corgi Cardigan Welsh Corgi Dog Cat Corgi Puppy Pembroke Welsh Corgi Cardigan Welsh Corgi Dog Cat Corgi Puppy Pembroke Welsh Corgi Cardigan Welsh Corgi Dense equivalent Prune states Can brute force Slide 29 A B F B E D C G F B C F HEX Graph Inference A B E D C G F A B E D C G F A B E D C G F A B F B E D C G F B C F 1. Sparsify (offline) 3.Densify (offline) 2.Build Junction Tree (offline) 4.Prune Clique States (offline) 5. Message Passing on legal states (online) Slide 30 Agenda Encoding prior knowledge (HEX graph) Classification model Efficient Exact Inference Experiments Conclusion and Future Work Slide 31 Exp 1: Learning with weak labels Many basic category labels Few fine-grained labels Dog Corgi Animal Weak labels: No information on subcategories. Slide 32 Corgi Puppy Dog Cat DNN Label: Dog Dog Corgi Puppy Cat 1 ? ? ? Hypothesis: HEX models can improve fine-grained recognition using basic level labels. Exp 1: Learning with weak labels Slide 33 ILSVRC 2012: relabel or weaken a portion of fine-grained leaf labels to basic level labels. Evaluate on fine-grained recognition Exp 1: Learning with weak labels Dog Corgi Animal Husky Dog Corgi Animal Husky Relabel Training (weakened labels) Test Dog Corgi Animal Husky Original ILSVRC 2012 (leaf labels) Slide 34 ILSVRC 2012: relabel or weaken a portion of fine-grained leaf labels to basic level labels. Evaluate on fine-grained recognition. Consistently outperforms baselines. Exp 1: Learning with weak labels Top 1 accuracy (top 5 accuracy) Slide 35 Exp 2: Zero-Shot Recognition using Object-Attribute Knowledge Animals with Attribute (AwA) dataset (Lampert et al. 2009) Training: Observe only a subset of animal labels. Given all animal-attribute relations Indirectly learns attributes. Test: predict new classes with no images in training. DAP (Lampert et al.)IAP (Lampert et al.)Ours 40.5%27.8%38.5% polar bear black: no white: yes brown: no stripes: no polar bear black white zebra brown stripes zebra black: yes white: yes brown: no stripes: yes Slide 36 Related Work Multilabel Annotation & Hierarchy [Lampert et al. NIPS11] [Chen et al. ICCV11] [Bi & Kwok, NIPS12] [Bucak et al. CVPR11] Ours: Unifies hierarchy and exclusion. Transfer learning & Attributes [Rohrbach et al. CVPR10] [Lampert et al. CVPR09] [Kuettel et al. ECCV12] [Akata et al. CVPR13] Ours: A classification model that allows transferring. Extracting Common Sense Knowledge [Chen et al. ICCV13] [Zitnick & Parikh CVPR13] Ours: Assumes knowledge is given. [Hwang et al. CVPR11] [Kang et al. CVPR06] [Marszalek & Schmid CVPR07] [Zweig & Weinshall CVPR07] [Farhadi et al. CVPR10] [Lim et al. NIPS11] [Yu et al. CVPR13] [Fergus et al. ECCV10] [Zhu et al. ECCV14] [Fouhey & Zitnick CVPR14] Visual Model External Knowledge Slide 37 Conclusions A unified framework for single object classification Generalizes standard classification models Leverages a knowledge graph Efficient exact inference Future work Non-absolute relations Spatial relations between object instances