exploring compositional high order pattern potentials for …yujiali/papers/cvpr13_slides.pdf ·...
TRANSCRIPT
![Page 1: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/1.jpg)
Exploring Compositional High Order Pattern
Potentials for Structured Output Learning
Yujia Li, Daniel Tarlow*, Richard Zemel
University of Toronto*Now at Microsoft Research Cambridge
June 25, 2013
![Page 2: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/2.jpg)
Structured Output Learning
● Lots of real world applications require structured outputs– Image segmentation, pose estimation, sequence labeling, etc.
Figures from Weizmann horse dataset
![Page 3: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/3.jpg)
Structured Output Learning
● Lots of real world applications require structured outputs– Image segmentation, pose estimation, sequence labeling, etc.
● Standard model – pairwise MRF/CRF
– Sparse connections – easier to learn and do inference
– Overly simplistic – only modeling up to 2nd order correlation in outputs
UnaryPotentials
PairwisePotentials
Figures from Weizmann horse dataset
![Page 4: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/4.jpg)
Moving to More Expressive Models
● Densely connected CRFs [P. Krahenbuhl et al. NIPS’12]– Still 2nd order connections but densely connected
● Robust High Order Potentials [P. Kohli et al. CVPR’08]– Smoothness in a region
● Global Connectivity Potentials [S. Nowozin et al. CVPR’09]– Require the output to be connected
● Pattern Potentials [C. Rother et al. CVPR’09]– Consistency between the output and learned patterns
![Page 5: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/5.jpg)
Pattern Potentials
● Penalize linearly if output deviates from a pattern
● Multiple base pattern potentials can be combined to form more expressive composite pattern potentials
Patterns
Weights
Pattern and weight figures: C. Rother et al. CVPR'09
Pairwise CRF
![Page 6: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/6.jpg)
Restricted Boltzmann Machines (RBMs)
● RBM probabilistic model
– Sum out h, RBM becomes a high order potential on y
● Some success modeling object shape– The Shape Boltzmann Machine [S. M. Ali Eslami et al., CVPR'12]
– Masked RBMs [N. Heess et al. ICANN'11]
Visible variables y
Hidden variables h
![Page 7: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/7.jpg)
CHOPP
● Compositional High Order Pattern Potential (CHOPP)
Compatibilitywith a pattern
Combineall patterns
Interpolate betweenRBMs and PPs
![Page 8: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/8.jpg)
CHOPP-Augmented CRF
● Compositional High Order Pattern Potential (CHOPP)
● CHOPP-augmented CRF Energy functionLabels y
Hidden variables h
Input x
Standard CRF
CHOPP
![Page 9: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/9.jpg)
“EM” Inference Algorithm
Hidden variables h
E-step: fix y compute h
Hidden variables h
M-step: fix h find optimal y
Labels yLabels y
Posterior inference
The impact of h factorizes
Just a pairwise CRFUse Graph Cuts
● Making predictions
![Page 10: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/10.jpg)
An Example for the “EM” Inference Algorithm
OriginalImage
UnaryPrediction
GroundTruth
Unary+Pairwise
Initialize EMwith this
![Page 11: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/11.jpg)
An Example for the “EM” Inference Algorithm
OriginalImage
UnaryPrediction
GroundTruth
Unary+Pairwise
Initialize EMwith this
Iteration #1 #2 #3 Convergence
Computeh
Graph Cuts
![Page 12: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/12.jpg)
An Example for the “EM” Inference Algorithm
OriginalImage
UnaryPrediction
GroundTruth
Unary+Pairwise
Initialize EMwith this
Iteration #1 #2 #3 Convergence
y computed by Graph Cuts
Computeh
Graph Cuts
![Page 13: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/13.jpg)
Learning by Minimizing Expected Loss
● Contrastive Divergence does not work well● Expected loss objective
● Estimate gradient using a set of samples from p(y|x)
Image x Sampley ~ p(y|x)
![Page 14: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/14.jpg)
Learning by Minimizing Expected Loss
● Contrastive Divergence does not work well● Expected loss objective
● Estimate gradient using a set of samples from p(y|x)
Image x Sampley ~ p(y|x)
GroundTruth
ComputeLoss
0.35
0.14
![Page 15: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/15.jpg)
Learning by Minimizing Expected Loss
● Contrastive Divergence does not work well● Expected loss objective
● Estimate gradient using a set of samples from p(y|x)
Image x Sampley ~ p(y|x)
GroundTruth
ComputeLoss
0.35
0.14
Probability
Probability
![Page 16: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/16.jpg)
Datasets and Settings
● Weizmann horse dataset● PASCAL VOC 2011: image inside the bounding box
– Class “person” and class “bird”
● All images resized to 32x32● T=1, Intersection Over Union (IOU) performance measure
![Page 17: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/17.jpg)
Experiment I
● Train RBM independently (unsupervised)
● Adding an RBM always helps– But not equally on different datasets
![Page 18: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/18.jpg)
Experiment I Analysis: Dataset Variability
● Dataset variability measure
– Person & Birds are harder than horses
Real Datasets Synthetic Datasets
Clustering Intra-cluster entropy Weighted average
![Page 19: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/19.jpg)
Experiment II and III
● Jointly learning RBM parameters by minimizing expected loss
![Page 20: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/20.jpg)
Experiment II and III
● Jointly learning RBM parameters by minimizing expected loss
● Making the RBM hidden bias conditioned on the image
![Page 21: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/21.jpg)
ExamplesCHOPPU+PGT
Most Improvement Average Improvement Least Improvement
![Page 22: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/22.jpg)
Conclusion and Future Work
● Theoretical contribution– Relationship between RBMs and Pattern Potentials
● Algorithmic contribution– Inference and learning algorithms for CHOPP-augmented CRFs
● Empirical contribution– Dataset variability measure
● Looking forward:– Convolutional and deeper models
– Fully explore the variants of CHOPP
– Challenge: lack of labeled data
![Page 23: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/23.jpg)
Q & A
Exploring Compositional High Order Pattern
Potentials for Structured Output Learning
Yujia Li, Daniel Tarlow*, Richard Zemel
University of Toronto*Now at Microsoft Research Cambridge
June 25, 2013
![Page 24: Exploring Compositional High Order Pattern Potentials for …yujiali/papers/cvpr13_slides.pdf · 2013. 7. 10. · Exploring Compositional High Order Pattern Potentials for Structured](https://reader034.vdocuments.us/reader034/viewer/2022052000/6012f9a6c9119f3e151da7c5/html5/thumbnails/24.jpg)
Learned Patterns