stacked hierarchical labeling
DESCRIPTION
Stacked Hierarchical Labeling. Dan Munoz Drew BagnellMartial Hebert. The Labeling Problem. Sky. Bldg. Tree. Fgnd. Road. Input. Our Predicted Labels. The Labeling Problem. The Labeling Problem. Needed: better representation & interactions Ohta ‘78. Using Regions. Input. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/1.jpg)
Stacked Hierarchical Labeling
Dan Munoz Drew Bagnell Martial Hebert
![Page 2: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/2.jpg)
2
The Labeling Problem
Input Our Predicted Labels
Road
Tree
Fgnd
BldgSky
![Page 3: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/3.jpg)
3
The Labeling Problem
![Page 4: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/4.jpg)
4
The Labeling Problem
• Needed: better representation & interactions– Ohta ‘78
![Page 5: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/5.jpg)
5
Using Regions
Input Ideal Regions
Slide from T. Malisiewicz
![Page 6: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/6.jpg)
6
Using Regions
Input Actual Regions
Slide from T. Malisiewicz
![Page 7: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/7.jpg)
7
Using Regions + Interactions
Image Representation Ideal Prob. Graphical Model• High-order• Expressive interactions
small regions
big regions
![Page 8: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/8.jpg)
8
Using Regions + Interactions
Actual PGM• Restrictive interactions• Still NP-hard
Image Representationsmall regions
big regions
![Page 9: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/9.jpg)
9
Learning with Approximate Inference• PGM learning requires exact inference– Otherwise, may diverge Kulesza and Pereira ’08
Simple Random Field
Learning Path
![Page 10: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/10.jpg)
10
PGM Approach
Input PGM Inference Output
![Page 11: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/11.jpg)
11
Our Approach
Input f1 OutputfN
Sequence of simple problems
…
Cohen ’05, Daume III ’06
![Page 12: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/12.jpg)
12
A Sequence of Simple Problems
• Training simple modules to net desired output– No searching in exponential space
• Not optimizing any joint distribution/energy– Not necessarily doing it before! Kulesza & Pereira ‘08
Input f1 fN Output…Stacked Hierarchical Labeling
![Page 13: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/13.jpg)
13
Our Contribution• An effective PGM alternative for labeling– Training a hierarchical procedure of simple problems
• Naturally analyzes multiple scales– Robust to imperfect segmentations
• Enables more expressive interactions– Beyond pair-wise smoothing
![Page 14: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/14.jpg)
14
Related Work
small regions
big regions
• Learning with multi-scale configurations– Joint probability distribution
Bouman ‘94, Feng ‘02, He ’04Borenstein ‘04, Kumar ’05
– Joint score/energyTu ‘03, S.C. Zhu ‘06, L. Zhu ‘08Munoz ‘09, Gould ’09, Ladicky ’09
• Mitigating the intractable joint optimization– Cohen ’05, Daume III ’06, Kou ‘07, Tu ’08, Ross ‘10
![Page 15: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/15.jpg)
15
. . . . . .
1
2
3
![Page 16: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/16.jpg)
16
. . . . . .
1
2
3
In this work, the segmentation tree is given
We use the technique from Arbelaez ’09
![Page 17: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/17.jpg)
17
1 2 3 4
Segmentation Tree(Arbelaez ’09)
![Page 18: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/18.jpg)
18
• Parent sees big picture• Naturally handles scales
Label Coarse To Fine1 2 3 4
Segmentation Tree(Arbelaez ’09)
![Page 19: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/19.jpg)
19
• Parent sees big picture• Naturally handles scales
• Break into simple tasks• Predict label mixtures
f1 f2 f3 f4
1 2 3 4
Segmentation Tree(Arbelaez ’09)
![Page 20: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/20.jpg)
20
Handling Real Segmentation• fi predicts mixture of labels for each region
Input Segmentation Map
![Page 21: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/21.jpg)
21
Actual Predicted Mixtures
P(Tree)P(Building)P(Fgnd)(brighter higher probability)
![Page 22: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/22.jpg)
22
Training Overview• How to train each module fi ?• How to use previous predictions?• How to train the hierarchical sequence?
f1
f2
![Page 23: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/23.jpg)
23
Training Overview• How to train each module fi ?• How to use previous predictions?• How to train the hierarchical sequence?
f1
f2
![Page 24: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/24.jpg)
24
Modeling Heterogeneous Regions
• Count true labels Pr present in each region r
• Train a model Q to match each Pr
– Logistic Regression
• minQ H(P,Q) Weighted Logistic Regression– Image features: texture, color, etc. (Gould ’08)
![Page 25: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/25.jpg)
25
Training Overview• How to train each module fi ?
• How to use previous predictions?• How to train the hierarchical sequence?
f1
f2
![Page 26: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/26.jpg)
26
Using Parent Predictions• Use broader context in the finer regions
– Allow finer regions access to all parent predictions• Create & append 3 types of context features– Kumar ’05, Sofman ’06, Shotton ’06, Tu ‘08
Parent regions Child regions
![Page 27: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/27.jpg)
27
Parent Context• Refining the parent
Parent
Child
![Page 28: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/28.jpg)
28
Detailed In Paper• Image-wise (co-occurrence)
• Spatial Neighborhood (center-surround)
Σregions
![Page 29: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/29.jpg)
29
Training Overview• How to train each module fi ?• How to use previous predictions?• How to train the hierarchical sequence?
f1
f2
![Page 30: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/30.jpg)
30
Approach #1• Train each module independently– Use ground truth context features
• Problem: Cascades of Errors– Modules depend on perfect context features– Observe no mistakes during training Propagate mistakes during testing
f1 f2 f3 f4
![Page 31: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/31.jpg)
31
Approach #2• Solution: Train in feed-forward manner– Viola-Jones ‘01, Kumar ‘05, Wainwright ’06, Ross ‘10
f1 f2 f3 f4
![Page 32: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/32.jpg)
32
Training Feed-Forward
fl(Parameters)
LogReg
A
B
C
![Page 33: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/33.jpg)
33
Training Feed-Forward
A
B
Cfl
fl
fl
![Page 34: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/34.jpg)
34
Cascades of Overfitting
• Solution: Stacking– Wolpert ’92, Cohen ’05– Similar to x-validation– Don’t predict on data
used for training
F.F. Train Confusions F.F. Test Confusions
Stacking Test Confusions
![Page 35: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/35.jpg)
35
Stacking
flLogReg
A
B
C
A
![Page 36: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/36.jpg)
36
Stacking
AflA
![Page 37: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/37.jpg)
37
Stacking
flLogReg
A
B
C
B
![Page 38: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/38.jpg)
38
Stacking
A
Bfl
flA
B
![Page 39: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/39.jpg)
39
Stacking
A
B
Cfl
fl
flA
B
C
![Page 40: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/40.jpg)
Learning to Fix Mistakes
Segments
Level 5 Level 6 Level 7
CurrentOutput
Person part of incorrect segmentPerson segmented, but relies on parentPerson fixes previous mistake
![Page 41: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/41.jpg)
Level 1/8 PredictionsSegmentation
![Page 42: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/42.jpg)
15%
Level 1/8 PredictionsP(Foreground)
P(Tree) P(Building) P(Road)
Segmentation
18% 12% 31%
![Page 43: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/43.jpg)
15%
Level 1/8 Predictions
Road
P(Foreground)
P(Tree) P(Building) P(Road)
Current Output Segmentation
18% 12% 31%
![Page 44: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/44.jpg)
P(Foreground)
P(Tree) P(Building) P(Road)
Level 2/8 PredictionsSegmentation
![Page 45: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/45.jpg)
P(Foreground)
P(Tree) P(Road)
Level 2/8 PredictionsCurrent Output Segmentation
P(Building)
![Page 46: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/46.jpg)
Level 3/8 PredictionsP(Foreground)
P(Tree) P(Building) P(Road)
Current Output Segmentation
![Page 47: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/47.jpg)
Level 4/8 PredictionsP(Foreground)
P(Tree) P(Building) P(Road)
Current Output Segmentation
![Page 48: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/48.jpg)
Level 5/8 PredictionsP(Foreground)
P(Tree) P(Building) P(Road)
Current Output Segmentation
![Page 49: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/49.jpg)
Level 6/8 PredictionsP(Foreground)
P(Tree) P(Building) P(Road)
Current Output Segmentation
![Page 50: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/50.jpg)
Level 7/8 PredictionsP(Foreground)
P(Tree) P(Building) P(Road)
Current Output Segmentation
![Page 51: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/51.jpg)
Level 8/8 PredictionsP(Foreground)
P(Tree) P(Building) P(Road)
Current Output Segmentation
![Page 52: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/52.jpg)
Level 1/8 PredictionsP(Foreground)
P(Tree) P(Building) P(Road)
Current Output Segmentation
![Page 53: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/53.jpg)
Level 2/8 PredictionsP(Foreground)
P(Tree) P(Building) P(Road)
Current Output Segmentation
![Page 54: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/54.jpg)
Level 3/8 PredictionsP(Foreground)
P(Tree) P(Building) P(Road)
Current Output Segmentation
![Page 55: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/55.jpg)
Level 4/8 PredictionsP(Foreground)
P(Tree) P(Building) P(Road)
Current Output Segmentation
![Page 56: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/56.jpg)
Level 5/8 PredictionsP(Foreground)
P(Tree) P(Building) P(Road)
Current Output Segmentation
![Page 57: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/57.jpg)
Level 6/8 PredictionsP(Foreground)
P(Tree) P(Building) P(Road)
Current Output Segmentation
![Page 58: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/58.jpg)
Level 7/8 PredictionsP(Foreground)
P(Tree) P(Building) P(Road)
Current Output Segmentation
![Page 59: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/59.jpg)
Level 8/8 PredictionsP(Foreground)
P(Tree) P(Building) P(Road)
Current Output Segmentation
![Page 60: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/60.jpg)
60
Stanford Background Dataset• 8 Classes• 715 Images
• Inference time– Segmentation & image features held constant
Method sec/imageGould ICCV ‘09 30 - 600SHL (Proposed) 10 - 12
Method Avg Class AccuracyGould ICCV ‘09 65.5LogReg (Baseline) 58.0SHL (Proposed) 66.2
![Page 61: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/61.jpg)
61
MSRC-21• 21 Classes• 591 Images
Method Avg Class AccuracyGould IJCV ‘08 64LogReg (Baseline) 60SHL (Proposed) 71
Ladicky ICCV ‘09 75
Lim ICCV’09 67Tu PAMI’09 69Zhu NIPS’08 74
![Page 62: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/62.jpg)
62
MSRC-21• 21 Classes• 591 Images
Method Avg Class AccuracyGould IJCV ‘08 64LogReg (Baseline) 60SHL (Proposed) 71
Ladicky ICCV ‘09 75LogReg (Baseline) 69SHL (Proposed) 75
Lim ICCV’09 67Tu PAMI’09 69Zhu NIPS’08 74
![Page 63: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/63.jpg)
63
Ongoing Work
Labeling 3-D Point Cloudswith Xuehan Xiong
Building
CarGround
Veg
Tree Trunk
Pole
![Page 64: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/64.jpg)
64
Conclusion• An effective structured prediction alternative– High performance with no graphical model
• Beyond site-wise representations– Robust to imperfect segmentations & multiple scales
• Prediction is a series of simple problems– Stacked to avoid cascading errors and overfitting
Input f1 fN Output…
![Page 65: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/65.jpg)
65
Thank You• Acknowledgements– QinetiQ North America Robotics Fellowship– ONR MURI: Reasoning in Reduced Information Spaces– Reviewers, S. Ross, A. Grubb, B. Becker, J.-F. Lalonde
• Questions?
![Page 66: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/66.jpg)
66
![Page 67: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/67.jpg)
67
Image-wise
Σregions
![Page 68: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/68.jpg)
68
Spatial neighborhood
![Page 69: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/69.jpg)
69
Interactions• Described in this talk
• Described in the paper
![Page 70: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/70.jpg)
70
SHL vs. M3N
![Page 71: Stacked Hierarchical Labeling](https://reader036.vdocuments.us/reader036/viewer/2022081502/568161e4550346895dd20227/html5/thumbnails/71.jpg)
71
SHL vs. M3N