cvpr 2013 diversity tutorial diverse m-best solutions in markov random fields dhruv batra virginia...

CVPR 2013 Diversity Tutorial

Diverse M-Best Solutions inMarkov Random Fields

Dhruv Batra

Virginia Tech

Joint work with: Students: Payman Yadollahpour (TTIC), Abner Guzman-Rivera (UIUC)

Colleagues: Chris Dyer (CMU), Greg Shakhnarovich (TTIC), Pushmeet Kohli (MSRC), Kevin Gimpel (TTIC)


(C) Dhruv Batra 2


Ambiguity Ambiguity Ambiguity

(C) Dhruv Batra 3

?

?

One instance / Two instances?


Problems with MAP

(C) Dhruv Batra 4

Model-Class is Wrong!-- Approximation Error

Not Enough Training Data!-- Estimation ErrorMAP is NP-Hard

-- Optimization ErrorInherent Ambiguity

-- Bayes ErrorMake Multiple Predictions!

Single Prediction = Uncertainty Mismanagement


Multiple Predictions

(C) Dhruv Batra 5

Porway & Zhu, 2011TU & Zhu, 2002Rich History

Sampling

xxx xx x xxx x xx x



(C) Dhruv Batra 6

Flerova et al., 2011Fromer et al., 2009Yanover et al., 2003

M-Best MAP Ideally:

M-Best Modes ✓Porway & Zhu,

2011TU & Zhu, 2002Rich History

Sampling



(C) Dhruv Batra 7

Flerova et al., 2011Fromer et al., 2009Yanover et al., 2003

M-Best MAP Ideally:

M-Best Modes ✓Porway & Zhu,

2011TU & Zhu, 2002Rich History

SamplingOur work: Diverse M-Best in MRFs [ECCV ‘12]

- Don’t hope for diversity. Explicitly encode it.

- Not guaranteed to be modes.


(C) Dhruv Batra 8

Example Result


(C) Dhruv Batra 9

Example Result

Discriminative Re-ranking of Diverse Segmentation

[Yadollahpour et al., CVPR13, Wednesday Poster]


MAP Integer Program

(C) Dhruv Batra 10

kx1


MAP Integer Program

(C) Dhruv Batra 11

kx1

1

0

0

0


MAP Integer Program

(C) Dhruv Batra 12

kx1

0

1

0

0


MAP Integer Program

(C) Dhruv Batra 13

kx1

0

0

1

0


MAP Integer Program

(C) Dhruv Batra 14

kx1

0

0

0

1


MAP Integer Program

(C) Dhruv Batra 15

kx1

0

0

0

1

k2x1


MAP Integer Program

(C) Dhruv Batra 16

kx1

0

0

0

1

k2x1


MAP Integer Program

(C) Dhruv Batra 17

Graphcuts, BP, Expansion, etc


Diverse 2nd-Best

(C) Dhruv Batra 18

MAPDiversity


Diverse M-Best

(C) Dhruv Batra 19


Diverse 2nd-Best

(C) Dhruv Batra 20

Q1: How do we solve DivMBest?

Q2: What kind of diversity functions are allowed?

Q3: How much diversity?


Diverse 2nd-Best

(C) Dhruv Batra 21

Dualize

Diversity-Augmented Score

Primal


Diverse 2nd-Best• Lagrangian Relaxation

(C) Dhruv Batra 22

Diversity-Augmented Score

Dual

Concave (Non-smooth)

Upper-Bound on Div2Best Score

Subgradient Descent

Div2Best score


Diverse 2nd-Best• Lagrangian Relaxation

(C) Dhruv Batra 23

Dualize

Diversity-Augmented EnergyMany ways to solve:

1. Subgradient Ascent. Optimal. Slow.

2. Binary Search. Optimal for M=2. Faster.

3. Grid-search on lambda. Sub-optimal. Fastest.


Theorem Statement• Theorem [Batra et al ’12]: Lagrangian Dual

corresponds to solving the Relaxed Primal:• Based on result from [Geoffrion ‘74]

(C) Dhruv Batra 24

Dual

Relaxed Primal


Effect of Lagrangian Relaxation

(C) Dhruv Batra 25


Effect of Lagrangian Relaxation

(C) Dhruv Batra 26


Effect of Lagrangian Relaxation• [Mezuman et al. UAI13]

(C) Dhruv Batra 27

Pairwise Potential Strength Pairwise Potential Strength


Diverse 2nd-Best

(C) Dhruv Batra 28





Diversity• [Special Case] 0-1 Diversity M-Best MAP

– [Yanover NIPS03; Fromer NIPS09; Flerova Soft11]

• [Special Case] Max Diversity [Park & Ramanan ICCV11]

• Hamming Diversity

• Cardinality Diversity

• Any Diversity

(C) Dhruv Batra 29


Hamming Diversity

(C) Dhruv Batra 30

0

1

0

0

0 1 0 0

0

1

0

0

1 0 0 0


Hamming Diversity

• Diversity Augmented Inference:

(C) Dhruv Batra 31


Hamming Diversity

• Diversity Augmented Inference:

(C) Dhruv Batra 32

Unchanged. Can still use graph-cuts!

Simply edit node-terms. Reuse MAP machinery!


Diverse 2nd-Best

(C) Dhruv Batra 33





How Much Diversity?

• Empirical Solution: Cross-Val for

• More Efficient: Cross-Val for

(C) Dhruv Batra 34


Experiments• 3 Applications

– Interactive Segmentation: Hamming, Cardinality (in paper)– Pose Estimation: Hamming– Semantic Segmentation: Hamming

• Baselines:– M-Best MAP (No Diversity)– Confidence-Based Perturbation (No Optimization)

(C) Dhruv Batra 35


Interactive Segmentation• Setup

– Model: Color/Texture + Potts Grid CRF– Inference: Graph-cuts– Dataset: 50 train/val/test images

(C) Dhruv Batra 36

Image + Scribbles Diverse 2nd Best2nd Best MAPMAP

1-2 Nodes Flipped 100-500 Nodes Flipped


Pose Tracking• Setup

– Model: Mixture of Parts from [Park & Ramanan, ICCV ‘11]– Inference: Dynamic Programming– Dataset: 4 videos, 585 frames

(C) Dhruv Batra 37Image Credit: [Yang & Ramanan, ICCV ‘11]


(C) Dhruv Batra 38


Pose Tracking• Chain CRF with M states at each time

(C) Dhruv Batra 39

M BestSolutions

Image Credit: [Yang & Ramanan, ICCV ‘11]


Pose Tracking

(C) Dhruv Batra 40

DivMBest + ViterbiMAP


Pose Tracking

(C) Dhruv Batra 41

1 51 101 151 201 251 30145%

50%

55%

60%

65%

70%

75%

80%

85%

DivMBest (Re-ranked)

[Park & Ramanan, ICCV ‘11] (Re-ranked)

Confidence-based Perturbation (Re-ranked)

13% Gain

Same FeaturesSame Model

#Solutions / Frame

PC

P A

ccur

acy

Better


Machine Translation

Input:Die Regierung will die Folter von “Hexen” unterbinden und gab eine Broschüre heraus

MAP Translation:The government wants the torture of ‘witch’ and gave out a booklet

(C) Dhruv Batra 42


Machine Translation


5-Best Translations:The government wants the torture of ‘witch’ and gave out a booklet The government wants the torture of “witch” and gave out a booklet The government wants the torture of ‘witch’ and gave out a brochure The government wants the torture of ‘witch’ and gave out a leaflet The government wants the torture of “witch” and gave out a brochure

(C) Dhruv Batra 43


Machine Translation


Diverse 5-Best Translations:The government wants the torture of ‘witch’ and gave out a bookletThe government wants to stop torture of “witch” and issued a leaflet issuedThe government wants to “stop the torture of” witches and gave out a brochureThe government intends to the torture of “witchcraft” and were issued a leafletThe government is the torture of “witches” stamp out and gave a brochure

(C) Dhruv Batra 44


Machine Translation


Diverse 5-Best Translations:The government wants the torture of ‘witch’ and gave out a bookletThe government wants to stop torture of “witch” and issued a leaflet issuedThe government wants to “stop the torture of” witches and gave out a brochureThe government intends to the torture of “witchcraft” and were issued a leafletThe government is the torture of “witches” stamp out and gave a brochure

Correct Translation:The government wants to limit the torture of “witches,” a brochure was released

(C) Dhruv Batra 45

cvpr 2013 diversity tutorial diverse m-best solutions in markov random fields dhruv batra virginia...

Documents

diversity tutorial diverse

diversity c dhruv batra30

diversity tutorial experiments

diversity tutorial problems

c dhruv batra31 slide

c dhruv batra34 slide

best c dhruv batra19

kind of diversity functions