cvpr 2013 diversity tutorial diverse m-best solutions in markov random fields dhruv batra virginia...

CVPR 2013 Diversity Tutorial

Diverse M-Best Solutions inMarkov Random Fields

Dhruv Batra

Virginia Tech

Joint work with: Students: Payman Yadollahpour (TTIC), Abner Guzman-Rivera (UIUC)

Colleagues: Chris Dyer (CMU), Greg Shakhnarovich (TTIC), Pushmeet Kohli (MSRC), Kevin Gimpel (TTIC)

CVPR 2013 Diversity Tutorial

(C) Dhruv Batra 2

CVPR 2013 Diversity Tutorial

Ambiguity Ambiguity Ambiguity

(C) Dhruv Batra 3

?

?

One instance / Two instances?

CVPR 2013 Diversity Tutorial

Problems with MAP

(C) Dhruv Batra 4

Model-Class is Wrong!-- Approximation Error

Not Enough Training Data!-- Estimation ErrorMAP is NP-Hard

-- Optimization ErrorInherent Ambiguity

-- Bayes ErrorMake Multiple Predictions!

Single Prediction = Uncertainty Mismanagement

CVPR 2013 Diversity Tutorial

Multiple Predictions

(C) Dhruv Batra 5

Porway & Zhu, 2011TU & Zhu, 2002Rich History

Sampling

xxx xx x xxx x xx x

CVPR 2013 Diversity Tutorial

Multiple Predictions

(C) Dhruv Batra 6

Flerova et al., 2011Fromer et al., 2009Yanover et al., 2003

M-Best MAP Ideally:

M-Best Modes ✓Porway & Zhu,

2011TU & Zhu, 2002Rich History

Sampling

CVPR 2013 Diversity Tutorial

Multiple Predictions

(C) Dhruv Batra 7

Flerova et al., 2011Fromer et al., 2009Yanover et al., 2003

M-Best MAP Ideally:

M-Best Modes ✓Porway & Zhu,

2011TU & Zhu, 2002Rich History

SamplingOur work: Diverse M-Best in MRFs [ECCV ‘12]

- Don’t hope for diversity. Explicitly encode it.

- Not guaranteed to be modes.

CVPR 2013 Diversity Tutorial

(C) Dhruv Batra 8

Example Result

CVPR 2013 Diversity Tutorial

(C) Dhruv Batra 9

Example Result

Discriminative Re-ranking of Diverse Segmentation

[Yadollahpour et al., CVPR13, Wednesday Poster]

CVPR 2013 Diversity Tutorial

MAP Integer Program

(C) Dhruv Batra 10

kx1

CVPR 2013 Diversity Tutorial

MAP Integer Program

(C) Dhruv Batra 11

kx1

1

0

0

0

CVPR 2013 Diversity Tutorial

MAP Integer Program

(C) Dhruv Batra 12

kx1

0

1

0

0

CVPR 2013 Diversity Tutorial

MAP Integer Program

(C) Dhruv Batra 13

kx1

0

0

1

0

CVPR 2013 Diversity Tutorial

MAP Integer Program

(C) Dhruv Batra 14

kx1

0

0

0

1

CVPR 2013 Diversity Tutorial

MAP Integer Program

(C) Dhruv Batra 15

kx1

0

0

0

1

k2x1

CVPR 2013 Diversity Tutorial

MAP Integer Program

(C) Dhruv Batra 16

kx1

0

0

0

1

k2x1

CVPR 2013 Diversity Tutorial

MAP Integer Program

(C) Dhruv Batra 17

Graphcuts, BP, Expansion, etc

CVPR 2013 Diversity Tutorial

Diverse 2nd-Best

(C) Dhruv Batra 18

MAPDiversity

CVPR 2013 Diversity Tutorial

Diverse M-Best

(C) Dhruv Batra 19

CVPR 2013 Diversity Tutorial

Diverse 2nd-Best

(C) Dhruv Batra 20

Q1: How do we solve DivMBest?

Q2: What kind of diversity functions are allowed?

Q3: How much diversity?

CVPR 2013 Diversity Tutorial

Diverse 2nd-Best

(C) Dhruv Batra 21

Dualize

Diversity-Augmented Score

Primal

CVPR 2013 Diversity Tutorial

Diverse 2nd-Best• Lagrangian Relaxation

(C) Dhruv Batra 22

Diversity-Augmented Score

Dual

Concave (Non-smooth)

Upper-Bound on Div2Best Score

Subgradient Descent

Div2Best score

CVPR 2013 Diversity Tutorial

Diverse 2nd-Best• Lagrangian Relaxation

(C) Dhruv Batra 23

Dualize

Diversity-Augmented EnergyMany ways to solve:

1. Subgradient Ascent. Optimal. Slow.

2. Binary Search. Optimal for M=2. Faster.

3. Grid-search on lambda. Sub-optimal. Fastest.

CVPR 2013 Diversity Tutorial

Theorem Statement• Theorem [Batra et al ’12]: Lagrangian Dual

corresponds to solving the Relaxed Primal:• Based on result from [Geoffrion ‘74]

(C) Dhruv Batra 24

Dual

Relaxed Primal

CVPR 2013 Diversity Tutorial

Effect of Lagrangian Relaxation

(C) Dhruv Batra 25

CVPR 2013 Diversity Tutorial

Effect of Lagrangian Relaxation

(C) Dhruv Batra 26

CVPR 2013 Diversity Tutorial

Effect of Lagrangian Relaxation• [Mezuman et al. UAI13]

(C) Dhruv Batra 27

Pairwise Potential Strength Pairwise Potential Strength

CVPR 2013 Diversity Tutorial

Diverse 2nd-Best

(C) Dhruv Batra 28

Q1: How do we solve DivMBest?

Q2: What kind of diversity functions are allowed?

Q3: How much diversity?

CVPR 2013 Diversity Tutorial

Diversity• [Special Case] 0-1 Diversity M-Best MAP

– [Yanover NIPS03; Fromer NIPS09; Flerova Soft11]

• [Special Case] Max Diversity [Park & Ramanan ICCV11]

• Hamming Diversity

• Cardinality Diversity

• Any Diversity

(C) Dhruv Batra 29

CVPR 2013 Diversity Tutorial

Hamming Diversity

(C) Dhruv Batra 30

0

1

0

0

0 1 0 0

0

1

0

0

1 0 0 0

CVPR 2013 Diversity Tutorial

Hamming Diversity

• Diversity Augmented Inference:

(C) Dhruv Batra 31

CVPR 2013 Diversity Tutorial

Hamming Diversity

• Diversity Augmented Inference:

(C) Dhruv Batra 32

Unchanged. Can still use graph-cuts!

Simply edit node-terms. Reuse MAP machinery!

CVPR 2013 Diversity Tutorial

Diverse 2nd-Best

(C) Dhruv Batra 33

Q1: How do we solve DivMBest?

Q2: What kind of diversity functions are allowed?

Q3: How much diversity?

CVPR 2013 Diversity Tutorial

How Much Diversity?

• Empirical Solution: Cross-Val for

• More Efficient: Cross-Val for

(C) Dhruv Batra 34

CVPR 2013 Diversity Tutorial

Experiments• 3 Applications

– Interactive Segmentation: Hamming, Cardinality (in paper)– Pose Estimation: Hamming– Semantic Segmentation: Hamming

• Baselines:– M-Best MAP (No Diversity)– Confidence-Based Perturbation (No Optimization)

(C) Dhruv Batra 35

CVPR 2013 Diversity Tutorial

Interactive Segmentation• Setup

– Model: Color/Texture + Potts Grid CRF– Inference: Graph-cuts– Dataset: 50 train/val/test images

(C) Dhruv Batra 36

Image + Scribbles Diverse 2nd Best2nd Best MAPMAP

1-2 Nodes Flipped 100-500 Nodes Flipped

CVPR 2013 Diversity Tutorial

Pose Tracking• Setup

– Model: Mixture of Parts from [Park & Ramanan, ICCV ‘11]– Inference: Dynamic Programming– Dataset: 4 videos, 585 frames

(C) Dhruv Batra 37Image Credit: [Yang & Ramanan, ICCV ‘11]

CVPR 2013 Diversity Tutorial

(C) Dhruv Batra 38

CVPR 2013 Diversity Tutorial

Pose Tracking• Chain CRF with M states at each time

(C) Dhruv Batra 39

M BestSolutions

Image Credit: [Yang & Ramanan, ICCV ‘11]

CVPR 2013 Diversity Tutorial

Pose Tracking

(C) Dhruv Batra 40

DivMBest + ViterbiMAP

CVPR 2013 Diversity Tutorial

Pose Tracking

(C) Dhruv Batra 41

1 51 101 151 201 251 30145%

50%

55%

60%

65%

70%

75%

80%

85%

DivMBest (Re-ranked)

[Park & Ramanan, ICCV ‘11] (Re-ranked)

Confidence-based Perturbation (Re-ranked)

13% Gain

Same FeaturesSame Model

#Solutions / Frame

PC

P A

ccur

acy

Better

CVPR 2013 Diversity Tutorial

Machine Translation

Input:Die Regierung will die Folter von “Hexen” unterbinden und gab eine Broschüre heraus

MAP Translation:The government wants the torture of ‘witch’ and gave out a booklet

(C) Dhruv Batra 42

CVPR 2013 Diversity Tutorial

Machine Translation

Input:Die Regierung will die Folter von “Hexen” unterbinden und gab eine Broschüre heraus

5-Best Translations:The government wants the torture of ‘witch’ and gave out a booklet The government wants the torture of “witch” and gave out a booklet The government wants the torture of ‘witch’ and gave out a brochure The government wants the torture of ‘witch’ and gave out a leaflet The government wants the torture of “witch” and gave out a brochure

(C) Dhruv Batra 43

CVPR 2013 Diversity Tutorial

Machine Translation

Input:Die Regierung will die Folter von “Hexen” unterbinden und gab eine Broschüre heraus

Diverse 5-Best Translations:The government wants the torture of ‘witch’ and gave out a bookletThe government wants to stop torture of “witch” and issued a leaflet issuedThe government wants to “stop the torture of” witches and gave out a brochureThe government intends to the torture of “witchcraft” and were issued a leafletThe government is the torture of “witches” stamp out and gave a brochure

(C) Dhruv Batra 44

CVPR 2013 Diversity Tutorial

Machine Translation

Input:Die Regierung will die Folter von “Hexen” unterbinden und gab eine Broschüre heraus

Diverse 5-Best Translations:The government wants the torture of ‘witch’ and gave out a bookletThe government wants to stop torture of “witch” and issued a leaflet issuedThe government wants to “stop the torture of” witches and gave out a brochureThe government intends to the torture of “witchcraft” and were issued a leafletThe government is the torture of “witches” stamp out and gave a brochure

Correct Translation:The government wants to limit the torture of “witches,” a brochure was released

(C) Dhruv Batra 45