3d layoutcrf

33
3D LayoutCRF Derek Hoiem Carsten Rother John Winn

Upload: sandra-townsend

Post on 01-Jan-2016

26 views

Category:

Documents


2 download

DESCRIPTION

3D LayoutCRF. Derek Hoiem Carsten Rother John Winn. Goal 1: Object Description. Object Description: Bounding Box Viewpoint Color Pose Subclass. Goal 2: Object Segmentation. Key Idea. Combine object-level and pixel-level reasoning. Recognition Requires Object-Level Reasoning. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: 3D LayoutCRF

3D LayoutCRF

Derek Hoiem

Carsten Rother

John Winn

Page 2: 3D LayoutCRF

2

Goal 1: Object Description

Object Description:

• Bounding Box

• Viewpoint

• Color

• Pose

• Subclass

Page 3: 3D LayoutCRF

3

Goal 2: Object Segmentation

Page 4: 3D LayoutCRF

4

• Combine object-level and pixel-level reasoning

Key Idea

Page 5: 3D LayoutCRF

5

Recognition Requires Object-Level Reasoning

• Position

• Shape/Size

• Viewpoint/Pose

• Style/Color

Page 6: 3D LayoutCRF

6

Recognition Requires Object-Level Reasoning

Page 7: 3D LayoutCRF

7

Solution: Window Detector?

• 45 degree range of viewpoints

• Minor scale/position variation

Page 8: 3D LayoutCRF

8

What if we have a really good model?

Page 9: 3D LayoutCRF

9

Recognition Requires Part-Level Reasoning

• Propose good global model

Page 10: 3D LayoutCRF

10

Recognition Requires Part-Level Reasoning

• Propose good global model

• Occlusions

Page 11: 3D LayoutCRF

11

Context Requires Both Object and Part-Level Info

• Size relationships require object model

Page 12: 3D LayoutCRF

12

Context Requires Both Object and Part-Level Info

• Surface relationships require occlusion info

Visibly sitting on ground

Not visibly sitting on ground

Page 13: 3D LayoutCRF

13

Our Object/Part Model

Ti = {

hj object parts

bounding box, viewpoint, color model, instance cost }

part consistency

occlusions

Tm

h1 h2 h3 h4

h5 h6 h7 h8

h9 h10 h11 hn

x

Extension from [Winn Shotton 2006]

T1…

Page 14: 3D LayoutCRF

14

Modeling Viewpoint

Parameterized by Bounding Box and Corner

Page 15: 3D LayoutCRF

15

Assigning Parts from Model

Training Image

FL

Training Annotation

Assigned Parts3D Parts Model

Page 16: 3D LayoutCRF

16

Part Assignment Consistency

Page 17: 3D LayoutCRF

17

Relabeling

• Allowing slight deformations, relabel training data

Training Image

Original Labels

New Labels

Page 18: 3D LayoutCRF

18

Eight Viewpoint/Scale Ranges

Height Range

• Appearance (but not location) constant within each range

Page 19: 3D LayoutCRF

20

Modeling Part Appearance

• Template patches (normalized xcorr)

• Intensity / Color

Image Edges (DT)

Page 20: 3D LayoutCRF

21

Modeling Part Appearance

• Randomized decision trees– 25 trees, 250 leaf nodes

• Once:– Learn structure on 50,000 object / 50,000 background

pixels

• For each appearance model:– Learn parameters on all pixels (850 LabelMe images)

Page 21: 3D LayoutCRF

22

Inference

Input Image

Page 22: 3D LayoutCRF

23

Inference

Input Image

Proposals

• One per appearance model

• Objects proposed by connected components

Page 23: 3D LayoutCRF

24

Proposal Stage Model

hi object parts

part consistency

occlusions

h1 h2 h3 h4

h5 h6 h7 h8

h9 h10 h11 hn

x

• CRF Inference (TRW-BP)

Page 24: 3D LayoutCRF

25

Inference

Refinement

• One per proposal

• Incorporate viewpoint, size information

Proposals

Input Image

Page 25: 3D LayoutCRF

26

Refinement Stage Model

Ti = {

hi object parts

bounding box, viewpoint }

part consistency

occlusions

T1

h1 h2 h3 h4

h5 h6 h7 h8

h9 h10 h11 hn

x

Page 26: 3D LayoutCRF

27

Inference

Refinement

Proposals

Arbitration

• Includes color model, instance penalty (graph cuts)

Input Image

Page 27: 3D LayoutCRF

28

Preliminary Results on UIUC

• Trained on 20, tested on rest• Quantitatively comparable to best

Page 28: 3D LayoutCRF

29

Preliminary Results on UIUC

Without Instance Cost

With Instance Cost

T1

h1 h2 h3 h4

h5 h6 h7 h8

h9 h10 h11 hn

x

Page 29: 3D LayoutCRF

30

Preliminary Results on PASCAL’06

• 25 images– One proposal (viewpoint within 45 degrees,

scale of 26-38 pixels)

Page 30: 3D LayoutCRF

31

Preliminary Results on PASCAL’06

Page 31: 3D LayoutCRF

32

Preliminary Results on PASCAL’06

Page 32: 3D LayoutCRF

33

Preliminary Results on PASCAL’06

Without Color Model

With Color Model

Page 33: 3D LayoutCRF

34

Conclusion

• Combined object-level and pixel-level reasoning – Object-level: Position/Size, Viewpoint, Color– Pixel-level: Part appearance, Occlusion

reasoning

• Good preliminary results