the layout consistent random field for detecting and segmenting occluded objects cvpr, june 2006...
TRANSCRIPT
![Page 1: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/1.jpg)
The Layout Consistent Random Fieldfor detecting and segmenting occluded objects
CVPR, June 2006
John Winn
Jamie
Shotton
![Page 2: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/2.jpg)
LayoutCRF contributions
Detection and segmentation
Handles occlusion and deformation
Multiple objects simultaneously
Multiple classes
![Page 3: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/3.jpg)
Related work
Layout consistency
Layout Consistent Random Field
Results
Roadmap
![Page 4: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/4.jpg)
Related work: constellation models
X
[Crandall et al. ECCV 2006]
[Fergus et al. CVPR 2003]
[Leibe et al. ECCV 2004][Kumar et al. CVPR 2005] …
![Page 5: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/5.jpg)
Related work: constellation models
[Crandall et al. ECCV 2006]
[Fergus et al. CVPR 2003]
[Leibe et al. ECCV 2004][Kumar et al. CVPR 2005] …
X
XX
X
![Page 6: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/6.jpg)
Related work: windowed detectors
[Viola and Jones CVPR 2001] [Shotton et al. ICCV 2005]…
Localised features
Classifier Car
Sliding window
![Page 7: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/7.jpg)
Related work: windowed detectors
Localised features
Classifier Car? Wall?
[Viola and Jones CVPR 2001] [Shotton et al. ICCV 2005]…
Sliding window
![Page 8: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/8.jpg)
Related work: multiclass segmentation
[Tu et al. CVPR 2003][He et al. CVPR 2004]
tree
car
road
building
Doesn’t exploit layout of parts – can’t identify object instances
TextonBoost [Shotton et al. ECCV 2006]
![Page 9: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/9.jpg)
Related work
Layout consistency
Layout Consistent Random Field
Results
Roadmap
![Page 10: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/10.jpg)
Dense part labelling
Automatic per-pixel labelling based on a grid of parts
Part labels (color-coded)
![Page 11: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/11.jpg)
Dense part labelling
Background label
Part labels (color-coded)
![Page 12: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/12.jpg)
Patch-based part detector
[Lepetit et al. CVPR 2005]
Decision forest classifier
Features are differences of pixel intensities
Classifier
![Page 13: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/13.jpg)
Decision trees
Extremely efficient at both training and test time. e.g. takes 2ms to apply to 160x120 image using difference of pixel intensities.
Improved performance with multiple decision trees (random forest).
Performs as well as boosting with shared features, but can process much more data in the same time.
![Page 14: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/14.jpg)
Patch-based part detector
Colors show posterior over part labels – part detectors are noisy!
Part color key
![Page 15: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/15.jpg)
Layout consistency
![Page 16: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/16.jpg)
Layout consistency
![Page 17: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/17.jpg)
Layout consistency
(8,3) (9,3)(7,3)
(8,2) (9,2)(7,2)
(8,4) (9,4)(7,4)
Neighboring pixels
(p,q) ?
![Page 18: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/18.jpg)
Layout consistency
(8,3) (9,3)(7,3)
(8,2) (9,2)(7,2)
(8,4) (9,4)(7,4)
Neighboring pixels
(p,q)
(p+1,q)
(p,q)
(p+1,q+1)
(p+1,q-1)
Allows fordeformation
/rotation
Layoutconsist
ent
![Page 19: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/19.jpg)
Layout consistency
(8,3) (9,3)(7,3)
(8,2) (9,2)(7,2)
(8,4) (9,4)(7,4)
Neighboring pixels
(p,q)
? (p,q+1)(p,q) (p+1,q
+1)(p-
1,q+1)
Layoutconsist
ent
![Page 20: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/20.jpg)
Occlusions
One object instance occludes another
‘Background’ occludes object
Object occludes background (object edge)
Not layout consistent = occlusion (or invalid)
![Page 21: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/21.jpg)
Effect of layout consistency
Input image
With layout consistency
Part detector output
Layout consistent regions
![Page 22: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/22.jpg)
Related work
Layout consistency
Layout Consistent Random Field
Results
Roadmap
![Page 23: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/23.jpg)
Layout Consistent Random Field
Part detectorPart labels h
Image I
![Page 24: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/24.jpg)
Layout Consistent Random Field
Part labels h
Layout consistency
Image I
Part detector
![Page 25: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/25.jpg)
Layout Consistent Random Field
Parameters θ’={ βbg , βoe , βco , βiif , e0 , γ }
(set by hand)
Layout consistency
Part detector
Edge weight
![Page 26: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/26.jpg)
Proposed
labelling
Inference of MAP labellingGraph cuts with customised alpha-expansion move
[Boykov and Jolly, ICCV 2001]
Part labels h
![Page 27: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/27.jpg)
Inference of MAP labelling
[Boykov and Jolly, ICCV 2001]
Graph cuts with customised alpha-expansion move
Proposed
labelling
Part labels h
![Page 28: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/28.jpg)
Inference of MAP labelling
[Boykov and Jolly, ICCV 2001]
Expansion move not accepted
Graph cuts with customised alpha-expansion move
Proposed
labelling
Part labels h
![Page 29: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/29.jpg)
Inference of MAP labelling
[Boykov and Jolly, ICCV 2001]
Graph cuts with customised alpha-expansion move
Proposed
labelling
Part labels h
![Page 30: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/30.jpg)
Example inference
![Page 31: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/31.jpg)
Decision tree re-learningPart-labels are inferred (constrained by known mask) and decision forest re-trained
![Page 32: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/32.jpg)
Limitation of layout consistency Allows arbitrary stretching/scaling
![Page 33: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/33.jpg)
Part labels h
Global layout
Global layout
Instance T1
InstanceT2
Global layout constraint is (weak) star-shaped constellation model
Constrains part locationsrelative to centroid
Allows competition between different object instances
Image I
![Page 34: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/34.jpg)
Example with global consistency
Input image
Layout consistent regions
Instance labelling
T1
T2
T3 T1
T2
![Page 35: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/35.jpg)
Related work
Layout consistency
Layout Consistent Random Field
Results
Roadmap
![Page 36: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/36.jpg)
UIUC car database
Segmentation accuracy: 96.5% pixels correct (assessed on 20 randomly selected, hand-labelled images)
![Page 37: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/37.jpg)
UIUC car database
Segmentation accuracy: 96.5% pixels correct (assessed on 20 randomly selected, hand-labelled images)
![Page 38: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/38.jpg)
UIUC car database: detection
Results refer to detection of unoccluded cars only.
![Page 39: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/39.jpg)
Detecting heavily occluded faces Caltech face database with artificial occlusions AR face database with real occlusions
![Page 40: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/40.jpg)
Stability of part labelling
Part color key
![Page 41: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/41.jpg)
Multi-class detection
Can extend to multiple classes with different numbers of part labels for each class
Example: building has multiple parts, other classes have one
![Page 42: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/42.jpg)
Summary + future directionsSummary:LayoutCRF achieves multi-class detection and segmentation of occluded, deformable objects
Future directions: Extend to multiple viewpoints and multiple
scales Share parts between classes Incorporate object context (‘car above road’) Incorporate geometric cues
![Page 44: The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006 John Winn Jamie Shotton](https://reader036.vdocuments.us/reader036/viewer/2022070306/5519de065503467a178b4693/html5/thumbnails/44.jpg)