elastic edge boxes for object proposal on rgb-d...
TRANSCRIPT
Elastic Edge Boxes for Object
Proposal on RGB-D Images
Jing Liu, Tongwei Ren, Jia Bei
Nanjing University
January 5, 2016
MMM 2016, Paper ID: 86
Multimedia AnalyzinG
and UnderStanding
MAGUS
NANJING UNIVERSITY
Multimedia AnalyzinG
and UnderStanding
MAGUSOutline
• Motivation
• Elastic Edge Boxes Method
• Experiments
• Conclusion
2
NANJING UNIVERSITY
Multimedia AnalyzinG
and UnderStanding
MAGUSObject Proposal
3
Object detection Image segmentation Image retrieval
• Aims to detect bounding box which possibly contains
class-independent objects in an image
hit
miss
• Applications
NANJING UNIVERSITY
Multimedia AnalyzinG
and UnderStanding
MAGUS
• High recall
• High efficiency
• High accuracy
• Low intersection over union (IoU) is not enough
Object Proposal is Challenging
4
IoU = 0.5
IoU = 0.8
NANJING UNIVERSITY
Multimedia AnalyzinG
and UnderStanding
MAGUSCurrent Methods
5
• Generate a pool of boxes and score the boxes
• Efficient but not accurate enough
• Over-segment images and merge the segments
• Accurate but not efficient enough
[Uijlings et. al, IJCV13][Cheng et. al, CVPR14]
How to combine these two strategies to obtain good performancein both efficiency and accuracy?
Window scoring Grouping
NANJING UNIVERSITY
Multimedia AnalyzinG
and UnderStanding
MAGUSOutline
• Motivation
• Elastic Edge Boxes Method
• Experiments
• Conclusion
6
NANJING UNIVERSITY
Multimedia AnalyzinG
and UnderStanding
MAGUSOverview
Initial boxes generation
Elastic range search
Bounding box adjustment
7
Elastic edge box for RGB-D object proposal
Step 1
resultRGB channels and depth channel
Step 2
Step 3
NANJING UNIVERSITY
Multimedia AnalyzinG
and UnderStanding
MAGUSInitial Boxes Generation
8
• Perform sliding window to sample boxes• Calculate score by contours wholly enclosed
in a box• Utilize edge boxes method [Dolla ́r et. al, ECCV 14]
Initial boxes generation
Edge detection result Initial boxes
NANJING UNIVERSITY
Multimedia AnalyzinG
and UnderStanding
MAGUS
Initial boxes generation
Elastic Range Search
9
• Super-pixels straddling the box are elastic
range
• Use Super-pixels wholly included in the box to
represent object (cyan)
• Use super-pixels adjacent to elastic range of
similar sum as object part to represent
background (blue)
Elastic range (yellow super-pixels)
Elastic range search
NANJING UNIVERSITY
Multimedia AnalyzinG
and UnderStanding
MAGUSBounding Box Adjustment
• Compute color distance, spatial distance
and depth distance as similar
measurement
• Only super-pixels more similar to object
than background in both RGB and depth
channels will be assigned to object
10
Bounding box adjustment
Adjusted bounding box (red box)Decision
Initial boxes generation
Elastic range search
NANJING UNIVERSITY
Multimedia AnalyzinG
and UnderStanding
MAGUSOutline
• Motivation
• Elastic Edge Boxes Method
• Experiments
• Conclusion
11
NANJING UNIVERSITY
Multimedia AnalyzinG
and UnderStanding
MAGUSDataset
• Improvement
• More balanced
• 300 images in each group (2, 3, 4,
5, 5+ objects, respectively)
• Higher average object number
• PASCAL VOC 2012: 2.38
• Stereo objectness: 2.98
• NJU1500: 4.22
• NJU1500: 1,500 stereo images for object proposal
• Extend from stereo objectness dataset [Xu et. al, ICME15]
NANJING UNIVERSITY
Multimedia AnalyzinG
and UnderStanding
MAGUSResult
• Suitable for various images under high IoU
• Challenging situations
13
0.843
0.878
0.875
0.817
0.852
0.7860.876
Obscure(sword)
Small(dustbin)
Occluded(Papa Smurf)
0.845
0.903 0.796 0.825
0.83
0.79
0.738
0.876
0.829
0.8540.802
0.825
0.860.889
0.854
0.694
0.795
0.928
0.826
0.846
0.67
0.76
0.832
0.876
0.791
0.857
0.852
0.821
0.846
0.831
0.95
0.845
0.903 0.796 0.825
0.83
0.79
0.794
0.789
0.941
0.8440.708
0.893
0.802
NANJING UNIVERSITY
Multimedia AnalyzinG
and UnderStanding
MAGUS
• Compare with eight state-of-the-art methods
• Including AIDC, BING, EB, OBJ, GOP, MCG, SS and MEB
• Under IoU = 0.5 and IoU = 0.8, respectively
Comparison
14
IoU = 0.5 IoU = 0.8
Comparable to other methods when IoU = 0.5
Better than other methods when IoU = 0.8
Ours5.78s per image
MCG60.12s per image
NANJING UNIVERSITY
Multimedia AnalyzinG
and UnderStanding
MAGUSOutline
• Motivation
• Elastic Edge Boxes Method
• Experiments
• Conclusion
15
NANJING UNIVERSITY
Multimedia AnalyzinG
and UnderStanding
MAGUSConclusion
• Contribution
• First attempt to integrate window scoring and
grouping strategies for RGB-D object proposal
• Provide an RGB-D image dataset NJU1500 for object
proposal
• Future work
• Object proposal for video analysis
• Usage of object proposal in multimedia applications
16