design and perceptual validation of performance measures for salient object segmentation vida...
TRANSCRIPT
![Page 1: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/1.jpg)
Design and Perceptual Validation
of Performance Measures for Salient Object Segmentation
Vida Movahedi, James H. Elder
Centre for Vision Research
York University, Canada
![Page 2: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/2.jpg)
Evaluation of Salient Object Segmentation
Centre for Vision Research, York University2
Source: Berkeley Segmentation Dataset
![Page 3: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/3.jpg)
Evaluation of Salient Object Segmentation
Centre for Vision Research, York University3
How do we measure success?
![Page 4: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/4.jpg)
Existing literature Salient object segmentation
[Liu07, Zhang07, Park07, Zhuang09, Achanta09, Pirnog09, …]
Evaluation of salient object segmentation algorithms [Ge06,?]
Evaluation of segmentation algorithms [Huang95, Zhang96, Martin01, Monteiro06, Goldmann08,
Estrada09]
Centre for Vision Research, York University4
![Page 5: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/5.jpg)
Contributions
Centre for Vision Research, York University5
Analysis of previously suggested measures
Contour Mapping Measure (CM) Order-preserving matching
A new dataset of salient objects (SOD)
Psychophysics experiments Evaluation of above measures
Matching paradigm in Precision and Recall measures
![Page 6: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/6.jpg)
Evaluation measures in literature
Centre for Vision Research, York University6
Region-based error measures Based on false positive/ false negative pixels [Young05], [Ge06], [Goldmann08], ...
Boundary-based error measures Based on distance between boundaries [Huttenlocher93], [Monteiro06], ...
Mixed measures Based on distance of misclassified pixels to the
boundaries [Young05], [Monteiro06], ...
![Page 7: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/7.jpg)
Region-based error measures[Young05], [Ge06], [Goldmann08], ...
Centre for Vision Research, York University7
A and B two boundaries RA the region corresponding to a boundary A and |
RA| the area of this region,
BA
BA
RR
RRBARI
1),(BA
BAB
BA
BAA
RR
RRR
RR
RRR
||||
False Negatives
False Positives
Not sensitive to shape differences
![Page 8: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/8.jpg)
Boundary-based error measures[Huang95],[Huttenlocher93], [Monteiro06], ...
Centre for Vision Research, York University8
A and B two boundaries Distance of one point a on A from B is
Hausdorff distance:
Mean distance:
),(min)( badadBb
B
)(max),(maxmax),( bdadBAHD ABb
BAa
)(mean),(meanmean),( bdadBAMD ABb
BAa
Not sensitive to shape differences
a
![Page 9: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/9.jpg)
Penalizing the over-detected and under-detected regions by their distances to intersection
Mixture error measures [Young05], [Monteiro06], ...
Centre for Vision Research, York University9
fpfn N
kkB
fp
N
jjA
fndiag
qdN
pdND
BAMM11
)(1
)(1
2
1),(
False Negatives
False Positives
Not sensitive to shape difference
![Page 10: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/10.jpg)
Another example
Centre for Vision Research, York University10
Different shapes with low errors
![Page 11: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/11.jpg)
Comparing two boundaries
Centre for Vision Research, York University11
The two boundaries need to follow each other Thus it is not sufficient to map points to the
closest point on the other boundary The ordering of mapped points must be preserved
B ASmall False
Negative Region
Small False Positive Region
![Page 12: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/12.jpg)
The order of mapped points on the two boundaries must be monotonically non-decreasing.
Allowing for different levels of detail: One-to-one Many-to-one One-to-many
Order-preserving Mapping
nmjibaba njmi then and ,If
Centre for Vision Research, York University12
![Page 13: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/13.jpg)
Contour Mapping Measure
Centre for Vision Research, York University13
Given two contours A=a1a2..an and B=b1b2..bm, Find the correct order-preserving mapping
Contour mapping error measure:
Average distance between matched pairs of points
Bimorphism [Tagare02]
Elastic Matching [Geiger95, Basri98, Sebastian03, ..]
![Page 14: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/14.jpg)
A dynamic programming implementation to find the optimum mapping Closed contours point indices are assigned cyclically
Based on string correction techniques [Maes90]
Complexity: if m<n and m, n points on two boundaries
Contour Mapping Measure
Centre for Vision Research, York University14
)log( mnmO
![Page 15: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/15.jpg)
Contour Mapping Example
Centre for Vision Research, York University15
Ground Truth Boundary
Algorithm Boundary
Matched pairs shown as line segments
CM= average length of line segments
connecting matched pairs
![Page 16: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/16.jpg)
Contour Mapping Measure Order- preserving mapping avoids problems
experienced by other measures
Centre for Vision Research, York University16
![Page 17: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/17.jpg)
SOD: Salient Object Dataset
Centre for Vision Research, York University17
A dataset of salient objects Based on Berkeley Segmentation Dataset
(BSD) [Martin01]
300 images 7 subjects
1
1
1
1
1
Source: Berkeley Segmentation Dataset Available in SOD
![Page 18: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/18.jpg)
Psychophysical experiments
Centre for Vision Research, York University18
Which error measure is closer to human judgements of shape similarity?
9 subjects 5 error measures Regional Intersection (RI) Mean distance (MD) Hausdorff distance (HD) Mixed distance (MM) Contour Mapping (CM)
![Page 19: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/19.jpg)
Psychophysical Experiments
Experiment 1 - SOD
Reference & test shapes all from SOD
Experiment 2 - ALG
Reference from SOD, test shapes algorithm-generated
Centre for Vision Research, York University19
Reference: Human
segmentation
Test cases:
Algorithm-
generated
Test cases: Human
segmentations
Reference: Human
segmentation
![Page 20: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/20.jpg)
Agreement with Human Subjects Human subject chooses Left
or Right
An error measure M also chooses Left or Right, based on their error w.r.t. the reference shape
If M chooses the same as the human, it is a case of agreement
Human-Human consistency: defined based on agreement between human subjects
Centre for Vision Research, York University20
Left Right
Reference
![Page 21: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/21.jpg)
Psychophysical Experiments
Centre for Vision Research, York University21
Experiment 1- SOD
Reference & tests shapes all
from SOD
Experiment 2 - ALG
Reference from SOD, test shapes algorithm-
generated
RI: region intersection, MD: mean distance, HD: Hausdorff distance, MM: mixed measure, CM: contour mapping
![Page 22: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/22.jpg)
Precision and Recall measures
Centre for Vision Research, York University22
For algorithm boundary A and ground truth boundary B
Precision: proportion of true positives on A
Recall: proportion of detected points on B
Martin’s PR (M-PR)[Martin04] Minimum cost bipartite matching, cost proportional to
distance
Estrada’s PR (E-PR)[Estrada09] ‘No intervening contours’ and ‘Same side’ constraints
Contour Mapping PR (CM-PR) Order-preserving mapping
||
),matched(
A
BA
||
),matched(
B
AB
![Page 23: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/23.jpg)
Matching paradigm in Precision/Recall
Centre for Vision Research, York University23
Experiment 1- SOD
Reference & test shapes all
from SOD
Experiment 2 - ALG
Reference from SOD, test shapes algorithm-
generated
![Page 24: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/24.jpg)
Summary
Centre for Vision Research, York University24
Analysis of available measures for evaluation of salient object segmentation algorithms
A new measure- contour mapping measure (CM) Code available online: http://elderlab.yorku.ca/ContourMapping
A new dataset of salient objects Dataset available online: http://elderlab.yorku.ca/SOD
Psychophysical Experiment CM has a higher agreement with human subjects
Order-preserving matching paradigm in Precision/Recall analysis Code available online: http://elderlab.yorku.ca/ContourMapping
![Page 25: Design and Perceptual Validation of Performance Measures for Salient Object Segmentation Vida Movahedi, James H. Elder Centre for Vision Research York](https://reader035.vdocuments.us/reader035/viewer/2022062318/551acfa2550346856e8b5ede/html5/thumbnails/25.jpg)
Centre for Vision Research, York University25
Thank You!