recognition using regions (demo) sudheendra v. outline generating multiple segmentations...
TRANSCRIPT
![Page 1: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/1.jpg)
Recognition using Regions (Demo)
Sudheendra V
![Page 2: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/2.jpg)
Outline
• Generating multiple segmentations– Normalized cuts [Ren & Malik (2003)]
• Uniform regions– Watershed transform [Arbel´aez1et al. (2009)]
• Non-uniform• Multiple scales
• Discovering object regions using LDA– ground truth segments– multiple segmentation– single segmentation– hierarchical segmentation
![Page 3: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/3.jpg)
Multiple Segmentations
• Normalized cuts– segmentation as graph partitioning
• nodes -> pixels, edge between neighboring pixels
• edge weight -> affinity between pixels• partition graph into K components
– parameters• number of partitions K
– properties• similar sized partitions (normalized)• preserves region boundaries for large enough
K– multiple segmentations
• vary number of partitions K• resize image to different resolutions
http://www.cs.sfu.ca/~mori/research/superpixels/
affinity matrix
![Page 4: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/4.jpg)
Normalized cuts (examples)
K = 4
K = 6
K = 7
![Page 5: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/5.jpg)
Normalized cuts (examples)
K = 3
K = 5
K = 7
Extra edge •“normalized” regions
![Page 6: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/6.jpg)
Normalized cuts (examples)
K = 3
K = 5
K = 7
![Page 7: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/7.jpg)
Multiple Segmentations• Watershed transform
– contours are detected using texture, edge cues and oriented watershed transform used to determine contour scale
– parameters• thresholding scale for contours, k
– properties• variable sized regions• preserves region boundaries
– multiple segmentations• vary thresholding scale k• resize image to different resolutions
http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/
Contours at multiple scales Threshold at a scale
![Page 8: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/8.jpg)
Watershed (examples)
K = 200
K = 180
K = 160
Threshold at different contour scales K to generate multiple segmentations
![Page 9: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/9.jpg)
Watershed (examples)
K = 195
K = 183
K = 169
Threshold at different contour scales K to generate multiple segmentations
![Page 10: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/10.jpg)
Watershed (Hierarchical segmentation)
Thresholding scales in an increasing sequence produces a hierarchical segmentation
K = 175
K = 155
K = 190
K = 140
![Page 11: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/11.jpg)
Watershed (Hierarchical segmentation)
Thresholding scales in an increasing sequence produces a hierarchical segmentation
K = 175
K = 155
K = 200
K = 145
![Page 12: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/12.jpg)
Ncuts vs Watershed Ncuts
Watershed
Comparison of multiple segmentations generated using Ncuts vs Watershed
![Page 13: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/13.jpg)
Ncuts vs Watershed Ncuts
Watershed
Comparison of multiple segmentations generated using Ncuts vs Watershed
![Page 14: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/14.jpg)
Outline
• Generating multiple segmentations– Normalized cuts [Ren & Malik (2003)]
• Uniform regions– Watershed transform [Arbel´aez1et al. (2009)]
• Non-uniform• Multiple scales
• Discovering object regions using LDA– ground truth segments– multiple segmentation– single segmentation– hierarchical segmentation
![Page 15: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/15.jpg)
Multiple Segmentations
![Page 16: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/16.jpg)
Discovering object regions using LDA
• Approach
• Parameters – number of topics to discover
Generate multiple segmentations
Extract local features (SIFT)
Bag of words rep for each
segment
Use LDA to discover topics
based on word co-occurrence
Rank segments based on
similarity to topic
![Page 17: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/17.jpg)
Dataset
Note that the paper uses a larger set containing ~ 4000 images (MSRC_v0)
MSRC_v2 dataset
• 23 categories
• 591 images
• 1648 objects
Distribution of categories in MSRC_v2
![Page 18: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/18.jpg)
Implementation details
– Dense sift on edge points and 3 different scales
– 2000 visual words– 8 segmentations using different parameters– ~ 40k segments in total– LDA takes ~ 10 mins
![Page 19: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/19.jpg)
Ground Truth Segments
• ground truth segments are directly used• number of topics set to 25 (~ num categories)
Top 20 segments in terms of similarity of word distribution to a topic
![Page 20: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/20.jpg)
Ground Truth SegmentsNumber of topics = 25
![Page 21: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/21.jpg)
Ground Truth SegmentsNumber of topics = 50
![Page 22: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/22.jpg)
Ground Truth SegmentsNumber of topics = 75
![Page 23: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/23.jpg)
• Quantitative results
Overlap score on top 20 segments
Average precision (area under precision-recall curve)
Ground Truth Segments
![Page 24: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/24.jpg)
Multiple segmentations• Normalized cuts
– k = {3, 5, 7, 9}– 2 resolutions
![Page 25: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/25.jpg)
Multiple segmentations
![Page 26: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/26.jpg)
Multiple segmentations
Multiple segmentations vs. Ground truth
![Page 27: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/27.jpg)
• Quantitative results
Overlap score on top 20 segments
Average precision (area under precision-recall curve)
Multiple segmentations
![Page 28: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/28.jpg)
Multiple segmentations• Effect of number of images
Overlap score on top 20 segments
• topics are easier to discover with more object instances
![Page 29: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/29.jpg)
Single segmentation
Multiple segmentations vs. Single segmentation
Single segmentation returns partial objects for some classes
![Page 30: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/30.jpg)
Single segmentation
Multiple segmentations vs. Single segmentation
Single segmentation returns partial objects for some classes
![Page 31: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/31.jpg)
Single segmentation
Multiple segmentations vs. Single segmentation
![Page 32: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/32.jpg)
• Quantitative results
Overlap score on top 20 segments
Average precision (area under precision-recall curve)
Single/Multiple segmentations
![Page 33: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/33.jpg)
Hierarchical segmentationHierarchical segmentation vs. Multiple segmentation (Ncuts)
Watershed threshold set such there are 12 leaf nodes and entire hierarchical tree is used by LDA
![Page 34: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/34.jpg)
Hierarchical segmentation
Hierarchical segmentation vs. Multiple segmentation (Ncuts)
![Page 35: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/35.jpg)
Hierarchical segmentationHierarchical segmentation vs. Multiple segmentation (Ncuts)
![Page 36: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/36.jpg)
• Quantitative results
Overlap score on top 20 segments
Hierarchical segmentation
Contour-based watershed method
• does better for objects with few internal contours (grass, sky)
• is worse for objects with large number of contours (flower, airplane)
![Page 37: Recognition using Regions (Demo) Sudheendra V. Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed](https://reader030.vdocuments.us/reader030/viewer/2022032606/56649e935503460f94b98a42/html5/thumbnails/37.jpg)
Conclusion• Generating multiple segmentations
– ncuts and watershed provide different tradeoffs– bottom-up segmentation needs different parameters
for different objects
• Discovering objects using LDA– number of topics matters quite a bit– topics are easier to discover with more examples– multiple segmentation does better than because
different objects require different parameters – contour-based watershed method does better for
objects with few internal contours