modeling visual clutter perception using proto-objects chen-ping yu prof. dimitris samaras prof....
TRANSCRIPT
![Page 1: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/1.jpg)
MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS
Chen-Ping Yu
Prof. Dimitris Samaras
Prof. Greg Zelinsky
![Page 2: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/2.jpg)
Introduction• The goal: model human visual clutter perception.
• Visual clutter: A “confused collection”, or a “crowded disorderly state”. Increasing visual clutter.
• Set Size Effect: search performance decreases as set size increases. How to quantify set size?
![Page 3: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/3.jpg)
Introduction• What are proto-objects?
• Regions of locally similar features, they can be objects, object parts, or just pieces that come together to form objects.
• What is the proto-object clutter model?• Segments an image into proto-objects, use the normalized number
of proto-objects as the clutter measure of the image.
![Page 4: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/4.jpg)
Stimuli• 90 800x600 images of real world contexts
• Selected from the SUN09 Database• 6 object-count groups, each contains 15 images• Human labeled objects are provided with SUN09
31 32 33
36 37 39
31~40 objects
15 images
51 52 53
55 57 58
51~60 objects
15 images
3 5 7
7 9 10
1~10 objects
15 images
90 images total
![Page 5: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/5.jpg)
Stimuli
17
48
3
57
![Page 6: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/6.jpg)
Behavioral method• Subjects
• 15 human subjects age from 18 to 32
• Method• Rank order the 90 images from least to most cluttered• Using a Matlab GUI• Participants were told to use their own definition of clutter
• Practice• 12 practice images prior to actual testing
![Page 7: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/7.jpg)
Behavioral method• 2 displays were used
• Images were shown at random
• Bottom monitor subtended a visual angle of 27° x 20°
• Had the option to correct the ordering
• A experiment lasted roughly 45~60 min
• Average pair-wise rater agreement: ρ = 0.692
![Page 8: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/8.jpg)
Computational method• 1. superpixel preprocessing
• K = 600• Need to merge the resulting over-segmentation
![Page 9: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/9.jpg)
Computational method• 2. Mean-shift clustering
![Page 10: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/10.jpg)
Computational method• 2. Mean-shift clustering, more examples
![Page 11: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/11.jpg)
Computational method• 2. Mean-shift clustering in HSV color space
• Median color of each superpixel
![Page 12: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/12.jpg)
Computational method• 3. Merge neighboring superpixels that belong to the same
color cluster
600 superpixels 207 proto-objects (0.345)
![Page 13: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/13.jpg)
Computational method• Proto-object visualization
• Fill each proto-object using the median of the member-pixel colors
![Page 14: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/14.jpg)
Results• Spearman’s rank order correlation
• Ρ = 0.814, p < 0.001
![Page 15: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/15.jpg)
Results• Robust to different parameter/color space settings
• Each correlation is computed using the optimal MS bandwidth
![Page 16: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/16.jpg)
Results• Comparing to other clutter models
![Page 17: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/17.jpg)
Results• More visualized proto-object results
![Page 18: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/18.jpg)
Some further experiments• Does visual clutter perception change when viewing
images of different sizes?
• Experiment: Large images vs small images• Same 90-image dataset, large images = original 800x600 size;
small images = quarter size (200x150)
• Same behavioral setting• 12 practice images• Same Matlab GUI
![Page 19: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/19.jpg)
Further experiments• Small images subtended a
visual angle of 6.75° x 5 °
• 20 undergraduate students from SBU
• Followed the same procedure as the large-image setup
• Average inter subject correlation: 0.58
![Page 20: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/20.jpg)
Further experiments• Highlights
• Human’s small image rating vs large image rating: ρ = 0.953..!• Proto-object model’s small image correlation: ρ = 0.852
![Page 21: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/21.jpg)
Further experiments• Comparing with other clutter models
• Proto-object model stayed the most consistent
![Page 22: MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS Chen-Ping Yu Prof. Dimitris Samaras Prof. Greg Zelinsky](https://reader031.vdocuments.us/reader031/viewer/2022032313/56649e555503460f94b4c7ab/html5/thumbnails/22.jpg)
Conclusion• Number of object (set size) is a poor predictor to visual
clutter
• Set size may be better quantified/represented by proto-objects• All segmentation-based methods outperformed the feature/non-
segmentation based models
• Can proto-object’s spatial density predict search performance, and/or number estimation?
• Can proto-object’s spatial distribution predict gaze?