tutorial overview - mit saliency...
TRANSCRIPT
![Page 1: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/1.jpg)
New directions in saliency research: Developments in architectures, datasets, and evaluation
Tutorial OverviewZoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016
![Page 2: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/2.jpg)
Ali BorjiCenter for Research in
Computer Vision,University of Central Florida
Zoya BylinskiiComputer Science and
Artificial Intelligence Lab,MIT
Tilke JuddProduct Manager,
GooglePreviously: MIT
MIT Saliency Benchmark Team
Tutorial organized by:
![Page 3: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/3.jpg)
Judd et al. “A Benchmark of Computational Models of Saliency to Predict Human Fixations” [MIT Tech Report 2012]
![Page 4: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/4.jpg)
![Page 5: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/5.jpg)
• 10 saliency models
• 3 baslines
• 3 metrics (AUC, SIM, EMD)
• 1 dataset
• 66 saliency models
• 5 baslines
• 8 metrics
• 2 datasets
2011 2016
![Page 6: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/6.jpg)
![Page 7: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/7.jpg)
large boosts in performance
in recent yearsdriven by neural
network architectures
tracking progress with the MIT Saliency Benchmark
• 66 models evaluated on MIT benchmark (as of Sept 30, 2016)
• top 8/10 are neural networks
• neural networks make up 25% of all submissions (17 models since 2014)
![Page 8: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/8.jpg)
What are the common architectures used for neural network models of saliency?
How are these models trained?
![Page 9: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/9.jpg)
PDP - Jetley et al. [CVPR 2016]
DeepGaze - Kümmerer et al. [ICLR 2015 workshop]
ML-Net - Cornia et al. [ICPR 2016]
SALICON - Huang et al. [ICCV 2015]
DeepFix - Kruthiventi et al. [ArXiv 2015]
eDN - Vig et al. [CVPR 2014]
![Page 10: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/10.jpg)
loss functions & evaluation
datasets
architectures
applications
semanticsegmentation
objectdetection
objectproposals
image clustering & retrieval
cognitivesaliency
Tutorial overview“Deep networks for
saliency map prediction” Naila Murray
![Page 11: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/11.jpg)
tracking progress with the MIT Saliency Benchmark
![Page 12: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/12.jpg)
• 66 models evaluated on MIT benchmark
• using 8 evaluation metrics
• on 2 datasets (MIT300, CAT2000)
tracking progress with the MIT Saliency Benchmark
![Page 13: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/13.jpg)
0 0.2 0.4 0.6 0.8 1159
1317212529333741454953576165
AUC-Judd
0 0.2 0.4 0.6 0.8159
1317212529333741454953576165
sAUC
0 0.5 1 1.5 2 2.5159
1317212529333741454953576165
NSS
0 0.2 0.4 0.6 0.8 11591317212529333741454953576165
CC
0 0.2 0.4 0.6 0.8159
1317212529333741454953576165
SIM
gradual improvementover time
but saturating
tracking progress with the MIT Saliency Benchmarkm
odel
s
![Page 14: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/14.jpg)
0 0.2 0.4 0.6 0.8 1159
1317212529333741454953576165
AUC-Judd
0 0.2 0.4 0.6 0.8159
1317212529333741454953576165
sAUC
0 0.5 1 1.5 2 2.5159
1317212529333741454953576165
NSS
0 0.2 0.4 0.6 0.8 11591317212529333741454953576165
CC
0 0.2 0.4 0.6 0.8159
1317212529333741454953576165
SIM
tracking progress with the MIT Saliency Benchmarkm
odel
s
![Page 15: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/15.jpg)
0 0.2 0.4 0.6 0.8 1159
1317212529333741454953576165
AUC-Judd
0 0.2 0.4 0.6 0.8159
1317212529333741454953576165
sAUC
0 0.5 1 1.5 2 2.5159
1317212529333741454953576165
NSS
0 0.2 0.4 0.6 0.8 11591317212529333741454953576165
CC
0 0.2 0.4 0.6 0.8159
1317212529333741454953576165
SIM
large boosts in performance
in recent yearsdriven by neural
network architectures
tracking progress with the MIT Saliency Benchmarkm
odel
s
![Page 16: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/16.jpg)
As model performances continue to approach ground truth, how can we have
more meaningful evaluation?
![Page 17: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/17.jpg)
loss functions & evaluation
datasets
architectures
applications
semanticsegmentation
objectdetection
objectproposals
image clustering & retrieval
cognitivesaliency
Tutorial overview“Evaluating saliency models in a
probabilistic framework”Matthias Kümmerer
![Page 18: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/18.jpg)
New data collection methodologies
How can we collect enough training data for neural network models of saliency?
![Page 19: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/19.jpg)
Krafka et al. [CVPR 2016]
Papoutsaki et al. [IJCAI 2016]
Xu et al. [ArXiv 2015]
New data collection methodologies
iSUN
Webcam-based
![Page 20: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/20.jpg)
Krafka et al. [CVPR 2016]
Papoutsaki et al. [IJCAI 2016]
Xu et al. [ArXiv 2015]
New data collection methodologies
Kim et al. [CHI EA 2015]
Jiang et al. [CVPR 2015]
iSUN SALICON
Click-basedWebcam-based
![Page 21: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/21.jpg)
What kind of new applications are possiblewith state-of-the-art saliency architectures
and big data?
![Page 22: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/22.jpg)
loss functions & evaluation
datasets
architectures
applications
semanticsegmentation
objectdetection
objectproposals
image clustering & retrieval
cognitivesaliency
Tutorial overview“Saliency for image
understanding and manipulation”Ming-Ming Cheng
![Page 23: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/23.jpg)
Is saliency solved?
![Page 24: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/24.jpg)
Is saliency solved?NO!
“Towards cognitive saliency”Zoya Bylinskii
Tutorial overview
![Page 25: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/25.jpg)
Gaze Text Face Action Main Focus
Inpu
tH
uman
Fi
xatio
nsM
odel
Pr
edic
tions
“Towards cognitive saliency”Zoya Bylinskii
Tutorial overview
![Page 26: Tutorial Overview - MIT Saliency Benchmarksaliency.mit.edu/ECCVTutorial/newDirectionsInSaliency.pdf · Tutorial Overview Zoya Bylinskii & Tilke Judd, ECCV Oct. 8, 2016. Ali Borji](https://reader033.vdocuments.us/reader033/viewer/2022042308/5ed4762c9ba44106c1056e35/html5/thumbnails/26.jpg)