Upload File

end to-end convolutional network for saliency prediction

  • Home
  • Technology
  • End to-end convolutional network for saliency prediction
28
End-to-end convolutional network for saliency prediction Junting Pan Xavier Giró-i-Nieto Slides online @DocXavi Large-scale Scene Understanding (LSUN) Challenge 2015 http://bit.ly/juntingnet

Upload: xavier-giro

Post on 05-Aug-2015

167 views

Category:

Technology


2 download

Report

TRANSCRIPT

Page 1: End to-end convolutional network for saliency prediction

End-to-end convolutional network for saliency prediction

Junting Pan Xavier Giró-i-Nieto

Slides online@DocXavi

Large-scale Scene Understanding (LSUN)

Challenge 2015

http://bit.ly/juntingnet

http://bit.ly/juntingnet
http://bit.ly/juntingnet
Page 2: End to-end convolutional network for saliency prediction

2

Financial supportTechnical support

Albert Gil Josep Pujal

ACKNOWLEDGMENTS

Page 3: End to-end convolutional network for saliency prediction

3

LSUN SALIENCY CHALLENGE: A Déjà vu ?

John Markoff, “Scientists see promise in deep learning Programs”, The New York Times (Nov2012).

Photo: Keith Penner

http://www.nytimes.com/2012/11/24/science/scientists-see-advances-in-deep-learning-a-part-of-artificial-intelligence.html?_r=0
Page 4: End to-end convolutional network for saliency prediction

4

LSUN SALIENCY CHALLENGE: A Déjà vu ?

[Mohedano’14]

https://imatge.upc.edu/web/publications/object-segmentation-images-using-eeg-signals
https://imatge.upc.edu/web/publications/object-segmentation-images-using-eeg-signals
Page 5: End to-end convolutional network for saliency prediction

5

LSUN SALIENCY CHALLENGE: A Déjà vu ?

Page 6: End to-end convolutional network for saliency prediction

6

RELATED WORK: Deep Saliency

Kümmerer, Matthias, Lucas Theis, and Matthias Bethge. "Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet." arXiv preprint arXiv:1411.1045 (2014).

http://arxiv.org/abs/1411.1045
Page 7: End to-end convolutional network for saliency prediction

7

RELATED WORK: Deep Saliency

Vig, Eleonora, Michael Dorr, and David Cox. "Large-scale optimization of hierarchical features for saliency prediction in natural images." Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on. IEEE, 2014.

http://dx.doi.org/10.1109/CVPR.2014.358
Page 8: End to-end convolutional network for saliency prediction

8

RELATED WORK: Fully convolutional

Long, Jonathan, Evan Shelhamer, and Trevor Darrell. "Fully convolutional networks for semantic segmentation." Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on. IEEE, 2015.

http://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Long_Fully_Convolutional_Networks_2015_CVPR_paper.pdf
Page 9: End to-end convolutional network for saliency prediction

9

RELATED WORK: Image Classification

CaffeNet

ARCHITECTURE[Khrizevsky’12]

DATA[Deng’09]

FRAMEWORK[Jia’14]

http://dx.doi.org/10.1109/CVPR.2009.5206848
Page 10: End to-end convolutional network for saliency prediction

10

SALIENCY PREDICTION: JuntingNet

JuntingNet

Page 11: End to-end convolutional network for saliency prediction

11

SALIENCY PREDICTION: JuntingNet

JuntingNet

DATAiSun [Xu’15]

SALICON [Jiang’15]

http://vision.princeton.edu/projects/2014/iSUN/
http://arxiv.org/abs/1504.06755
http://vision.princeton.edu/projects/2014/iSUN/
http://salicon.net/
http://www.cv-foundation.org/openaccess/content_cvpr_2015/html/Jiang_SALICON_Saliency_in_2015_CVPR_paper.html
http://salicon.net/
Page 12: End to-end convolutional network for saliency prediction

12

SALIENCY PREDICTION: Data

TRAIN VALIDATION TEST

SALICON [Jiang’15] 10,000 5,000 5,000

iSun [Xu’15] 6,000 926 2,000

CAT2000 [Borji’15] 2,000 - 2,000

MIT300 [Judd’12] 300 . -

LargeScale

http://salicon.net/
http://www.cv-foundation.org/openaccess/content_cvpr_2015/html/Jiang_SALICON_Saliency_in_2015_CVPR_paper.html
http://salicon.net/
http://vision.princeton.edu/projects/2014/iSUN/
http://arxiv.org/abs/1504.06755
http://vision.princeton.edu/projects/2014/iSUN/
http://saliency.mit.edu/results_cat2000.html
http://saliency.mit.edu/results_cat2000.html
http://saliency.mit.edu/results_mit300.html
http://saliency.mit.edu/results_mit300.html
Page 13: End to-end convolutional network for saliency prediction

13

SALIENCY PREDICTION: JuntingNet

JuntingNet

ARCHITECTURE[Pan’15]

DATAiSun [Xu’15]

SALICON [Jiang’15]

http://vision.princeton.edu/projects/2014/iSUN/
http://arxiv.org/abs/1504.06755
http://vision.princeton.edu/projects/2014/iSUN/
http://salicon.net/
http://www.cv-foundation.org/openaccess/content_cvpr_2015/html/Jiang_SALICON_Saliency_in_2015_CVPR_paper.html
http://salicon.net/
Page 14: End to-end convolutional network for saliency prediction

14

SALIENCY PREDICTION: Architecture

Page 15: End to-end convolutional network for saliency prediction

15

SALIENCY PREDICTION: Architecture

End to end + regression = JuntingNet

Page 16: End to-end convolutional network for saliency prediction

16

SALIENCY PREDICTION: Architecture

Resize

96x96

Upsample + filter

4608 = 48x48

2D map

Page 17: End to-end convolutional network for saliency prediction

17

SALIENCY PREDICTION: JuntingNet

JuntingNet

ARCHITECTURE[Pan’15] (soon)

DATAiSun [Xu’15]

SALICON [Jiang’15]

FRAMEWORK[Bergstra’10][Bastien’12]

http://vision.princeton.edu/projects/2014/iSUN/
http://arxiv.org/abs/1504.06755
http://vision.princeton.edu/projects/2014/iSUN/
http://salicon.net/
http://www.cv-foundation.org/openaccess/content_cvpr_2015/html/Jiang_SALICON_Saliency_in_2015_CVPR_paper.html
http://salicon.net/
http://www.iro.umontreal.ca/~lisa/pointeurs/theano_scipy2010.pdf
http://arxiv.org/pdf/1211.5590.pdf
http://deeplearning.net/software/theano/
Page 18: End to-end convolutional network for saliency prediction

18

SALIENCY PREDICTION: Framework

Tutorial by Daniel Nouri (*) on regression for facial points for Kaggle.

(*) Daniel Nouri, “Using convolution networks to detect facil points” (Dec 2014).

on Lasagne

http://deeplearning.net/software/theano/
http://danielnouri.org/notes/2014/12/17/using-convolutional-neural-nets-to-detect-facial-keypoints-tutorial/
https://github.com/Lasagne/Lasagne
Page 19: End to-end convolutional network for saliency prediction

19

SALIENCY PREDICTION: Training

Data augmentation with horizontal mirroring.

Page 20: End to-end convolutional network for saliency prediction

20

SALIENCY PREDICTION: Training

Loss function Mean Square Error (MSE)

Weight initialization Gaussian distribution

Learning rate 0.03 to 0.0001

Mini batch size 128

Training time 7h (SALICON) / 3h (iSUN)

Acceleration Sigmoid + nesterov momentum 0.9

Regularisation Maxout norm

GPU NVidia GTX 980

Page 21: End to-end convolutional network for saliency prediction

21

RESULTS: Qualitative (iSUN)

JuntingNetGround TruthPixels

Page 22: End to-end convolutional network for saliency prediction

22

RESULTS: Qualitative (iSUN)

JuntingNetGround TruthPixels

Page 23: End to-end convolutional network for saliency prediction

23

RESULTS: Quantitative (iSUN)

Page 24: End to-end convolutional network for saliency prediction

24

RESULTS: Qualitative (SALICON)

JuntingNetGround TruthPixels

Page 25: End to-end convolutional network for saliency prediction

25

RESULTS: Qualitative (SALICON)

JuntingNetGround TruthPixels

Page 26: End to-end convolutional network for saliency prediction

26

RESULTS: Quantitative (SALICON)

Page 27: End to-end convolutional network for saliency prediction

27

RESULTS: Publications by end of June

http://bit.ly/juntingnet

http://bit.ly/juntingnet
http://bit.ly/juntingnet
Page 28: End to-end convolutional network for saliency prediction

28

Thank you LSUN ! Thank you Boston !

http://bit.ly/juntingnetSlides online @DocXavi

http://bit.ly/juntingnet
http://bit.ly/juntingnet

DeepVO: Towards End-to-End Visual Odometry with Deep ... · DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks Sen Wang, Ronald Clark, Hongkai

JOURNAL OF LA An End-to-end 3D Convolutional Neural ...crcv.ucf.edu/papers/ST-CNN-arXiv.pdf · for Action Detection and Segmentation in Videos Rui Hou, ... 3D convolutional neural

Part-Level Convolutional Neural Networks for Pedestrian … · 2018-10-02 · 1 Part-Level Convolutional Neural Networks for Pedestrian Detection Using Saliency and Boundary Box Alignment

Co-Saliency Detection via Mask-Guided Fully Convolutional

Learning Uncertain Convolutional Features for Accurate Saliency …openaccess.thecvf.com/content_ICCV_2017/supplemental/... · 2017-10-20 · Learning Uncertain Convolutional Features

Pixel Saliency based Encoding for Fine-Grained Image Classi cation · very subtle and highly local. Recently, Convolutional Neural Networks (CNNs) have almost yielded the best results

Hierarchical Saliency Detection

Saliency Guided End-to-End Learning for Weakly Supervised … · The prediction results are used to implicitly guide the localization procedure. The en-tire network is trained end-to-end

Convolutional Neural Networks and Language Embeddings for … · 2018. 4. 24. · Convolutional Neural Networks and Language Embeddings for End-to-End Dialect Recognition Suwon Shon

Urban Localization with Street Views Using a Convolutional ......Urban Localization with Street Views using a Convolutional Neural Network for End-to-End Camera Pose Regression Guillaume

Visual Saliency in Image Quality Assessmentorca.cf.ac.uk/100239/1/2017zhangwphd.pdf · saliency and quality perception. Issues regarding how to simulate and integrate visual saliency

Towards End-To-End Text Spotting With Convolutional ...openaccess.thecvf.com/content_ICCV_2017/papers/Li_Towards_End-To... · Towards End-to-end Text Spotting with Convolutional Recurrent

Tactile Mesh Saliency

Recent Advances in Visual Trackingvalser.org/2017/ppt/APR/valse-2017-tracking_wy.pdf · 2017-11-07 · Discriminative Saliency Map with Convolutional Neural Network,” in ICML, 2015

Saliency Retargeting: An Approach to Enhance …lowkl/publications/saliency...saliency retargeting algorithm to generate an enhanced image in which the average saliency values of the

A Lightweight Convolutional Neural Network Based on Visual ... › 1742 › cc9437ecd0d56a8336a8c2… · Sensors 2018, 18, 3039 3 of 21 the SIFT method with a saliency attention model

Spatiotemporal Saliency: Towards a Hierarchical ...Spatiotemporal Saliency: Towards a Hierarchical Representation of Visual Saliency Neil D.B. Bruce and John K. Tsotsos Department

Co-Saliency Detection via Mask-Guided Fully Convolutional …openaccess.thecvf.com/content_CVPR_2019/papers/Zhang_Co... · 2019-06-10 · Co-saliency Detection via Mask-guided Fully

End-to-End Text Recognition with Convolutional Neural Networks

Saliency Weighted Convolutional Features for Instance Searchdoras.dcu.ie/22456/1/PID5469663.pdf · Saliency Weighted Convolutional Features for Instance Search Eva Mohedano , Kevin

Convolutional Neural Networks 4 Saliency Maps...Using Taylor expansions, we can show that: where Using backpropagation, we can highlight pixels of the input image based on the amount

End-To-End Saliency Mapping via Probability …...End-to-End Saliency Mapping via Probability Distribution Prediction Saumya Jetley University of Oxford Oxford, United Kingdom [email protected]

Beyond Universal Saliency: Personalized Saliency ... · Beyond Universal Saliency: Personalized Saliency Prediction with Multi-task CNNy Yanyu Xu1, Nianyi Li2;3, Junru Wu1, Jingyi

JOURNAL OF LA An End-to-end 3D Convolutional Neural ... · JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 1 An End-to-end 3D Convolutional Neural Network for Action Detection

An End-To-End System for Unconstrained Face …...An End-to-End System for Unconstrained Face Verification with Deep Convolutional Neural Networks Jun-Cheng Chen 1 , Rajeev Ranjan

Introduction of Saliency Map

End-to-End Polyphonic Sound Event Detection Using …tuomasv/papers/end_to_end_sed_with_crnn_ijcnn_2018… · End-to-End Polyphonic Sound Event Detection Using Convolutional Recurrent

Saliency-based color accessibility

Computational Cognitive Neuroscience Approach for Saliency ... · Computational Cognitive Neuroscience Approach for Saliency Map detection using Graph-Based Visual Saliency (GBVS)

Saliency map presentation

End-to-End Unsupervised Deformable Image Registration with ...End-to-End Unsupervised Deformable Image Registration with a Convolutional Neural Network Bob D. de Vos1, Floris F. Berendsen

Object saliency

DeepVO: Towards End-to-End Visual Odometry with Deep … · 2018-02-23 · DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks The goal is

Saliency-based Sequential Image Attention with Multiset ...papers.nips.cc/paper/7102-saliency-based... · Saliency-based Sequential Image Attention with Multiset Prediction Sean Welleck

Adaptive Graph Convolutional Network With Attention Graph ...€¦ · Adaptive Graph Convolutional Network with Attention Graph Clustering for Co-saliency Detection Kaihua Zhang1,