visual attention reading - mitweb.mit.edu/zoya/www/visual_attention_reading.pdf · visual attention...

Visualattention[withandfor]

deepneuralnets

June16,2016AdobeCTLVisionandLearningReadingGroup

ZoyaBylinskii

Visualattentionfor deepneuralnets[captioning]Paperdiscussion:“Show,AttendandTell:NeuralImageCaptionGenerationwithVisualAttention”K.Xu,J.Ba,R.Kiros,K.Cho,A.Courville,R.Salakhutdinov,R.Zemel,Y.Bengio[ICML2015]

? Abirdflyingoverabodyofwater

VisualAttentionforImageCaptioning



CNNforimagefeatureextraction

RNNwithattentionoverimagefeaturesforcaptiongeneration

1 2


?


Lastconvolutional layerbeforemaxpoolinge.g.VGGnet:14x14x512featuremap->196imagefeaturevectorsofdimension 512

1


?


a1 a2 a196…

1


?


a1 a2 a196…

1Wanttocompute“weights”α1α2α196


a1 a2 a196…

Equationsonthisslidecourtesyof:http://people.ee.duke.edu/~lcarin/Yunchen9.25.2015.pdf

Wanttocompute“weights”α1α2α196


Equationsonthisslidecourtesyof:http://people.ee.duke.edu/~lcarin/Yunchen9.25.2015.pdf

Probability thatlocationiisrightplacetofocusforproducing nextword

Relativeimportanceoflocationi forproducingnextword


• LSTMnetworkgeneratesonewordytateverytimestepconditionedonacontextvector,theprevioushiddenstateandthepreviouslygeneratedwords• Thecontextvectorzt isadynamicrepresentationoftherelevantpartoftheimageinputattimet




RNNwithattentionoverimagefeaturesforcaptiongeneration

1 2


“Show,AttendandTell:NeuralImageCaptionGenerationwithVisualAttention”K.Xu,J.Ba,R.Kiros,K.Cho,A.Courville,R.Salakhutdinov,R.Zemel,Y.Bengio [ICML2015]

Visualattentionfor deepneuralnets[questionanswering]Paperdiscussion:“StackedAttentionNetworksforImageQuestionAnswering”,Z.Yang,X.He,J.Gao,L.Deng,A.Smola [arXiv,Jan2016]

VisualAttentionforQuestionAnswering

? Answer:Dogs

Question:Whataresittinginthebasketonabicycle?


? Answer:Dogs


1 2


CNNorLSTMforquestion representation

3stackedattentionmodel

?


f1 f2 a196…

1



?

LSTMforquestionrepresentation

2



?

CNNforquestionrepresentation

2


LSTMforquestionrepresentation

CNNforquestionrepresentation

2a

2b


Questionsfordiscussion

• Whenmightwepreferoneoftheselanguagerepresentationsoveranother?• (2a)anLSTMthatbuildsuparepresentationovermultipletimesteps

• Dowekeepenoughinformationfromearlyoninthesentence?Shouldwefavorlatterpartsofasentence?

• (2b)aCNNthatstaticallycombinesword/sentencefeaturesatafewscales• TowhatextentdoestheNusedforN-gramsaffecttheresultingrepresentation?


CNN f1 f2 f196…

1

Question:Whataresittinginthebasketonabicycle?2

CNNor

LSTMvQ


v1 v2 v196

Question:Whataresittinginthebasketonabicycle?2

CNNor

LSTMvQ

CNN1 f1 f2 f196…


?


vQvI


• Whenmightwewanttomodulatethequestion/languagerepresentationovertime,andwhenmightweprefertomodulatethevisualfeaturerepresentation?• Correspondstorecursing onvQ orvI

Visualattentionwith deepneuralnetsPaperdiscussions:“SALICON:ReducingtheSemanticGapinSaliencyPredictionbyAdaptingDeepNeuralNetworks”,X.Huang,C.Shen,X.Boix,Q.Zhao[CVPR2015]“DeepFix:AFullyConvolutionalNeuralNetworkforpredictingHumanEyeFixations”,S.Kruthiventi,K.Ayush,R.Babu [arXiv Oct2015]“DeepGazeI:BoostingSaliencyPredictionwithFeatureMapsTrainedonImageNet”,M.Kümmerer,L.Theis,M.Bethge [ICLR2015workshop]“PredictingEyeFixationsusingConvolutionalNeuralNetworks”,N.Liu,J.Han,D.Zhang,S.Wen,T.Liu[CVPR2015]

SaliencyPredictionwithNeuralNets

?


• Bottom-uppop-out• Semanticobjectsofinterest• Salientnon-objectregions(”abstractconcepts”)• Multi-scale,context-sensitive

• Challenge:verysmalldatasets


“SALICON:ReducingtheSemanticGapinSaliencyPredictionbyAdaptingDeepNeuralNetworks”,X.Huang,C.ShenX.Boix,Q.Zhao.[ICCV2015]


Capturemulti-scalefeatures- Imagedown-sampled- SameDNNapplied



AlexNet,VGG,orGoogleNet- RemoveallFClayers- Adddepth-1convolutional layerfor

saliencyprediction (aftercombiningresponses frombothscales)



- fine-tuning pretrained networks- optimizesaliencyevaluation

metricsdirectly“SALICON:ReducingtheSemanticGapinSaliencyPredictionbyAdaptingDeepNeuralNetworks”,X.Huang,C.ShenX.Boix,Q.Zhao.[ICCV2015]


• First5layersinitializedwithVGG-16weights• Note:aschanneldepthdoubles,spatialdimensionsarehalvedwithstride-2poolinglayers

“DeepFix:AFullyConvolutionalNeuralNetworkforpredictingHumanEyeFixations”,S.Kruthiventi,K.Ayush,R.Babu [arXivOct2015]



• Holesofsize2introducedinkernelsof5th layertoincreasereceptivefieldwithoutincreasingmemoryfootprint




• Holesofsize2introducedinkernelsof5th layertoincreasereceptivefieldwithoutincreasingmemoryfootprint• Twoinception-styleconvolutionalmodulestocapturemulti-scalesemanticstructure• Convolutionallayersin7th layerwithholesofsize6operateonlargereceptivefieldsformoreglobalcontext




• Holesofsize2introducedinkernelsof5th layertoincreasereceptivefieldwithoutincreasingmemoryfootprint• Twoinception-styleconvolutionalmodulestocapturemulti-scalesemanticstructure• Convolutionallayersin7th layerwithholesofsize6operateonlargereceptivefieldsformoreglobalcontext• Finallayerup-sampledtodepth-1saliencymap




Introducing location-biasedbehaviorwithoutdrasticallyincreasingnumberofnetworkparameters


• RemovefinalFClayers• Rescaleresponsesofallotherlayerstolargestsize(->3712filterresponsesperimagelocation)• Eachfilterindividuallynormalizedacrossdataset,thenGaussianblurredwithsomesigma• Saliencymapisweightedcombinationbetweenthesepost-processedfiltersandacenterbias• L1regularizationonweightstoencouragesparsity• Softmax producesfinaloutputmap

“DeepGazeI:BoostingSaliencyPredictionwithFeatureMapsTrainedonImageNet”,M.Kümmerer,L.Theis,M.Bethge [ICLR2015workshop]

FixationPredictionwithNeuralNets

“PredictingEyeFixationsusingConvolutional NeuralNetworks”,N.Liu,J.Han,D.Zhang,S.Wen,T.Liu[CVPR2015]


• Candirectlyoptimizingforvisualattention/saliencyleadtobenefitsforothercomputervisionapplicationsor shouldvisualattentionnaturallycomeoutofthespecificapplication?• Canmodelsofvisualattention/saliencyhelpbootstrapindividualtasksorleadtogeneralizationacrosstasks?

visual attention reading - mitweb.mit.edu/zoya/www/visual_attention_reading.pdf · visual attention...

Documents