fully convolutional networks for semantic segmentationjhoffman/yahoojapan... · 2016. 3. 25. ·...
TRANSCRIPT
![Page 1: Fully Convolutional Networks for Semantic Segmentationjhoffman/yahooJapan... · 2016. 3. 25. · Fully Convolutional Networks for Semantic Segmentation Jonathan Long* Evan Shelhamer*](https://reader036.vdocuments.us/reader036/viewer/2022081407/60520d8bec553574bb4dc577/html5/thumbnails/1.jpg)
UC Berkeley
Fully Convolutional Networksfor Semantic Segmentation
Jonathan Long* Evan Shelhamer* Trevor Darrell1
![Page 2: Fully Convolutional Networks for Semantic Segmentationjhoffman/yahooJapan... · 2016. 3. 25. · Fully Convolutional Networks for Semantic Segmentation Jonathan Long* Evan Shelhamer*](https://reader036.vdocuments.us/reader036/viewer/2022081407/60520d8bec553574bb4dc577/html5/thumbnails/2.jpg)
- what kind of thingis each pixel part of?
- what kind of stuffis each pixel?
Challenges- tension between
recognition and localization- amount of computation
Semantic Segmentation
2
person
horse
car
![Page 3: Fully Convolutional Networks for Semantic Segmentationjhoffman/yahooJapan... · 2016. 3. 25. · Fully Convolutional Networks for Semantic Segmentation Jonathan Long* Evan Shelhamer*](https://reader036.vdocuments.us/reader036/viewer/2022081407/60520d8bec553574bb4dc577/html5/thumbnails/3.jpg)
Segmentation: PASCAL VOC
3
per-son
horse
car
deep learning with Caffe
end-to-end networks lead to50% relative improvement or 30 points absolute and >100x speedup in 1 year!
FCN:pixelwise convnet
state-of-the-art, in Caffe
Leaderboard
![Page 4: Fully Convolutional Networks for Semantic Segmentationjhoffman/yahooJapan... · 2016. 3. 25. · Fully Convolutional Networks for Semantic Segmentation Jonathan Long* Evan Shelhamer*](https://reader036.vdocuments.us/reader036/viewer/2022081407/60520d8bec553574bb4dc577/html5/thumbnails/4.jpg)
4
“tabby cat”
1000-dim vector
~1 millisecond
convnets perform classification
end-to-end learning
![Page 5: Fully Convolutional Networks for Semantic Segmentationjhoffman/yahooJapan... · 2016. 3. 25. · Fully Convolutional Networks for Semantic Segmentation Jonathan Long* Evan Shelhamer*](https://reader036.vdocuments.us/reader036/viewer/2022081407/60520d8bec553574bb4dc577/html5/thumbnails/5.jpg)
5
~100 ms
end-to-end learning
???
convnets perform segmentation?
![Page 6: Fully Convolutional Networks for Semantic Segmentationjhoffman/yahooJapan... · 2016. 3. 25. · Fully Convolutional Networks for Semantic Segmentation Jonathan Long* Evan Shelhamer*](https://reader036.vdocuments.us/reader036/viewer/2022081407/60520d8bec553574bb4dc577/html5/thumbnails/6.jpg)
“tabby cat”
6
a classification network
![Page 7: Fully Convolutional Networks for Semantic Segmentationjhoffman/yahooJapan... · 2016. 3. 25. · Fully Convolutional Networks for Semantic Segmentation Jonathan Long* Evan Shelhamer*](https://reader036.vdocuments.us/reader036/viewer/2022081407/60520d8bec553574bb4dc577/html5/thumbnails/7.jpg)
7
becoming fully convolutional
![Page 8: Fully Convolutional Networks for Semantic Segmentationjhoffman/yahooJapan... · 2016. 3. 25. · Fully Convolutional Networks for Semantic Segmentation Jonathan Long* Evan Shelhamer*](https://reader036.vdocuments.us/reader036/viewer/2022081407/60520d8bec553574bb4dc577/html5/thumbnails/8.jpg)
8
becoming fully convolutional
![Page 9: Fully Convolutional Networks for Semantic Segmentationjhoffman/yahooJapan... · 2016. 3. 25. · Fully Convolutional Networks for Semantic Segmentation Jonathan Long* Evan Shelhamer*](https://reader036.vdocuments.us/reader036/viewer/2022081407/60520d8bec553574bb4dc577/html5/thumbnails/9.jpg)
9
upsampling output
![Page 10: Fully Convolutional Networks for Semantic Segmentationjhoffman/yahooJapan... · 2016. 3. 25. · Fully Convolutional Networks for Semantic Segmentation Jonathan Long* Evan Shelhamer*](https://reader036.vdocuments.us/reader036/viewer/2022081407/60520d8bec553574bb4dc577/html5/thumbnails/10.jpg)
conv, pool,nonlinearity
upsampling
pixelwiseoutput + loss
end-to-end, pixels-to-pixels network
10
![Page 11: Fully Convolutional Networks for Semantic Segmentationjhoffman/yahooJapan... · 2016. 3. 25. · Fully Convolutional Networks for Semantic Segmentation Jonathan Long* Evan Shelhamer*](https://reader036.vdocuments.us/reader036/viewer/2022081407/60520d8bec553574bb4dc577/html5/thumbnails/11.jpg)
resultsFCN SDS* Truth Input
11*Simultaneous Detection and Segmentation Hariharan et al. ECCV14
Relative to prior state-of-the-art SDS:
- 30% relative improvementin accuracy(67.2% on VOC 2012)
- 286× faster
![Page 12: Fully Convolutional Networks for Semantic Segmentationjhoffman/yahooJapan... · 2016. 3. 25. · Fully Convolutional Networks for Semantic Segmentation Jonathan Long* Evan Shelhamer*](https://reader036.vdocuments.us/reader036/viewer/2022081407/60520d8bec553574bb4dc577/html5/thumbnails/12.jpg)
spectrum of deep features
combine where (local, shallow) with what (global, deep)
fuse features into deep jet
(cf. Hariharan et al. CVPR15 “hypercolumn”) 12
![Page 13: Fully Convolutional Networks for Semantic Segmentationjhoffman/yahooJapan... · 2016. 3. 25. · Fully Convolutional Networks for Semantic Segmentation Jonathan Long* Evan Shelhamer*](https://reader036.vdocuments.us/reader036/viewer/2022081407/60520d8bec553574bb4dc577/html5/thumbnails/13.jpg)
stride 32
no skips
stride 16
1 skip
stride 8
2 skips
ground truthinput image
skip layer refinement
13
![Page 14: Fully Convolutional Networks for Semantic Segmentationjhoffman/yahooJapan... · 2016. 3. 25. · Fully Convolutional Networks for Semantic Segmentation Jonathan Long* Evan Shelhamer*](https://reader036.vdocuments.us/reader036/viewer/2022081407/60520d8bec553574bb4dc577/html5/thumbnails/14.jpg)
[ comparison credit: CRF as RNN, Zheng* & Jayasumana* et al. ICCV 2015 ]
14DeepLab: Chen* & Papandreou* et al. ICLR 2015. CRF-RNN: Zheng* & Jayasumana* et al. ICCV 2015
graphical model refinement
![Page 15: Fully Convolutional Networks for Semantic Segmentationjhoffman/yahooJapan... · 2016. 3. 25. · Fully Convolutional Networks for Semantic Segmentation Jonathan Long* Evan Shelhamer*](https://reader036.vdocuments.us/reader036/viewer/2022081407/60520d8bec553574bb4dc577/html5/thumbnails/15.jpg)
nets for many pixelwise tasks
semanticsegmentation
15
monocular depth estimation (Eigen & Fergus 2015)
boundary prediction (Xie & Tu 2015)optical flow Fischer et al. 2015
![Page 16: Fully Convolutional Networks for Semantic Segmentationjhoffman/yahooJapan... · 2016. 3. 25. · Fully Convolutional Networks for Semantic Segmentation Jonathan Long* Evan Shelhamer*](https://reader036.vdocuments.us/reader036/viewer/2022081407/60520d8bec553574bb4dc577/html5/thumbnails/16.jpg)
fcn.berkeleyvision.org
conclusionfully convolutional networks are fast, end-to-end models for pixelwise problems
- code in Caffe master- models for PASCAL VOC, NYUDv2,
SIFT Flow, PASCAL-Context
16
caffe.berkeleyvision.org
github.com/BVLC/caffe
model exampleinference examplesolving example