![Page 1: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/1.jpg)
CNNs for dense image labeling
semantic segmentation instance segmentation
image classification object detection
![Page 2: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/2.jpg)
Outline• Early “hacks”
• Hypercolumns• Zoom-out features• Fully convolutional networks
• Deep network operations for dense prediction• Transposed convolutions• Unpooling• Dilated convolutions
• Instance segmentation• Mask R-CNN
• Other dense prediction problems
![Page 3: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/3.jpg)
Early “hacks”• Do dense prediction as a post-process on top of an image
classification CNN
Have: feature maps from image classification network
Want: pixel-wise predictions
![Page 4: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/4.jpg)
Hypercolumns• Idea: to obtain a feature representation for an
individual pixel, upsample all feature maps to original image resolution and concatenate values from feature maps “above” that pixel
B. Hariharan, P. Arbelaez, R. Girshick, and J. Malik, Hypercolumns for Object Segmentation and Fine-grained Localization, CVPR 2015
Representation as an end-to-end network:
![Page 5: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/5.jpg)
Zoom-out features
M. Mostajabi, P. Yadollahpour and G. Shakhnarovich, Feedforward semantic segmentation with zoom-out features,
CVPR 2015
![Page 6: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/6.jpg)
Zoom-out features: Example results
![Page 7: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/7.jpg)
Zoom-out features: Evaluation• Metric: mean IoU
• Intersection over union of predicted and ground truth pixels for each class, averaged over classes
![Page 8: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/8.jpg)
Fully convolutional networks• Design a network with only convolutional layers, make
predictions for all pixels at once
J. Long, E. Shelhamer, and T. Darrell, Fully Convolutional Networks for Semantic Segmentation, CVPR 2015
![Page 9: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/9.jpg)
Dense prediction: Outline• Early “hacks”
• Hypercolumns• Zoom-out features• Fully convolutional networks
• Deep network operations for dense prediction• Transposed convolutions• Unpooling• Dilated convolutions
• Instance segmentation• Mask R-CNN
• Other dense prediction problems
![Page 10: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/10.jpg)
Fully convolutional networks
Source: Stanford CS231n
• Design a network with only convolutional layers, make predictions for all pixels at once
• Can the network operate at full image resolution?
![Page 11: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/11.jpg)
Fully convolutional networks
Source: Stanford CS231n
• Design a network with only convolutional layers, make predictions for all pixels at once
• Can the network operate at full image resolution? • Practical solution: first downsample, then upsample
![Page 12: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/12.jpg)
Fully convolutional networks (FCN)
Comparison on a subset of PASCAL 2011 validation data:
J. Long, E. Shelhamer, and T. Darrell, Fully Convolutional Networks for Semantic Segmentation, CVPR 2015
![Page 13: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/13.jpg)
Outline• Early “hacks”
• Hypercolumns• Zoom-out features• Fully convolutional networks
• Deep network operations for dense prediction• Transposed convolutions• Unpooling• Dilated convolutions
![Page 14: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/14.jpg)
Upsampling in a deep network• Regular convolution (stride 1, pad 0)
!"" !"# !"$ !"%!#" !## !#$ !#%!$" !$# !$$ !$%!%" !%# !%$ !%%
&"" &"# &"$&#" &## &#$&$" &$# &$$
'"" '"#'#" '##
&"" &"# &"$ 0 &#" &## &#$ 0 &$" &$# &$$ 0 0 0 0 00 &"" &"# &"$ 0 &#" &## &#$ 0 &$" &$# &$$ 0 0 0 00 0 0 0 &"" &"# &"$ 0 &#" &## &#$ 0 &$" &$# &$$ 00 0 0 0 0 &"" &"# &"$ 0 &#" &## &#$ 0 &$" &$# &$$
!""!"#!"$!"%⋮!%%
'""'"#'#"'##
• Matrix-vector form:
∗ =
=
4x4 input, 2x2 output
![Page 15: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/15.jpg)
Upsampling in a deep network• Transposed convolution
!""!"#!"$!"%!#"!##!#$!#%!$"!$#!$$!$%!%"!%#!%$!%%
&""&"#&#"&##
'"" 0 0 0'"# '"" 0 0'"$ '"# 0 00 '"$ 0 0'#" 0 '"" 0'## '#" '"# '""'#$ '## '"$ '"#0 '#$ 0 '"$'$" 0 '#" 0'$# '$" '## '#"'$$ '$# '#$ '##0 '$$ 0 '#$0 0 '$" 00 0 '$# '$"0 0 '$$ '$#0 0 0 '$$
!"" !"# !"$ !"%
!#" !## !#$ !#%
!$" !$# !$$ !$%
!%" !%# !%$ !%%
'"" '"# '"$
'#" '## '#$
'$" '$# '$$
&"" &"#
&#" &##
=2x2 input, 4x4 output
=∗*
Not an inverse of the original convolution operation, simply reverses dimension change!
Source
![Page 16: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/16.jpg)
Upsampling in a deep network• Transposed convolution
!""!"#!"$!"%!#"!##!#$!#%!$"!$#!$$!$%!%"!%#!%$!%%
&""&"#&#"&##
'"" 0 0 0'"# '"" 0 0'"$ '"# 0 00 '"$ 0 0'#" 0 '"" 0'## '#" '"# '""'#$ '## '"$ '"#0 '#$ 0 '"$'$" 0 '#" 0'$# '$" '## '#"'$$ '$# '#$ '##0 '$$ 0 '#$0 0 '$" 00 0 '$# '$"0 0 '$$ '$#0 0 0 '$$
!"" !"# !"$ !"%
!#" !## !#$ !#%
!$" !$# !$$ !$%
!%" !%# !%$ !%%
'"" '"# '"$
'#" '## '#$
'$" '$# '$$
&"" &"#
&#" &##
!"" = '""&""
=
=∗+
![Page 17: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/17.jpg)
Upsampling in a deep network• Transposed convolution
!""!"#!"$!"%!#"!##!#$!#%!$"!$#!$$!$%!%"!%#!%$!%%
&""&"#&#"&##
'"" 0 0 0'"# '"" 0 0'"$ '"# 0 00 '"$ 0 0'#" 0 '"" 0'## '#" '"# '""'#$ '## '"$ '"#0 '#$ 0 '"$'$" 0 '#" 0'$# '$" '## '#"'$$ '$# '#$ '##0 '$$ 0 '#$0 0 '$" 00 0 '$# '$"0 0 '$$ '$#0 0 0 '$$
!"" !"# !"$ !"%
!#" !## !#$ !#%
!$" !$# !$$ !$%
!%" !%# !%$ !%%
'"" '"# '"$
'#" '## '#$
'$" '$# '$$
&"" &"#
&#" &##
!"# = '"#&"" + '""&"#
=
Convolve input with flipped filter
=∗,
![Page 18: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/18.jpg)
Upsampling in a deep network• Transposed convolution
!""!"#!"$!"%!#"!##!#$!#%!$"!$#!$$!$%!%"!%#!%$!%%
&""&"#&#"&##
'"" 0 0 0'"# '"" 0 0'"$ '"# 0 00 '"$ 0 0'#" 0 '"" 0'## '#" '"# '""'#$ '## '"$ '"#0 '#$ 0 '"$'$" 0 '#" 0'$# '$" '## '#"'$$ '$# '#$ '##0 '$$ 0 '#$0 0 '$" 00 0 '$# '$"0 0 '$$ '$#0 0 0 '$$
!"" !"# !"$ !"%
!#" !## !#$ !#%
!$" !$# !$$ !$%
!%" !%# !%$ !%%
'"" '"# '"$
'#" '## '#$
'$" '$# '$$
&"" &"#
&#" &##
!"$ = '"$&"" + '"#&"#
=
=∗,
Convolve input with flipped filter
![Page 19: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/19.jpg)
Upsampling in a deep network• Transposed convolution
!""!"#!"$!"%!#"!##!#$!#%!$"!$#!$$!$%!%"!%#!%$!%%
&""&"#&#"&##
'"" 0 0 0'"# '"" 0 0'"$ '"# 0 00 '"$ 0 0'#" 0 '"" 0'## '#" '"# '""'#$ '## '"$ '"#0 '#$ 0 '"$'$" 0 '#" 0'$# '$" '## '#"'$$ '$# '#$ '##0 '$$ 0 '#$0 0 '$" 00 0 '$# '$"0 0 '$$ '$#0 0 0 '$$
!"" !"# !"$ !"%
!#" !## !#$ !#%
!$" !$# !$$ !$%
!%" !%# !%$ !%%
'"" '"# '"$
'#" '## '#$
'$" '$# '$$
&"" &"#
&#" &##
!"% = '"$&"#
=
=∗+
Convolve input with flipped filter
![Page 20: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/20.jpg)
Upsampling in a deep network• Transposed convolution
!""!"#!"$!"%!#"!##!#$!#%!$"!$#!$$!$%!%"!%#!%$!%%
&""&"#&#"&##
'"" 0 0 0'"# '"" 0 0'"$ '"# 0 00 '"$ 0 0'#" 0 '"" 0'## '#" '"# '""'#$ '## '"$ '"#0 '#$ 0 '"$'$" 0 '#" 0'$# '$" '## '#"'$$ '$# '#$ '##0 '$$ 0 '#$0 0 '$" 00 0 '$# '$"0 0 '$$ '$#0 0 0 '$$
!"" !"# !"$ !"%
!#" !## !#$ !#%
!$" !$# !$$ !$%
!%" !%# !%$ !%%
'"" '"# '"$
'#" '## '#$
'$" '$# '$$
&"" &"#
&#" &##
!#" = '#"&"" + '""&#"
=
=∗,
Convolve input with flipped filter
![Page 21: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/21.jpg)
Upsampling in a deep network• Transposed convolution
!""!"#!"$!"%!#"!##!#$!#%!$"!$#!$$!$%!%"!%#!%$!%%
&""&"#&#"&##
'"" 0 0 0'"# '"" 0 0'"$ '"# 0 00 '"$ 0 0'#" 0 '"" 0'## '#" '"# '""'#$ '## '"$ '"#0 '#$ 0 '"$'$" 0 '#" 0'$# '$" '## '#"'$$ '$# '#$ '##0 '$$ 0 '#$0 0 '$" 00 0 '$# '$"0 0 '$$ '$#0 0 0 '$$
!"" !"# !"$ !"%
!#" !## !#$ !#%
!$" !$# !$$ !$%
!%" !%# !%$ !%%
'"" '"# '"$
'#" '## '#$
'$" '$# '$$
&"" &"#
&#" &##
=
=
!## = '##&"" + '#"&"# + '"#&#" + '""&##
∗,
Convolve input with flipped filter
![Page 22: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/22.jpg)
Upsampling in a deep network• Transposed convolution
!""!"#!"$!"%!#"!##!#$!#%!$"!$#!$$!$%!%"!%#!%$!%%
&""&"#&#"&##
'"" 0 0 0'"# '"" 0 0'"$ '"# 0 00 '"$ 0 0'#" 0 '"" 0'## '#" '"# '""'#$ '## '"$ '"#0 '#$ 0 '"$'$" 0 '#" 0'$# '$" '## '#"'$$ '$# '#$ '##0 '$$ 0 '#$0 0 '$" 00 0 '$# '$"0 0 '$$ '$#0 0 0 '$$
!"" !"# !"$ !"%
!#" !## !#$ !#%
!$" !$# !$$ !$%
!%" !%# !%$ !%%
'"" '"# '"$
'#" '## '#$
'$" '$# '$$
&"" &"#
&#" &##
=
=
!#$ = '#$&"" + '##&"# + '"$&#" + '"#&##
∗,
Convolve input with flipped filter
![Page 23: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/23.jpg)
Recall: Backward pass for conv layer!"
!#$%= '
(,*
!"!+$,(,%,*
!+$,(,%,*!#$%
= '(,*
!"!+$,(,%,*
-.(,.*
-!"!#
!"!+
= ∗01, 2 1, 21 + 4, 2 + 5
−4, −5
• Transposed convolution is the same operation as backwards pass for regular convolution
![Page 24: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/24.jpg)
Upsampling in a deep network• Transposed convolution
!"" !"# !"$ !"%
!#" !## !#$ !#%
!$" !$# !$$ !$%
!%" !%# !%$ !%%
&"" &"#
&#" &##
!""!"#!"$!"%!#"!##!#$!#%!$"!$#!$$!$%!%"!%#!%$!%%
&""&"#!#"&##
'"" 0 0 0'"# '"" 0 0'"$ '"# 0 00 '"$ 0 0'#" 0 '"" 0'## '#" '"# '""'#$ '## '"$ '"#0 '#$ 0 '"$'$" 0 '#" 0'$# '$" '## '#"'$$ '$# '#$ '##0 '$$ 0 '#$0 0 '$" 00 0 '$# '$"0 0 '$$ '$#0 0 0 '$$
=
'"" '"# '"$
'#" '## '#$
'$" '$# '$$
=∗*
![Page 25: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/25.jpg)
Upsampling in a deep network• Transposed convolution
!"" !"# !"$ !"%
!#" !## !#$ !#%
!$" !$# !$$ !$%
!%" !%# !%$ !%%
&"" &"#
&#" &##
Alternate view:• Place copies of the filter on the
output, weighted by entries of the input
!""!"#!"$!"%!#"!##!#$!#%!$"!$#!$$!$%!%"!%#!%$!%%
&""&"#!#"&##
'"" 0 0 0'"# '"" 0 0'"$ '"# 0 00 '"$ 0 0'#" 0 '"" 0'## '#" '"# '""'#$ '## '"$ '"#0 '#$ 0 '"$'$" 0 '#" 0'$# '$" '## '#"'$$ '$# '#$ '##0 '$$ 0 '#$0 0 '$" 00 0 '$# '$"0 0 '$$ '$#0 0 0 '$$
=
'"" '"# '"$
'#" '## '#$
'$" '$# '$$
=∗*
![Page 26: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/26.jpg)
Upsampling in a deep network• Transposed convolution
!"" !"# !"$ !"%
!#" !## !#$ !#%
!$" !$# !$$ !$%
!%" !%# !%$ !%%
&"" &"#
&#" &##
Alternate view:• Place copies of the filter on the
output, weighted by entries of the input
• Sum where copies of the filter overlap
!""!"#!"$!"%!#"!##!#$!#%!$"!$#!$$!$%!%"!%#!%$!%%
&""&"#!#"&##
'"" 0 0 0'"# '"" 0 0'"$ '"# 0 00 '"$ 0 0'#" 0 '"" 0'## '#" '"# '""'#$ '## '"$ '"#0 '#$ 0 '"$'$" 0 '#" 0'$# '$" '## '#"'$$ '$# '#$ '##0 '$$ 0 '#$0 0 '$" 00 0 '$# '$"0 0 '$$ '$#0 0 0 '$$
=
='"" '"# '"$
'#" '## '#$
'$" '$# '$$∗*
![Page 27: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/27.jpg)
Upsampling in a deep network• Transposed convolution
!"" !"# !"$ !"%
!#" !## !#$ !#%
!$" !$# !$$ !$%
!%" !%# !%$ !%%
&"" &"#
&#" &##
Alternate view:• Place copies of the filter on the
output, weighted by entries of the input
• Sum where copies of the filter overlap
!""!"#!"$!"%!#"!##!#$!#%!$"!$#!$$!$%!%"!%#!%$!%%
&""&"#!#"&##
'"" 0 0 0'"# '"" 0 0'"$ '"# 0 00 '"$ 0 0'#" 0 '"" 0'## '#" '"# '""'#$ '## '"$ '"#0 '#$ 0 '"$'$" 0 '#" 0'$# '$" '## '#"'$$ '$# '#$ '##0 '$$ 0 '#$0 0 '$" 00 0 '$# '$"0 0 '$$ '$#0 0 0 '$$
=
='"" '"# '"$
'#" '## '#$
'$" '$# '$$∗*
![Page 28: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/28.jpg)
Upsampling in a deep network• Transposed convolution
!"" !"# !"$ !"%
!#" !## !#$ !#%
!$" !$# !$$ !$%
!%" !%# !%$ !%%
&"" &"#
&#" &##
Alternate view:• Place copies of the filter on the
output, weighted by entries of the input
• Sum where copies of the filter overlap
!""!"#!"$!"%!#"!##!#$!#%!$"!$#!$$!$%!%"!%#!%$!%%
&""&"#!#"&##
'"" 0 0 0'"# '"" 0 0'"$ '"# 0 00 '"$ 0 0'#" 0 '"" 0'## '#" '"# '""'#$ '## '"$ '"#0 '#$ 0 '"$'$" 0 '#" 0'$# '$" '## '#"'$$ '$# '#$ '##0 '$$ 0 '#$0 0 '$" 00 0 '$# '$"0 0 '$$ '$#0 0 0 '$$
=
='"" '"# '"$
'#" '## '#$
'$" '$# '$$∗*
![Page 29: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/29.jpg)
Upsampling in a deep network• 1D example
Animation: https://distill.pub/2016/deconv-checkerboard/
input
output
!" !# !$
%"!"
!& !' !( !) !*
filter: %" %# %$
![Page 30: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/30.jpg)
Upsampling in a deep network• 1D example
Animation: https://distill.pub/2016/deconv-checkerboard/
input
output
!" !# !$
%#!" + %"!#
!' !( !) !* !+
filter: %" %# %$
![Page 31: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/31.jpg)
Upsampling in a deep network• 1D example
Animation: https://distill.pub/2016/deconv-checkerboard/
input
output
!" !# !$
%$!" + %#!# + %"!$
!' !( !) !* !+
filter: %" %# %$
![Page 32: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/32.jpg)
Upsampling in a deep network• 1D example
Animation: https://distill.pub/2016/deconv-checkerboard/
input
output
!" !# !$
%$!# + %#!$ + %"!'
!' !( !) !* !+
filter: %" %# %$
![Page 33: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/33.jpg)
Upsampling in a deep network• Backwards-strided convolution: to increase
resolution, use output stride > 1
Animation: https://distill.pub/2016/deconv-checkerboard/
input
output
stride 1
!" !# !$
%$!# + %#!$ + %"!'
!' !( !) !* !+
![Page 34: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/34.jpg)
Upsampling in a deep network• Backwards-strided convolution: to increase
resolution, use output stride > 1
Animation: https://distill.pub/2016/deconv-checkerboard/
stride 2input
output
!" !# !$
%"!"
!& !' !( !) !*
![Page 35: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/35.jpg)
Upsampling in a deep network• Backwards-strided convolution: to increase
resolution, use output stride > 1
Animation: https://distill.pub/2016/deconv-checkerboard/
stride 2input
output
!" !# !$
%#!"
!& !' !( !) !*
![Page 36: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/36.jpg)
Upsampling in a deep network• Backwards-strided convolution: to increase
resolution, use output stride > 1
Animation: https://distill.pub/2016/deconv-checkerboard/
stride 2input
output
!" !# !$
%$!" + %"!#
!' !( !) !* !+
![Page 37: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/37.jpg)
Upsampling in a deep network• Backwards-strided convolution: to increase
resolution, use output stride > 1
Animation: https://distill.pub/2016/deconv-checkerboard/
stride 2input
output
!" !# !$
%#!#
!& !' !( !) !*
![Page 38: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/38.jpg)
Upsampling in a deep network• Backwards-strided convolution: to increase
resolution, use output stride > 1
Animation: https://distill.pub/2016/deconv-checkerboard/
stride 2input
output
!" !# !$
%$!# + %"!$
!' !( !) !* !+
![Page 39: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/39.jpg)
Upsampling in a deep network• Backwards-strided convolution: to increase
resolution, use output stride > 1• For stride 2, dilate the input by inserting rows and columns of
zeros between adjacent entries, convolve with flipped filter• Sometimes called convolution with fractional input stride 1/2
V. Dumoulin and F. Visin, A guide to convolution arithmetic for deep learning, arXiv 2018
input
outputQ: What 3x3 filter would correspond to bilinear upsampling?
14
12
14
12 1 1
214
12
14
![Page 40: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/40.jpg)
Upsampling in a deep network• Alternative to transposed convolution:
max unpooling
1 2 6 3
3 5 2 1
1 2 2 1
7 3 4 8
5 6
7 8
Max pooling
Remember pooling indices (which
element was max)
0 0 6 0
0 5 0 0
0 0 0 0
7 0 0 8
Max unpooling
Output is sparse, so need to follow this with a transposed
convolution layer
(sometimes called deconvolution instead of transposed convolution, but this is not accurate)
![Page 41: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/41.jpg)
DeconvNet
H. Noh, S. Hong, and B. Han, Learning Deconvolution Network for Semantic Segmentation, ICCV 2015
![Page 42: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/42.jpg)
DeconvNet
H. Noh, S. Hong, and B. Han, Learning Deconvolution Network for Semantic Segmentation, ICCV 2015
Original image 14x14 deconv 28x28 unpooling 28x28 deconv 54x54 unpooling
54x54 deconv 112x112 unpooling 112x112 deconv 224x224 unpooling 224x224 deconv
![Page 43: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/43.jpg)
DeconvNet results
PASCAL VOC 2012 mIoUHypercolumns 59.2ZoomOut 64.4FCN-8 62.2DeconvNet 69.6Ensemble of DeconvNet and FCN 71.7
![Page 44: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/44.jpg)
Similar architecture: SegNet
V. Badrinarayanan, A. Kendall and R. Cipolla, SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation, PAMI 2017
Drop the FC layers, get better results
![Page 45: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/45.jpg)
• Like FCN, fuse upsampled higher-level feature maps with higher-res, lower-level feature maps
• Unlike FCN, fuse by concatenation, predict at the end
U-Net
O. Ronneberger, P. Fischer, T. Brox U-Net: Convolutional Networks for Biomedical Image Segmentation, MICCAI 2015
![Page 47: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/47.jpg)
Recall: Feature pyramid networks
• Improve predictive power of lower-level feature maps by adding contextual information from higher-level feature maps
• Predict different sizes of bounding boxes from different levels of the pyramid (but share parameters of predictors)
T.-Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie, Feature pyramid networks for object detection, CVPR 2017
![Page 48: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/48.jpg)
Outline• Early “hacks”
• Hypercolumns• Zoom-out features• Fully convolutional networks
• Deep network operations for dense prediction• Transposed convolutions• Unpooling• Dilated convolutions
![Page 49: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/49.jpg)
Dilated convolutions• Idea: instead of reducing spatial resolution of feature maps,
use a large sparse filter• Also known as à trous convolution
Image source
Dilation factor 1 Dilation factor 2 Dilation factor 3
![Page 50: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/50.jpg)
Dilated convolutions• Idea: instead of reducing spatial resolution of feature maps,
use a large sparse filter
V. Dumoulin and F. Visin, A guide to convolution arithmetic for deep learning, arXiv 2018
input
output
Like 2x downsamplingfollowed by 3x3
convolution followed by 2x upsampling
![Page 51: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/51.jpg)
Dilated convolutions
L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs, PAMI 2017
![Page 52: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/52.jpg)
Dilated convolutions• Can be used in FCN to remove downsampling:
change stride of max pooling layer from 2 to 1, dilate subsequent convolutions by factor of 2 (in theory, can be done without re-training any parameters)
L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs, PAMI 2017
![Page 53: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/53.jpg)
Dilated convolutions• Can increase receptive field size exponentially with a linear
growth in the number of parameters
F. Yu and V. Koltun, Multi-scale context aggregation by dilated convolutions, ICLR 2016
Feature map 1 (F1) produced from F0 by 1-dilated convolution
F2 produced from F1 by 2-dilated
convolution
F3 produced from F2 by 4-dilated
convolution
Receptive field: 3x3 Receptive field: 7x7 Receptive field: 15x15
![Page 54: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/54.jpg)
Dilated convolutions• Context module with dilation
• Returns same number of feature maps at the same resolution as the input, so can be plugged in to replace components of existing dense prediction architectures
• Requires identity initialization
F. Yu and V. Koltun, Multi-scale context aggregation by dilated convolutions, ICLR 2016
![Page 55: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/55.jpg)
Dilated convolutions: Evaluation
Results on VOC 2012
*Front end: re-implementation of FCN-8 with last two pooling layers dropped (5% better than original FCN-8)
F. Yu and V. Koltun, Multi-scale context aggregation by dilated convolutions, ICLR 2016
![Page 56: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/56.jpg)
Dilated convolutions: Evaluation
![Page 57: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/57.jpg)
Outline• Early “hacks”
• Hypercolumns• Zoom-out features• Fully convolutional networks
• Deep network operations for dense prediction• Transposed convolutions• Unpooling• Dilated convolutions
• Instance segmentation• Mask R-CNN
![Page 58: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/58.jpg)
Instance segmentation
Source: Kaiming He
![Page 59: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/59.jpg)
Mask R-CNN
• Mask R-CNN = Faster R-CNN + FCN on RoIs
K. He, G. Gkioxari, P. Dollar, and R. Girshick, Mask R-CNN, ICCV 2017 (Best Paper Award)
Mask branch: separately predict segmentation for each possible class
Classification+regressionbranch
![Page 60: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/60.jpg)
RoIAlign vs. RoIPool• RoIPool: nearest neighbor quantization
K. He, G. Gkioxari, P. Dollar, and R. Girshick, Mask R-CNN, ICCV 2017 (Best Paper Award)
![Page 61: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/61.jpg)
RoIAlign vs. RoIPool• RoIPool: nearest neighbor quantization• RoIAlign: bilinear interpolation
K. He, G. Gkioxari, P. Dollar, and R. Girshick, Mask R-CNN, ICCV 2017 (Best Paper Award)
![Page 62: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/62.jpg)
Mask R-CNN• From RoIAlign features, predict class label, bounding box,
and segmentation mask
K. He, G. Gkioxari, P. Dollar, and R. Girshick, Mask R-CNN, ICCV 2017 (Best Paper Award)
Separately predict binary mask for each class with per-pixel sigmoids, use average binary cross-entropy loss
Classification/regression head from an established object detector (e.g., FPN)
![Page 63: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/63.jpg)
Mask R-CNN
K. He, G. Gkioxari, P. Dollar, and R. Girshick, Mask R-CNN, ICCV 2017 (Best Paper Award)
![Page 64: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/64.jpg)
Example results
![Page 65: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/65.jpg)
Example results
![Page 66: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/66.jpg)
Instance segmentation results on COCO
K. He, G. Gkioxari, P. Dollar, and R. Girshick, Mask R-CNN, ICCV 2017 (Best Paper Award)
AP at different IoUthresholds
AP for different size instances
![Page 67: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/67.jpg)
Keypoint prediction• Given ! keypoints, train model to predict ! " ×" one-hot maps
with cross-entropy losses over "$ outputs
![Page 68: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/68.jpg)
Outline• Early “hacks”
• Hypercolumns• Zoom-out features• Fully convolutional networks
• Deep network operations for dense prediction• Transposed convolutions• Unpooling• Dilated convolutions
• Instance segmentation• Mask R-CNN
• Other dense prediction problems
![Page 69: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/69.jpg)
Recently proposed task: Panoptic segmentation
A. Kirillov et al. Panoptic segmentation. CVPR 2019
![Page 70: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/70.jpg)
Panoptic feature pyramid networks
A. Kirillov et al. Panoptic feature pyramid networks. CVPR 2019
![Page 71: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/71.jpg)
Panoptic feature pyramid networks
A. Kirillov et al. Panoptic feature pyramid networks. CVPR 2019
![Page 72: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/72.jpg)
Another recent task: Amodal instance segmentation
K. Li and J. Malik. Amodal instance segmentation. ECCV 2016
![Page 73: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/73.jpg)
Even more dense prediction problems• Depth estimation• Surface normal estimation• Colorization• ….
![Page 74: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/74.jpg)
Depth and normal estimationPredicted depth Ground truth
D. Eigen and R. Fergus, Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture, ICCV 2015
![Page 75: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/75.jpg)
Depth and normal estimationPredicted normals Ground truth
D. Eigen and R. Fergus, Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture, ICCV 2015
![Page 76: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/76.jpg)
Estimation of everything at the same time
I. Kokkinos, UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memory,
ICCV 2017
![Page 77: CNNs for dense image labeling - University of Illinois at ...L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. Yuille, DeepLab: Semantic Image Segmentation with Deep Convolutional](https://reader036.vdocuments.us/reader036/viewer/2022071502/61215a1073191c3394109bd6/html5/thumbnails/77.jpg)
Colorization
R. Zhang, P. Isola, and A. Efros, Colorful Image Colorization, ECCV 2016