semantic image segmentation via deep parsing...
TRANSCRIPT
![Page 1: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/1.jpg)
Semantic Image Segmentation via Deep Parsing Network
Ziwei Liu*, Xiaoxiao Li*, Ping Luo, Chen Change Loy, Xiaoou Tang
Multimedia Lab, The Chinese University of Hong Kong
![Page 2: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/2.jpg)
Problem
![Page 3: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/3.jpg)
Problem
Person
Table
TV
Plant
Background
![Page 4: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/4.jpg)
Previous Attempts
SVM SVM + MRF
CNN CNN + MRF ?
![Page 5: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/5.jpg)
State-of-the-arts
Fully Convolutional Network [Long et al. CVPR 2015]
Learned Features ✓ Pairwise Relations ✗ Joint Training - # Iterations -
![Page 6: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/6.jpg)
State-of-the-arts
DeepLab [Chen et al. ICLR 2015]
Learned Features ✓ Pairwise Relations ✓ Joint Training ✗ # Iterations 10
![Page 7: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/7.jpg)
State-of-the-arts
CRF as RNN [Zheng et al. ICCV 2015]
Learned Features ✓ Pairwise Relations ✓ Joint Training ✓ # Iterations 10
![Page 8: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/8.jpg)
State-of-the-arts
Deep Parsing Network (DPN)
Learned Features ✓ Pairwise Relations ✓ Joint Training ✓ # Iterations 1
![Page 9: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/9.jpg)
Contributions
• Extend MRF to incorporate richer relationships • Formulate mean field inference of high-order MRF as CNN • Capable of joint training and one-pass inference
![Page 10: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/10.jpg)
Revisit MRF
𝑝𝑝𝑖𝑖 𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙 =′ 𝑡𝑡𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑡 = 0.8
𝑈𝑈𝑈𝑈𝑙𝑙𝑈𝑈𝑈𝑈 = −� ln 𝑝𝑝𝑖𝑖(𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙)𝑖𝑖
Unary Term 𝑖𝑖
min 𝐸𝐸 = 𝑈𝑈𝑈𝑈𝑙𝑙𝑈𝑈𝑈𝑈 + 𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈
Energy Function
![Page 11: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/11.jpg)
Revisit MRF
𝑈𝑈𝑈𝑈𝑙𝑙𝑈𝑈𝑈𝑈 = −� ln 𝑝𝑝𝑖𝑖(𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙)𝑖𝑖
Unary Term
𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈 = �𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡(𝑖𝑖) ∗ 𝑑𝑑𝑖𝑖𝑜𝑜𝑜𝑜(𝑖𝑖, 𝑗𝑗)𝑖𝑖,𝑗𝑗
Pairwise Term
𝑖𝑖 𝑗𝑗
min 𝐸𝐸 = 𝑈𝑈𝑈𝑈𝑙𝑙𝑈𝑈𝑈𝑈 + 𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈
Energy Function 𝑑𝑑𝑖𝑖𝑜𝑜𝑜𝑜 𝑖𝑖, 𝑗𝑗 = , = 0.8 𝑖𝑖 𝑗𝑗
Appearance Consistency
![Page 12: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/12.jpg)
Revisit MRF
𝑈𝑈𝑈𝑈𝑙𝑙𝑈𝑈𝑈𝑈 = −� ln 𝑝𝑝𝑖𝑖(𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙)𝑖𝑖
Unary Term
𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈 = �𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡(𝑖𝑖) ∗ 𝑑𝑑𝑖𝑖𝑜𝑜𝑜𝑜(𝑖𝑖, 𝑗𝑗)𝑖𝑖,𝑗𝑗
Pairwise Term
𝑖𝑖 𝑗𝑗
min 𝐸𝐸 = 𝑈𝑈𝑈𝑈𝑙𝑙𝑈𝑈𝑈𝑈 + 𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈
Energy Function 𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡 𝑖𝑖; 𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙 =′ 𝑡𝑡𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑡 = 0.1
Label Consistency
![Page 13: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/13.jpg)
Richer Relationships in DPN
𝑈𝑈𝑈𝑈𝑙𝑙𝑈𝑈𝑈𝑈 = −� ln 𝑝𝑝𝑖𝑖(𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙)𝑖𝑖
Unary Term
𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈 = �𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡(𝑖𝑖) ∗ 𝑑𝑑𝑖𝑖𝑜𝑜𝑜𝑜(𝑖𝑖, 𝑗𝑗)𝑖𝑖,𝑗𝑗
Pairwise Term
𝑖𝑖 𝑗𝑗
min 𝐸𝐸 = 𝑈𝑈𝑈𝑈𝑙𝑙𝑈𝑈𝑈𝑈 + 𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈
Energy Function
![Page 14: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/14.jpg)
Richer Relationships in DPN
𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈 = �𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡(𝑖𝑖) ∗ 𝑑𝑑𝑖𝑖𝑜𝑜𝑜𝑜(𝑖𝑖, 𝑗𝑗)𝑖𝑖,𝑗𝑗
Pairwise Term
𝑖𝑖 𝑗𝑗
( , )
Triple Penalty
![Page 15: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/15.jpg)
Richer Relationships in DPN
𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈 = �𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡(𝑖𝑖) ∗ 𝑑𝑑𝑖𝑖𝑜𝑜𝑜𝑜(𝑖𝑖, 𝑗𝑗)𝑖𝑖,𝑗𝑗
Pairwise Term
𝑖𝑖 𝑗𝑗
( , , ) …
𝑧𝑧1
𝑧𝑧𝑛𝑛 𝑧𝑧1
𝑧𝑧𝑛𝑛
Triple Penalty
![Page 16: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/16.jpg)
Richer Relationships in DPN
𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈 = �𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡(𝑖𝑖) ∗�𝑑𝑑𝑖𝑖𝑜𝑜𝑜𝑜(𝑖𝑖, 𝑗𝑗; 𝑧𝑧)𝑧𝑧𝑖𝑖,𝑗𝑗
Pairwise Term
𝑖𝑖 𝑗𝑗
( , , ) …
𝑧𝑧1
Triple Penalty
𝑧𝑧𝑛𝑛 𝑧𝑧1
𝑧𝑧𝑛𝑛
Triple Penalty
![Page 17: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/17.jpg)
Richer Relationships in DPN
𝑖𝑖
0.8
0.6
𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡 = 0.7
𝑡𝑡𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙
𝑙𝑙𝑏𝑏𝑜𝑜
𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈 = �𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡(𝑖𝑖) ∗�𝑑𝑑𝑖𝑖𝑜𝑜𝑜𝑜(𝑖𝑖, 𝑗𝑗; 𝑧𝑧)𝑧𝑧𝑖𝑖,𝑗𝑗
Pairwise Term Mixture of Label Contexts
𝑖𝑖
![Page 18: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/18.jpg)
Richer Relationships in DPN
𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈 = �𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡(𝑖𝑖, 𝑗𝑗) ∗�𝑑𝑑𝑖𝑖𝑜𝑜𝑜𝑜(𝑖𝑖, 𝑗𝑗; 𝑧𝑧)𝑧𝑧𝑖𝑖,𝑗𝑗
Pairwise Term
𝑖𝑖 𝑗𝑗
0.8 𝑡𝑡𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙
0.6 𝑝𝑝𝑙𝑙𝑈𝑈𝑜𝑜𝑜𝑜𝑈𝑈
𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡 , = 0.2
Mixture of Label Contexts
𝑖𝑖 𝑗𝑗
![Page 19: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/19.jpg)
Richer Relationships in DPN
𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈 = �𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡(𝑖𝑖, 𝑗𝑗) ∗�𝑑𝑑𝑖𝑖𝑜𝑜𝑜𝑜(𝑖𝑖, 𝑗𝑗; 𝑧𝑧)𝑧𝑧𝑖𝑖,𝑗𝑗
Pairwise Term
𝑖𝑖 𝑗𝑗
0.8 𝑡𝑡𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙
0.6 𝑝𝑝𝑙𝑙𝑈𝑈𝑜𝑜𝑜𝑜𝑈𝑈
𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡 , = 0.2
Mixture of Label Contexts
𝑖𝑖 𝑗𝑗
![Page 20: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/20.jpg)
Richer Relationships in DPN
𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈 = �𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡(𝑖𝑖, 𝑗𝑗) ∗�𝑑𝑑𝑖𝑖𝑜𝑜𝑜𝑜(𝑖𝑖, 𝑗𝑗; 𝑧𝑧)𝑧𝑧𝑖𝑖,𝑗𝑗
Pairwise Term
𝑗𝑗
0.6
0.8
𝑝𝑝𝑙𝑙𝑈𝑈𝑜𝑜𝑜𝑜𝑈𝑈
𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡 , = 0.8
𝑡𝑡𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙
𝑖𝑖
Spatial Order
Mixture of Label Contexts
𝑗𝑗 𝑖𝑖
![Page 21: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/21.jpg)
Richer Relationships in DPN
𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈 = �𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡(𝑖𝑖, 𝑗𝑗) ∗�𝑑𝑑𝑖𝑖𝑜𝑜𝑜𝑜(𝑖𝑖, 𝑗𝑗; 𝑧𝑧)𝑧𝑧𝑖𝑖,𝑗𝑗
Pairwise Term
𝑖𝑖 𝑗𝑗
0.8 𝑡𝑡𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙
0.6 𝑝𝑝𝑙𝑙𝑈𝑈𝑜𝑜𝑜𝑜𝑈𝑈
𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡 , = 0.2
Mixture of Label Contexts
𝑖𝑖 𝑗𝑗
![Page 22: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/22.jpg)
Richer Relationships in DPN
𝑖𝑖 𝑗𝑗
0.8 𝑡𝑡𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙
0.6 𝑝𝑝𝑙𝑙𝑈𝑈𝑜𝑜𝑜𝑜𝑈𝑈
𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡 , = 0.2
𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈 = ��𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡𝑘𝑘(𝑖𝑖, 𝑗𝑗)𝑘𝑘
∗�𝑑𝑑𝑖𝑖𝑜𝑜𝑜𝑜(𝑖𝑖, 𝑗𝑗; 𝑧𝑧)𝑧𝑧𝑖𝑖,𝑗𝑗
Pairwise Term
𝑘𝑘
Mixture of Label Contexts
Mixture of Label Contexts
𝑖𝑖 𝑗𝑗
![Page 23: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/23.jpg)
Solve High-order MRF as Convolution
𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈 = ��𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡𝑘𝑘(𝑖𝑖, 𝑗𝑗)𝑘𝑘
∗�𝑑𝑑𝑖𝑖𝑜𝑜𝑜𝑜(𝑖𝑖, 𝑗𝑗; 𝑧𝑧)𝑧𝑧𝑖𝑖,𝑗𝑗
Pairwise Term
𝑝𝑝𝑖𝑖 ∝ 𝑙𝑙𝑒𝑒𝑝𝑝 − 𝑈𝑈𝑈𝑈𝑙𝑙𝑈𝑈𝑈𝑈𝑖𝑖 + �𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈𝑖𝑖,𝑗𝑗 ∗ 𝑝𝑝𝑗𝑗𝑗𝑗
Mean Field Solver 𝑖𝑖
𝑗𝑗
![Page 24: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/24.jpg)
Solve High-order MRF as Convolution
Iterative Updating Formula
𝑖𝑖 𝑗𝑗
Convolution Summation
𝑝𝑝𝑖𝑖 ∝ 𝑙𝑙𝑒𝑒𝑝𝑝 − 𝑈𝑈𝑈𝑈𝑙𝑙𝑈𝑈𝑈𝑈𝑖𝑖 + �𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈𝑖𝑖,𝑗𝑗 ∗ 𝑝𝑝𝑗𝑗𝑗𝑗
𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈𝑖𝑖,𝑗𝑗 : Different Types of Local and Global Filters
![Page 25: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/25.jpg)
Deep Parsing Network
Max Pooling
Deconvolution Local Convolution Convolution
Unary Term Pairwise Term
Triple Penalty Label Contexts
Min Pooling
![Page 26: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/26.jpg)
Deep Parsing Network Unary Term
Fine-tuned VGG-16 Network
Max Pooling Deconvolution Convolution
![Page 27: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/27.jpg)
Deep Parsing Network
Original Image Unary Term Ground Truth
![Page 28: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/28.jpg)
Deep Parsing Network
Unary Term Pairwise Term
Triple Penalty Label Contexts
Max Pooling
Deconvolution Local Convolution Convolution
Min Pooling
![Page 29: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/29.jpg)
�𝑑𝑑𝑖𝑖𝑜𝑜𝑜𝑜 𝑗𝑗; 𝑧𝑧 ∗ 𝑝𝑝𝑧𝑧𝑧𝑧
Deep Parsing Network Triple Penalty
𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈 = ��𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡𝑘𝑘 𝑖𝑖, 𝑗𝑗𝑘𝑘
∗�𝑑𝑑𝑖𝑖𝑜𝑜𝑜𝑜 𝑖𝑖, 𝑗𝑗; 𝑧𝑧 ∗ 𝑝𝑝𝑧𝑧𝑧𝑧𝑖𝑖,𝑗𝑗
�𝑑𝑑𝑖𝑖𝑜𝑜𝑜𝑜 𝑖𝑖, 𝑗𝑗; 𝑧𝑧 ∗ 𝑝𝑝𝑧𝑧𝑧𝑧
![Page 30: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/30.jpg)
Deep Parsing Network Triple Penalty
j z
Unary Term
# classes
�𝑑𝑑𝑖𝑖𝑜𝑜𝑜𝑜 𝑗𝑗; 𝑧𝑧 ∗ 𝑝𝑝𝑧𝑧𝑧𝑧
𝑝𝑝𝑧𝑧 𝑑𝑑𝑖𝑖𝑜𝑜𝑜𝑜 𝑗𝑗; 𝑧𝑧
# classes
Local Conv
j
![Page 31: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/31.jpg)
Deep Parsing Network
Original Image Unary Term Ground Truth
Triple Penalty
![Page 32: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/32.jpg)
Deep Parsing Network
Unary Term Pairwise Term
Triple Penalty Label Contexts
Max Pooling
Deconvolution Local Convolution Convolution
Min Pooling
![Page 33: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/33.jpg)
Deep Parsing Network Mixture of Label Contexts
𝑃𝑃𝑙𝑙𝑖𝑖𝑈𝑈 = ��𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡𝑘𝑘 𝑖𝑖, 𝑗𝑗 ∗�𝑑𝑑𝑖𝑖𝑜𝑜𝑡𝑡 𝑖𝑖, 𝑗𝑗; 𝑧𝑧 ∗ 𝑝𝑝𝑧𝑧𝑧𝑧𝑘𝑘𝑖𝑖,𝑗𝑗
��𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡𝑘𝑘 𝑖𝑖, 𝑗𝑗𝑘𝑘𝑖𝑖,𝑗𝑗
![Page 34: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/34.jpg)
Deep Parsing Network Mixture of Label Contexts
Triple Penalty Result
# classes
𝑡𝑡𝑈𝑈𝑖𝑖
i j
𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡𝑘𝑘 𝑖𝑖, 𝑗𝑗
i
�𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡𝑘𝑘 𝑖𝑖, 𝑗𝑗 ∗ 𝑡𝑡𝑈𝑈𝑖𝑖(𝑗𝑗)𝑗𝑗
Min Pooling
��𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡𝑘𝑘 𝑖𝑖, 𝑗𝑗 ∗ 𝑡𝑡𝑈𝑈𝑖𝑖(𝑗𝑗)𝑘𝑘𝑗𝑗
class 1 class 1
![Page 35: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/35.jpg)
Deep Parsing Network Mixture of Label Contexts
Triple Penalty Result
# classes
𝑡𝑡𝑈𝑈𝑖𝑖
i j
𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡𝑘𝑘 𝑖𝑖, 𝑗𝑗
i
�𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡𝑘𝑘 𝑖𝑖, 𝑗𝑗 ∗ 𝑡𝑡𝑈𝑈𝑖𝑖(𝑗𝑗)𝑗𝑗
Min Pooling
��𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡𝑘𝑘 𝑖𝑖, 𝑗𝑗 ∗ 𝑡𝑡𝑈𝑈𝑖𝑖(𝑗𝑗)𝑘𝑘𝑗𝑗
class 2 class 2
![Page 36: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/36.jpg)
Deep Parsing Network Mixture of Label Contexts
Triple Penalty Result
# classes
𝑡𝑡𝑈𝑈𝑖𝑖
i j
𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡𝑘𝑘 𝑖𝑖, 𝑗𝑗
i
�𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡𝑘𝑘 𝑖𝑖, 𝑗𝑗 ∗ 𝑡𝑡𝑈𝑈𝑖𝑖(𝑗𝑗)𝑗𝑗
Min Pooling
��𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡𝑘𝑘 𝑖𝑖, 𝑗𝑗 ∗ 𝑡𝑡𝑈𝑈𝑖𝑖(𝑗𝑗)𝑘𝑘𝑗𝑗
class 3 class 3
![Page 37: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/37.jpg)
# classes
Deep Parsing Network Mixture of Label Contexts
Min Pooling
��𝑐𝑐𝑜𝑜𝑜𝑜𝑡𝑡𝑘𝑘 𝑖𝑖, 𝑗𝑗 ∗ 𝑡𝑡𝑈𝑈𝑖𝑖(𝑗𝑗)𝑘𝑘𝑗𝑗
![Page 38: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/38.jpg)
Deep Parsing Network
Original Image Unary Term Ground Truth
Triple Penalty Label Contexts
![Page 39: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/39.jpg)
Deep Parsing Network Joint Tuning
Unary Term Pairwise Term
Triple Penalty Label Contexts
![Page 40: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/40.jpg)
Deep Parsing Network
Original Image Unary Term Ground Truth
Triple Penalty Label Contexts Joint Tuning
![Page 41: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/41.jpg)
FCN 62.2 DeepLab† 73.9
CRFasRNN† 74.7 BoxSup† 75.2
DPN† 77.5
(PASCAL VOC 2012 Challenge test set)
Overall Performance (Published Results)
![Page 42: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/42.jpg)
Label Contexts Learned
favor
penalty
bkg
areo
bi
ke
bird
bo
at
bott
le
bus
car
cat
chai
r co
w
tabl
e do
g ho
rse
mbi
ke
pers
on
plan
t sh
eep
sofa
tr
ain
tv
bkg areo bike
tv train sofa
bird boat
bottle bus car cat
chair cow
table dog
horse
sheep
mbike person
plant
mbi
ke
bike
person
![Page 43: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/43.jpg)
Label Contexts Learned
person : mbike chair : person
favor
penalty
![Page 44: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/44.jpg)
Original Image Ground Truth FCN
DPN DeepLab CRFasRNN
Challenging Case
![Page 45: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/45.jpg)
Original Image Ground Truth Our Result
Failure Case
car
![Page 46: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/46.jpg)
Conclusions
• General framework of one-pass CNN to model high-order MRF
• Various types of pairwise terms are formulated as local and
global filters • High performance and easy to be speeded up
![Page 47: Semantic Image Segmentation via Deep Parsing …personal.ie.cuhk.edu.hk/~pluo/pdf/presentation_dpn.pdfSemantic Image Segmentation via Deep Parsing Network Ziwei Liu*, Xiaoxiao Li*,](https://reader030.vdocuments.us/reader030/viewer/2022040606/5eb50495e809f745e375b0d7/html5/thumbnails/47.jpg)
Semantic Image Segmentation via Deep Parsing Network
Thanks!
Project Page: http://personal.ie.cuhk.edu.hk/~lz013/projects/DPN.html