not all pixels are equal difficulty-aware semantic...
TRANSCRIPT
![Page 1: Not All Pixels are Equal Difficulty-aware Semantic ...luoping.me/project/layercascade/LayerCascade.pdf · Not All Pixels are Equal: Difficulty-aware Semantic Segmentation via Deep](https://reader035.vdocuments.us/reader035/viewer/2022081406/5f0f68f17e708231d4440561/html5/thumbnails/1.jpg)
Not All Pixels are Equal:Difficulty-aware Semantic Segmentation
via Deep Layer Cascade
Xiaoxiao Li, Ziwei Liu, Ping Luo, Chen Change Loy, Xiaoou Tang
Multimedia Lab, The Chinese University of Hong Kong
![Page 2: Not All Pixels are Equal Difficulty-aware Semantic ...luoping.me/project/layercascade/LayerCascade.pdf · Not All Pixels are Equal: Difficulty-aware Semantic Segmentation via Deep](https://reader035.vdocuments.us/reader035/viewer/2022081406/5f0f68f17e708231d4440561/html5/thumbnails/2.jpg)
State-of-the-art Method (4 FPS)
Deep Layer Cascade (17 FPS)
Problem
Input Video
![Page 3: Not All Pixels are Equal Difficulty-aware Semantic ...luoping.me/project/layercascade/LayerCascade.pdf · Not All Pixels are Equal: Difficulty-aware Semantic Segmentation via Deep](https://reader035.vdocuments.us/reader035/viewer/2022081406/5f0f68f17e708231d4440561/html5/thumbnails/3.jpg)
State-of-the-art
State-of-the-art Method (4 FPS) Fully Convolutional Network
• Why Slow?
• Very Deep Backbone Network
• High Resolution Feature Map
![Page 4: Not All Pixels are Equal Difficulty-aware Semantic ...luoping.me/project/layercascade/LayerCascade.pdf · Not All Pixels are Equal: Difficulty-aware Semantic Segmentation via Deep](https://reader035.vdocuments.us/reader035/viewer/2022081406/5f0f68f17e708231d4440561/html5/thumbnails/4.jpg)
Motivation
Image Easy Region Moderate Region Hard Region
![Page 5: Not All Pixels are Equal Difficulty-aware Semantic ...luoping.me/project/layercascade/LayerCascade.pdf · Not All Pixels are Equal: Difficulty-aware Semantic Segmentation via Deep](https://reader035.vdocuments.us/reader035/viewer/2022081406/5f0f68f17e708231d4440561/html5/thumbnails/5.jpg)
Contemporary Model
Image Stem Reduction-A5×IRNet-A 10×IRNet-B Reduction-B 5×IRNet-C
pooling
Fully-Connected Softmax
Image Stem Reduction-A5×IRNet-A 10×IRNet-B Reduction-B 5×IRNet-C ConvI L3
![Page 6: Not All Pixels are Equal Difficulty-aware Semantic ...luoping.me/project/layercascade/LayerCascade.pdf · Not All Pixels are Equal: Difficulty-aware Semantic Segmentation via Deep](https://reader035.vdocuments.us/reader035/viewer/2022081406/5f0f68f17e708231d4440561/html5/thumbnails/6.jpg)
Deep Layer Cascade
background
horse
person
car
unknown
Sta
ge 1
Image Stem Reduction-A5×IRNet-A
Conv
I
L1
![Page 7: Not All Pixels are Equal Difficulty-aware Semantic ...luoping.me/project/layercascade/LayerCascade.pdf · Not All Pixels are Equal: Difficulty-aware Semantic Segmentation via Deep](https://reader035.vdocuments.us/reader035/viewer/2022081406/5f0f68f17e708231d4440561/html5/thumbnails/7.jpg)
Deep Layer Cascade
background
horse
person
car
unknown
Sta
ge 1
Sta
ge 2
Image Stem Reduction-A5×IRNet-A 10×IRNet-B Reduction-B
ConvConv
I
L1 L2
![Page 8: Not All Pixels are Equal Difficulty-aware Semantic ...luoping.me/project/layercascade/LayerCascade.pdf · Not All Pixels are Equal: Difficulty-aware Semantic Segmentation via Deep](https://reader035.vdocuments.us/reader035/viewer/2022081406/5f0f68f17e708231d4440561/html5/thumbnails/8.jpg)
Deep Layer Cascade
background
horse
person
car
unknown
Sta
ge 1
Sta
ge 2
Sta
ge 3
Image Stem Reduction-A5×IRNet-A 10×IRNet-B Reduction-B 5×IRNet-C Conv
ConvConv
I
L1 L2
L3
![Page 9: Not All Pixels are Equal Difficulty-aware Semantic ...luoping.me/project/layercascade/LayerCascade.pdf · Not All Pixels are Equal: Difficulty-aware Semantic Segmentation via Deep](https://reader035.vdocuments.us/reader035/viewer/2022081406/5f0f68f17e708231d4440561/html5/thumbnails/9.jpg)
Region Convolution
Convolution Region Convolution
M
Region Convolution with Residual
+
![Page 10: Not All Pixels are Equal Difficulty-aware Semantic ...luoping.me/project/layercascade/LayerCascade.pdf · Not All Pixels are Equal: Difficulty-aware Semantic Segmentation via Deep](https://reader035.vdocuments.us/reader035/viewer/2022081406/5f0f68f17e708231d4440561/html5/thumbnails/10.jpg)
Performance
PASCAL VOC 2012
mIoU FPS (Backbone Network)
DPN 77.5 5.7
Adelaide 79.1 -
Deeplab-v2 79.7 7.1
LC(w/o COCO) 80.314.7
LC(with COCO) 82.7(PASCAL VOC 2012 Challenge test set)
![Page 11: Not All Pixels are Equal Difficulty-aware Semantic ...luoping.me/project/layercascade/LayerCascade.pdf · Not All Pixels are Equal: Difficulty-aware Semantic Segmentation via Deep](https://reader035.vdocuments.us/reader035/viewer/2022081406/5f0f68f17e708231d4440561/html5/thumbnails/11.jpg)
Stage Visualization
(a) input image (e) ground truth(b) stage-1 (c) stage-2 (d) stage-3
background aeroplane person carbottle cat busunknown
![Page 12: Not All Pixels are Equal Difficulty-aware Semantic ...luoping.me/project/layercascade/LayerCascade.pdf · Not All Pixels are Equal: Difficulty-aware Semantic Segmentation via Deep](https://reader035.vdocuments.us/reader035/viewer/2022081406/5f0f68f17e708231d4440561/html5/thumbnails/12.jpg)
• Difficulty-Aware Learning Paradigm
input video stage-1
stage-2 stage-3
• Region Convolution Real-Time
• End-To-End Trainable Framework
![Page 13: Not All Pixels are Equal Difficulty-aware Semantic ...luoping.me/project/layercascade/LayerCascade.pdf · Not All Pixels are Equal: Difficulty-aware Semantic Segmentation via Deep](https://reader035.vdocuments.us/reader035/viewer/2022081406/5f0f68f17e708231d4440561/html5/thumbnails/13.jpg)
Not All Pixels are Equal:Difficulty-aware Semantic Segmentation via Deep Layer Cascade
Thanks!
Code and models are available @
Project Page: http://personal.ie.cuhk.edu.hk/~lz013/projects/LayerCascade.html