robert bosch centre for data science and artificial intelligence … khapra.pdf · 2020. 9. 2. ·...
TRANSCRIPT
![Page 1: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/1.jpg)
Robert Bosch Centre for Data Science and Artificial IntelligenceDepartment of Computer Science and Engineering
Indian Institute of Technology Madras
Object Detection Over Scientific Plots
1
Dr. Mitesh Khapra Dr. Pratyush KumarNitesh Methani(Research Scholar, IIT Madras) (Assistant Professor, IIT Madras) (Assistant Professor, IIT Madras)
Pritha Ganguly(Research Scholar, IIT Madras)
![Page 2: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/2.jpg)
Introduction
Image Source: Google Images2
![Page 3: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/3.jpg)
Problem Statement
Fast and accurate detection of objects in scientific plots
Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.3
![Page 4: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/4.jpg)
Years Bulgaria Cuba
2002 677 593
2003 641 538
2004 604 485
2005 575 440
Introduction
Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.4
Q: What is the difference between the number of neonatal deaths in Bulgaria and Cuba in the year 2004?A: 119
Q: What is the average number of neonatal deaths in Cuba across years?A: 514
Q: In which year is the number of neonatal deaths in Bulgaria maximum?A: 2002
![Page 5: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/5.jpg)
Years Bulgaria Cuba
2002 677 593
2003 641 538
2004 604 485
2005 575 440
Introduction
Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.5
Q: What is the difference between the number of neonatal deaths in Bulgaria and Cuba in the year 2004?A: 119
Q: What is the average number of neonatal deaths in Cuba across years?A: 514
Q: In which year is the number of neonatal deaths in Cuba maximum?A: 2002
![Page 6: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/6.jpg)
Are existing object detection models good enough?
6
![Page 7: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/7.jpg)
Natural Images v/s Scientific Plots
7Image Source: M. Everingham, L. V. Gool, C. K. I. Williams, J. M. Winn, A. Zisserman, The Pascal VOC Challenge. Int. J. Comput. Vis., 2010Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.
Visual elements Visual elements Textual elements+
![Page 8: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/8.jpg)
Natural Images v/s Scientific Plots
8
Small to Large boxes X-Small to X-Large boxes
✔ ✔
Image Source: Google Images
![Page 9: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/9.jpg)
Natural Images v/s Scientific Plots
9
Structural Relationship Structural Relationship
❌
✔ ✔ ✔ ✔
✔
Image Source: Google Images
![Page 10: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/10.jpg)
Natural Images v/s Scientific Plots
10
0.5 IOU✔
✔ ✔
✔
✔ ✔
❌ 0.5 IOU❌
![Page 11: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/11.jpg)
Natural Images v/s Scientific Plots
11
0.5 IOU✔
✔ ✔
✔
✔ ✔
❌ 0.75 IOU❌
![Page 12: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/12.jpg)
Natural Images v/s Scientific Plots
12
0.5 IOU✔
✔ ✔
✔
✔ ✔
0.90 IOU❌ ✔
![Page 13: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/13.jpg)
Natural Images v/s Scientific Plots
13
✔
✔ ✔
✔
✔ ✔
Key Insight: OD over scientific plots has additional challenges as compared to OD over natural images
❌ 0.5 IOU 0.90 IOU✔
![Page 14: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/14.jpg)
Goal 1Investigate whether existing object detection methods are adequate for detecting text and visual elements in scientific plots which are arguably different than the objects found in natural images?
?
14
Twostage
Onestage
![Page 15: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/15.jpg)
Summary of Two Stage Detectors
15
⋯
w
h
w*
h*CNN
Feature Extractor
Warped Image regions
Regression output
Feature Volume
Classification output
InputImage
Flattened vector
⋰
WH
D
⋰
W H D
1R. B. Girshick, J. Donahue, T. Darrell, J. Malik, Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. CVPR14
![Page 16: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/16.jpg)
Summary of Two Stage Detectors
16
⋰CNN
Feature Extractor
Warped Image regions
Networkoutputs
Feature Volume
InputImage
Flattened vector
⋯
⋰CNN
⋰
⋯
CNN
WH
D
⋰
⋯
W H D
![Page 17: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/17.jpg)
ROI Poolfeatures
Summary of SOTA Models: Fast R-CNN2
17
CNN ⋰
⋰
⋰
h
Feature Extractor
Networkoutputs
Feature Volume
InputImage
Fully Connected Layers
x* y* w* h*
⋯
ROI Pool
2Ross B. Girshick, Fast R-CNN. ICCV 2015
![Page 18: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/18.jpg)
18
665
665
665/32 = 20.78800/32 = 25800
800
25
25
20
20 22
20/7 = 2.86
7
7
Coordinate on input image
Coordinate on feature map
CNNVGG16
{
{
Quantization Coordinate on ROI feature
Quantization
Fast R-CNN: ROI Pool
Image Source: https://ardianumam.wordpress.com/2017/12/16/understanding-how-mask-rcnn-works-for-semactic-segmentation/
![Page 19: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/19.jpg)
Summary of SOTA Models: Faster R-CNN3
19
CNN h
Feature Extractor
Feature Volume
InputImage
3S. Ren, K. He, Ross. B. Girshick, J. Sun, Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, NIPS 2015
![Page 20: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/20.jpg)
Summary of SOTA Models: Faster R-CNN
20
CNN
Region Proposal Network
h
Feature Extractor
Feature Volume
InputImage
⋰
⋰
⋰
ROI Poolfeatures
Network outputs
Fully Connected Layers
x* y* w* h*
⋯
![Page 21: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/21.jpg)
Summary of SOTA Models: Mask R-CNN4
21
CNN
Mask Branch
x* y* w* h*
h ⋰
Feature Extractor
ROI Align Network outputs
Feature Volume
InputImage
Fully Connected
⋯
Region Proposal Network
4Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross B. Girshick, Mask R-CNN, ICCV 2017
+ Mask Branch
![Page 22: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/22.jpg)
Summary of SOTA Models: Mask R-CNN
22
CNN h ⋰
Feature Extractor
ROI Align Network outputs
Feature Volume
InputImage
Fully Connected + Mask Branch
Mask Branch
x* y* w* h*
⋯
Region Proposal Network
![Page 23: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/23.jpg)
23
665
665
665/32 = 20.78800/32 = 25800
800
25
25
20.78
20.7
8
2.862.86
20/7 = 2.86
7
7
Coordinate on input image
Coordinate on feature map
CNNVGG16
{
{
No Quantization Coordinate on ROI feature
No Quantization
Mask R-CNN: ROI Align
Image Source: https://ardianumam.wordpress.com/2017/12/16/understanding-how-mask-rcnn-works-for-semactic-segmentation/
![Page 24: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/24.jpg)
24
ROI Pool vs ROI Align
RO
IPoo
lR
OIA
lign
![Page 25: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/25.jpg)
25
Summary of Two Stage Detectors
Model Key Ideas
FRCNN
FrRCNN
MRCNN
ROI Pool
Region Proposal Network
ROI Align
![Page 26: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/26.jpg)
26
c x y w h . . .
P(person)
P(horse)
P(cycle)
⋯
Summary of One Stage Detectors
Input Image Output vector for each grid cell
1 x (1 + 4 + K)
1J. Redmon, S. K. Divvala, R. B. Girshick, A. Farhadi, You Only Look Once: Unified, Real-Time Object Detection. CVPR 2016
![Page 27: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/27.jpg)
c x y w h . . .
27
P(person)
P(horse)
P(cycle)
⋯
Summary of One Stage Detectors
Input Image Output vector for each grid cell
1 x (1 + 4 + K)
![Page 28: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/28.jpg)
c x y w h . . .
28
P(person)
P(horse)
P(cycle)
⋯
Summary of One Stage Detectors
Input Image Output vector for each grid cell
1 x (1 + 4 + K)
![Page 29: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/29.jpg)
c x y w h . . .
29
P(person)
P(horse)
P(cycle)
⋯
Summary of One Stage Detectors
Input Image Output vector for each grid cell
1 x (1 + 4 + K)
![Page 30: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/30.jpg)
c x y w h . . .
30
P(person)
P(horse)
P(cycle)
⋯
Summary of One Stage Detectors
Input Image Output vector for each grid cell
1 x (1 + 4 + K)
![Page 31: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/31.jpg)
c x y w h . . .
31
P(person)
P(horse)
P(cycle)
⋯
Summary of One Stage Detectors
Input Image Output vector for each grid cell
1 x (1 + 4 + K)
![Page 32: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/32.jpg)
32
Summary of SOTA Models: YOLO
7
Input Image
1 + 4 + K7
DarkNetArchitecture
Output volume Feature Extractor
![Page 33: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/33.jpg)
33
Summary of SOTA Models: YOLO-v3
H
Input Image
B x (1 + 4 + K) W
DarkNetArchitecture
Output volume Feature Extractor
⋮ ‘B’ bounding box priors
![Page 34: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/34.jpg)
300x300x3
34
Summary of SOTA Models: YOLO
Input Image
DarkNet
Feature Extractor
Out
put
![Page 35: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/35.jpg)
300x300x3
35
Summary of SOTA Models: SSD2
Input Image Feature Extractor
38x38x512
VGG16
Out
put
conv5 last
2W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. E. Reed, C.Y Fu, A. C. Berg, SSD: Single Shot MultiBox Detector. ECCV 2016
![Page 36: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/36.jpg)
300x300x3
36
Summary of SOTA Models: SSD
Input Image Feature Extractor
38x38x512
VGG16
Out
put
conv4 last
![Page 37: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/37.jpg)
300x300x3 38x38x512 19x19x1024 19x19x1024
37
Out
put
1x1 conv
1x1 conv
Summary of SOTA Models: SSD
Input Image
conv4 last
conv6 (FC6)
conv7 (FC7)
VGG Layers
![Page 38: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/38.jpg)
300x300x3 38x38x512 19x19x1024 19x19x1024 10x10x512 5x5x256 3x3x256 1x1x256
38
1x1 conv
1x1 conv
conv4 last
conv6 (FC6)
conv7 (FC7)
conv8 conv9 conv10 conv11
Summary of SOTA Models: SSD
Input Image VGG Layers Extra Layers
Out
put
![Page 39: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/39.jpg)
c x y w h
39
Summary of SOTA Models: RetinaNet3
3T. Lin, P. Goyal, R. B. Girshick, K. He, and P. Dollár, Focal loss for dense object detection, ICCV 2017
Input Image
1 + 4 + K7
DarkNet
Output volume Feature Extractor
7
k
![Page 40: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/40.jpg)
40
Summary of SOTA Models: RetinaNet
Output volume Feature Pyramid Network
W ✕ H ✕ 4A
W ✕ H ✕ kA
Input Image
⋮ ‘A’ anchor boxes
![Page 41: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/41.jpg)
41
Summary of SOTA Models: RetinaNet
Output volume
W ✕ H ✕ 4A
W ✕ H ✕ kA
Input Image
⋮ ‘A’ anchor boxes
Feature Pyramid Network
![Page 42: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/42.jpg)
42
Summary of SOTA Models: RetinaNet
Output volume
W ✕ H ✕ 4A
W ✕ H ✕ kA
Input Image
⋮ ‘A’ anchor boxes
Feature Pyramid Network
![Page 43: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/43.jpg)
43
Image Conv1 Conv2 Conv3 Conv4 Conv5 FC Softmax
RetinaNet: Feature Pyramid Network4
4T.Y. Lin, P. Dollár, R. B. Girshick, K. He, B. Hariharan, S. J. Belongie, Feature Pyramid Networks for Object Detection. CVPR 2017
![Page 44: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/44.jpg)
44
RetinaNet: Feature Pyramid Network
Image Conv1(last)
Conv2(last)
Conv3(last)
Conv4(last)
Conv5(last)
![Page 45: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/45.jpg)
45
Image C1 C2 C3 C4 C5
RetinaNet: Feature Pyramid Network
![Page 46: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/46.jpg)
46
RetinaNet: Feature Pyramid Network
C1
C2
C3
C4C5
Bottom-Up Pathway
![Page 47: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/47.jpg)
47
1x1
RetinaNet: Feature Pyramid Network
C1
C2
C3
C4C5 M5
Bottom-Up Pathway Top-Down Pathway
![Page 48: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/48.jpg)
48
1x1
2x
RetinaNet: Feature Pyramid Network
C1
C2
C3
C4C5
M4
M5
Bottom-Up Pathway Top-Down Pathway
![Page 49: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/49.jpg)
49
+
1x1
1x1 2x
RetinaNet: Feature Pyramid Network
C1
C2
C3
C4C5
M4
M5
Bottom-Up Pathway Top-Down Pathway
![Page 50: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/50.jpg)
50
+
+
+
1x1
1x1
1x1
1x1
2x
2x
2x
RetinaNet: Feature Pyramid Network
C1
C2
C3
C4C5
M2
M3
M4
M5
Bottom-Up Pathway Top-Down Pathway
![Page 51: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/51.jpg)
51
+
+
+
1x1
1x1
1x1
1x1
3x3
2x
2x
2x
RetinaNet: Feature Pyramid Network
C1
C2
C3
C4C5
M2
P5
M3
M4
M5
Bottom-Up Pathway Top-Down Pathway Pyramid Feature Maps
![Page 52: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/52.jpg)
52
+
+
+
1x1
1x1
1x1
1x1
3x3
3x3
3x3
3x32x
2x
2x
RetinaNet: Feature Pyramid Network
C1
C2
C3
C4C5
M2 P2
P3
P4
P5
M3
M4
M5
Bottom-Up Pathway Top-Down Pathway Pyramid Feature Maps
![Page 53: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/53.jpg)
53
YOLO
SSD
RetinaNet
Summary of One Stage DetectorsModel Key Ideas
Grid based proposal
Featurised Pyramid
Feature Pyramid Network
![Page 54: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/54.jpg)
54
Recap of Goal 1Investigate whether existing object detection methods are adequate for detecting text and visual elements in scientific plots which are arguably different than the objects found in natural images?
?
Twostage
Onestage
![Page 55: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/55.jpg)
Dataset: PlotQA
Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.
● Contains over 220,000 scientific plots across three categories:○ Bar (both horizontal and vertical) plots○ Line, and ○ Dot-line plots
55
![Page 56: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/56.jpg)
Evaluation Metric
56
IOU = = 0.54
Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.
IOU = = 0.52
![Page 57: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/57.jpg)
Evaluation Metric
57
IOU = = 0.54 IOU = = 0.79 IOU = = 0.98
Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.
![Page 58: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/58.jpg)
Evaluation Metric
58
0.54 0.79 0.98
Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.
Mark a prediction as correct if it has a 50% IOU with the ground-truth box
![Page 59: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/59.jpg)
Evaluation Metric
59Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.
Mark a prediction as correct if it has a 50% IOU with the ground-truth box
0.54 0.79 0.98
![Page 60: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/60.jpg)
Evaluation Metric
60
0.54 0.79 0.98
Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.
Mark a prediction as correct if it has a 75% IOU with the ground-truth box
![Page 61: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/61.jpg)
Evaluation Metric
61Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.
Mark a prediction as correct if it has a 75% IOU with the ground-truth box
0.54 0.79 0.98
![Page 62: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/62.jpg)
Evaluation Metric
62Image Source: Nitesh Methani, Pritha Ganguly, Mitesh Khapra, Pratyush Kumar, PlotQA: Reasoning over Scientific Plots, WACV 2020.
Mark a prediction as correct if it has a 90% IOU with the ground-truth box
0.54 0.79 0.98
![Page 63: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/63.jpg)
Evaluation of SOTA Models
Models [email protected]
SSD 82.33
YOLO-v3 96.27
RetinaNet 90.13
FRCNN 72.83
FrRCNN 88.49
MRCNN 93.72
IOU = 0.53Table: Comparison of existing object detection models on the PlotQA dataset with mAP scores (in %) at IOU of 0.5.
63
![Page 64: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/64.jpg)
Evaluation of SOTA Models
Models [email protected] [email protected]
SSD 82.33 39.78
YOLO-v3 96.27 73.31
RetinaNet 90.13 81.13
FRCNN 72.83 63.68
FrRCNN 88.49 50.51
MRCNN 93.72 82.45
IOU = 0.59 IOU = 0.75Table: Comparison of existing object detection models on the PlotQA dataset with mAP scores (in %) at IOUs of 0.5, and 0.75.
64
![Page 65: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/65.jpg)
Evaluation of SOTA Models
Models [email protected] [email protected] [email protected]
SSD 82.33 39.78 1.53
YOLO-v3 96.27 73.31 7.43
RetinaNet 90.13 81.13 30.56
FRCNN 72.83 63.68 21.45
FrRCNN 88.49 50.51 4.08
MRCNN 93.72 82.45 35.70
IOU = 0.59 IOU = 0.75 IOU = 0.96Table: Comparison of existing object detection models on the PlotQA dataset with mAP scores (in %) at IOUs of 0.5, 0.75, and 0.9.
65
![Page 66: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/66.jpg)
Evaluation of SOTA Models
IOU 0.9 0.75 0.5
Models\Class
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
mAP mAP mAP
SSD 1.39 0.60 2.18 0.39 0.04 3.39 0.44 5.14 0.20 1.53 39.78 82.33
YOLO-v3 15.51 8.72 7.15 11.70 0.02 4.39 8.08 9.59 1.70 7.43 73.31 96.27
RetinaNet 16.51 18.50 77.26 29.74 16.58 67.62 28.40 3.14 17.31 30.56 81.13 90.13
FRCNN 53.38 1.68 12.59 14.06 0.03 42.13 25.49 11.68 31.98 21.45 63.68 72.83
FrRCNN 6.92 1.68 1.39 1.45 0.00 4.35 6.10 3.57 5.18 4.08 50.51 88.49
MRCNN 47.54 5.36 50.83 32.43 0.33 40.20 33.72 80.53 30.31 35.70 82.45 93.72
Table: Comparison of existing object detection models on the PlotQA dataset with mAP scores (in %) at IOUs of 0.5, 0.75, and 0.9.
66
![Page 67: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/67.jpg)
Evaluation of SOTA Models
IOU 0.9 0.75 0.5
Models\Class
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
mAP mAP mAP
SSD 1.39 0.60 2.18 0.39 0.04 3.39 0.44 5.14 0.20 1.53 39.78 82.33
YOLO-v3 15.51 8.72 7.15 11.70 0.02 4.39 8.08 9.59 1.70 7.43 73.31 96.27
RetinaNet 16.51 18.50 77.26 29.74 16.58 67.62 28.40 3.14 17.31 30.56 81.13 90.13
FRCNN 53.38 1.68 12.59 14.06 0.03 42.13 25.49 11.68 31.98 21.45 63.68 72.83
FrRCNN 6.92 1.68 1.39 1.45 0.00 4.35 6.10 3.57 5.18 4.08 50.51 88.49
MRCNN 47.54 5.36 50.83 32.43 0.33 40.20 33.72 80.53 30.31 35.70 82.45 93.72
Table: Comparison of existing object detection models on the PlotQA dataset with mAP scores (in %) at IOUs of 0.5, 0.75, and 0.9.
67
![Page 68: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/68.jpg)
Evaluation of SOTA Models
IOU 0.9 0.75 0.5
Models\Class
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
mAP mAP mAP
SSD 1.39 0.60 2.18 0.39 0.04 3.39 0.44 5.14 0.20 1.53 39.78 82.33
YOLO-v3 15.51 8.72 7.15 11.70 0.02 4.39 8.08 9.59 1.70 7.43 73.31 96.27
RetinaNet 16.51 18.50 77.26 29.74 16.58 67.62 28.40 3.14 17.31 30.56 81.13 90.13
FRCNN 53.38 1.68 12.59 14.06 0.03 42.13 25.49 11.68 31.98 21.45 63.68 72.83
FrRCNN 6.92 1.68 1.39 1.45 0.00 4.35 6.10 3.57 5.18 4.08 50.51 88.49
MRCNN 47.54 5.36 50.83 32.43 0.33 40.20 33.72 80.53 30.31 35.70 82.45 93.72
Table: Comparison of existing object detection models on the PlotQA dataset with mAP scores (in %) at IOUs of 0.5, 0.75, and 0.9.
68
![Page 69: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/69.jpg)
Qualitative Analysis: SSD
Figure: An example plot from PlotQA dataset.
69
Figure: Detected bounding boxes on an example plot from PlotQA dataset.
![Page 70: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/70.jpg)
Qualitative Analysis: YOLO-v3
Figure: An example plot from PlotQA dataset.
70
Figure: Detected bounding boxes on an example plot from PlotQA dataset.
![Page 71: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/71.jpg)
Qualitative Analysis: RetinaNet
Figure: An example plot from PlotQA dataset.
71
Figure: Detected bounding boxes on an example plot from PlotQA dataset.
![Page 72: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/72.jpg)
Qualitative Analysis: FRCNN
Figure: An example plot from PlotQA dataset.
72
Figure: Detected bounding boxes on an example plot from PlotQA dataset.
![Page 73: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/73.jpg)
Qualitative Analysis: FrRCNN
Figure: An example plot from PlotQA dataset.
73
Figure: Detected bounding boxes on an example plot from PlotQA dataset.
![Page 74: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/74.jpg)
Qualitative Analysis: MRCNN
Figure: An example plot from PlotQA dataset.
74
Figure: Detected bounding boxes on an example plot from PlotQA dataset.
![Page 75: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/75.jpg)
Qualitative Analysis: Summary
7575
SSD YOLO-v3
Retinanet FRCNN
FrRCNN MRCNN
Longer textual objects
Very short objects
Higher IOU settings
❌
❌
❌
Key Observations:
![Page 76: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/76.jpg)
Qualitative Analysis: Summary
7676
Retinanet FRCNN
FrRCNN MRCNN
SSD YOLO-v3Longer textual objects
Very short objects
Higher IOU settings
FPN helps
❌
❌
✔
Key Observations:
❌
![Page 77: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/77.jpg)
Longer textual objects
Very short objects
Higher IOU settings
FPN helps
ROIAlign helps
Qualitative Analysis: Summary
7777
❌
❌
✔
✔
Key Observations:
FrRCNN MRCNN
SSD YOLO-v3
Retinanet FRCNN
❌
![Page 78: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/78.jpg)
Design a deep learning based object detection network that accurately and efficiently detects all the textual and visual objects present in a scientific plot.
Goal 2
Accurate & Efficient
78Image Source: Google Images
![Page 79: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/79.jpg)
Faster R-CNN backbone Feature Pyramid Network (FPN) ROIAlign (RA)
+ +
79
A Hybrid Model
![Page 80: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/80.jpg)
A Hybrid Model: Results
IOU 0.9 0.75 0.5
Class\Models
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
mAP mAP mAP
FRCNN(RA) 83.25 16.32 62.31 59.10 0.18 65.77 78.72 42.94 78.87 54.16 68.92 72.46
FRCNN(FPN+RA) 87.59 31.62 79.05 66.39 0.22 69.78 88.29 46.63 84.60 61.57 69.82 72.18
FrRCNN(RA) 63.89 14.79 70.95 60.61 0.18 83.89 60.76 93.47 50.87 55.49 89.14 96.80
Hybrid Model
(FrRCNN+FPN+RA)
85.54 27.86 93.68 96.30 0.22 99.09 96.04 99.46 96.80 77.22 94.58 97.76
80
Table 2: Comparison of modified models on the PlotQA dataset with mAP scores (in %) at IOUs of 0.9, 0.75 & 0.5.
![Page 81: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/81.jpg)
A Hybrid Model: Results
IOU 0.9 0.75 0.5
Class\Models
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
mAP mAP mAP
FRCNN(RA) 83.25 16.32 62.31 59.10 0.18 65.77 78.72 42.94 78.87 54.16 68.92 72.46
FRCNN(FPN+RA) 87.59 31.62 79.05 66.39 0.22 69.78 88.29 46.63 84.60 61.57 69.82 72.18
FrRCNN(RA) 63.89 14.79 70.95 60.61 0.18 83.89 60.76 93.47 50.87 55.49 89.14 96.80
Hybrid Model
(FrRCNN+FPN+RA)
85.54 27.86 93.68 96.30 0.22 99.09 96.04 99.46 96.80 77.22 94.58 97.76
81
Table 2: Comparison of modified models on the PlotQA dataset with mAP scores (in %) at IOUs of 0.9, 0.75 & 0.5.
![Page 82: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/82.jpg)
A Hybrid Model: Results
IOU 0.9 0.75 0.5
Class\Models
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
mAP mAP mAP
FRCNN(RA) 83.25 16.32 62.31 59.10 0.18 65.77 78.72 42.94 78.87 54.16 68.92 72.46
FRCNN(FPN+RA) 87.59 31.62 79.05 66.39 0.22 69.78 88.29 46.63 84.60 61.57 69.82 72.18
FrRCNN(RA) 63.89 14.79 70.95 60.61 0.18 83.89 60.76 93.47 50.87 55.49 89.14 96.80
Hybrid Model
(FrRCNN+FPN+RA)
85.54 27.86 93.68 96.30 0.22 99.09 96.04 99.46 96.80 77.22 94.58 97.76
82
Table 2: Comparison of modified models on the PlotQA dataset with mAP scores (in %) at IOUs of 0.9, 0.75 & 0.5.
![Page 83: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/83.jpg)
A Hybrid Model: Results
IOU 0.9 0.75 0.5
Class\Models
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
mAP mAP mAP
FRCNN(RA) 83.25 16.32 62.31 59.10 0.18 65.77 78.72 42.94 78.87 54.16 68.92 72.46
FRCNN(FPN+RA) 87.59 31.62 79.05 66.39 0.22 69.78 88.29 46.63 84.60 61.57 69.82 72.18
FrRCNN(RA) 63.89 14.79 70.95 60.61 0.18 83.89 60.76 93.47 50.87 55.49 89.14 96.80
Hybrid Model
(FrRCNN+FPN+RA)
85.54 27.86 93.68 96.30 0.22 99.09 96.04 99.46 96.80 77.22 94.58 97.76
83
Table 2: Comparison of modified models on the PlotQA dataset with mAP scores (in %) at IOUs of 0.9, 0.75 & 0.5.
![Page 84: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/84.jpg)
A Hybrid Model: Results
IOU 0.9 0.75 0.5
Class\Models
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
mAP mAP mAP
FRCNN(RA) 83.25 16.32 62.31 59.10 0.18 65.77 78.72 42.94 78.87 54.16 68.92 72.46
FRCNN(FPN+RA) 87.59 31.62 79.05 66.39 0.22 69.78 88.29 46.63 84.60 61.57 69.82 72.18
FrRCNN(RA) 63.89 14.79 70.95 60.61 0.18 83.89 60.76 93.47 50.87 55.49 89.14 96.80
Hybrid Model
(FrRCNN+FPN+RA)
85.54 27.86 93.68 96.30 0.22 99.09 96.04 99.46 96.80 77.22 94.58 97.76
84
Table 2: Comparison of modified models on the PlotQA dataset with mAP scores (in %) at IOUs of 0.9, 0.75 & 0.5.
![Page 85: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/85.jpg)
A Hybrid Model: Qualitative Analysis
FRCNN(RA) FRCNN(FPN+RA)
FrRCNN(RA) Hybrid Model ( FrRCNN+FPN+RA)
Figure: Detected bounding boxes on an example plot from PlotQA dataset for different hybrid models corresponding to Table 2 at an IOU threshold of 0.9.
85
![Page 86: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/86.jpg)
Figure: mAP (in %) v/s Inference Time per image (in ms) for different object detection models on PlotQA at an IOU setting of 0.9. (x, y) represents the tuple (mAP, time). 86
A Hybrid Model: Summary
Existing Models
![Page 87: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/87.jpg)
Figure: mAP (in %) v/s Inference Time per image (in ms) for different object detection models on PlotQA at an IOU setting of 0.9. (x, y) represents the tuple (mAP, time). 87
A Hybrid Model: Summary
Existing Models
Hybrid Model
![Page 88: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/88.jpg)
Figure: mAP (in %) v/s Inference Time per image (in ms) for different object detection models on PlotQA at an IOU setting of 0.9. (x, y) represents the tuple (mAP, time). 88
A Hybrid Model: Summary
Preferred region
![Page 89: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/89.jpg)
Can we do better (faster and more efficient)?
89
![Page 90: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/90.jpg)
1x1 FC
90
Proposed Model: PlotNet
CH
RH
LH
ROI Mask Feature MapRGB Image ROI Aligned Features ROI Volumes Output VectorsFinal Vector
1024
25632
0
![Page 91: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/91.jpg)
1x1 FC
91
Proposed Model: PlotNet
CH
RH
LH
ROI Mask Feature MapRGB Image ROI Aligned Features ROI Volumes Output VectorsFinal Vector
1024
320
256
CV-based proposal
![Page 92: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/92.jpg)
1x1 FC
92
Proposed Model: PlotNet
CH
RH
LH
Feature Map ROI Aligned Features ROI Volumes Output VectorsFinal Vector
1024
320
256
Feature Extractor
ROI Mask RGB Image
![Page 93: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/93.jpg)
1x1 FC
93
Proposed Model: PlotNet
CH
RH
LH
ROI Aligned Features ROI Volumes Output VectorsFinal Vector
1024
320
256
ROI Align
ROI Mask RGB Image Feature Map
![Page 94: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/94.jpg)
1x1 FC
94
Proposed Model: PlotNet
CH
RH
LH
ROI Volumes Output VectorsFinal Vector
1024
320
256
ROI Mask Feature MapRGB Image ROI Aligned Features
AN-ROI Layer
![Page 95: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/95.jpg)
95
Proposed Model: PlotNet
FC
CH
RH
LH
Output VectorsFinal Vector
1024
Class, Regress and Linking Heads
ROI Mask Feature MapRGB Image ROI Aligned Features
1x1
320
256
ROI Volumes
![Page 96: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/96.jpg)
96
Proposed Model: PlotNet
FC
CH
RH
LH
Output VectorsFinal Vector
1024
Class, Regress and Linking Heads
ROI Mask Feature MapRGB Image ROI Aligned Features
1x1
320
256
ROI Volumes
![Page 97: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/97.jpg)
RGB Image Grayscale Image
Preprocess-ing
97
PlotNet: CV-based Region Proposal
![Page 98: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/98.jpg)
RGB Image Laplacian Edges
LaplacianEdge
Detector
98
PlotNet: CV-based Region Proposal
![Page 99: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/99.jpg)
RGB Image Contoured Image
Contour Detection
99
PlotNet: CV-based Region Proposal
![Page 100: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/100.jpg)
RGB Image Proposed ROIs
Fit Bounding
Boxes
100
PlotNet: CV-based Region Proposal
![Page 101: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/101.jpg)
RGB Image Proposed ROIs 1-D ROI Mask
CV techniques
Mask formation
101
PlotNet: CV-based Region Proposal
![Page 102: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/102.jpg)
102
PlotNet: Feature Extractor
ROI Mask Feature MapRGB Image
![Page 103: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/103.jpg)
103
PlotNet: Feature Extractor
ROI Mask Feature MapRGB Image
![Page 104: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/104.jpg)
104
PlotNet: ROI Align Layer
ROI Mask Feature MapRGB Image ROI Aligned Features
![Page 105: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/105.jpg)
105
PlotNet: AN-ROI Layer
ROI Mask Feature MapRGB Image ROI Aligned Features ROI Volumes
Volume size:14 x 14 x 320
![Page 106: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/106.jpg)
106
Volume size:14 x 14 x 256
PlotNet: AN-ROI Layer
ROI Mask Feature MapRGB Image ROI Aligned Features ROI Volumes
1x1
320
![Page 107: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/107.jpg)
1x1 FC
107
PlotNet: Class, Regress, and Linking Heads
CH
RH
LH
ROI Mask Feature MapRGB Image ROI Aligned Features ROI Volumes Output VectorsFinal Vector
1024
25632
0
![Page 108: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/108.jpg)
1x1 FC
108
PlotNet: Class, Regress, and Linking Heads
CH
RH
LH
ROI Mask Feature MapRGB Image ROI Aligned Features ROI Volumes Output VectorsFinal Vector
1024
25632
0
for every ROI
![Page 109: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/109.jpg)
Figure: Comparison of different loss functions at varying IOUs. 109
PlotNet: Loss Function
IOU
Loss
High IOU region
Key Insight
Existing losses give negligible values at high IoUs
![Page 110: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/110.jpg)
Figure: Comparison of different loss functions at varying IOUs. 110
PlotNet: Loss Function
IOU
Loss
● Gives non-negligible values at high IOUs
● Mathematically, the loss is defined:
● 𝛄 determines the rate of the scaling factor
Our Contribution
![Page 111: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/111.jpg)
Figure: Comparison of different loss functions at varying IOUs. 111
PlotNet: Loss Function
IOU
Loss
● Gives non-negligible values at high IOUs
● Mathematically, the loss is defined:
● 𝛄 determines the rate of the scaling factor
Non-negligible values
Our Contribution
![Page 112: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/112.jpg)
Figure: Comparison of different loss functions at varying IOUs. 112
PlotNet: Loss Function
IOU
Loss
● Gives non-negligible values at high IOUs
● Mathematically, the loss is defined:
● 𝛄 determines the rate of the scaling factor
Our Contribution
![Page 113: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/113.jpg)
113
PlotNet: Different Configurations
1)
2)
3)
4)
5)
6)
7)
8)
: AN-ROI Layer : Regression Loss: ROI Features
![Page 114: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/114.jpg)
WACV2021: Additional Experiments
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89
92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77
91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87
91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34
91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88
91.71 49.30 95.99 93.55 98.42 95.03 89.77 94.08 86.06 88.21
91.35 52.22 96.31 93.45 96.82 96.18 89.63 95.46 94.07 89.50
91.15 55.03 97.89 92.99 99.46 96.33 91.30 90.40 95.48 90.00
114
Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9.
![Page 115: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/115.jpg)
WACV2021: Additional Experiments
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19
92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39
92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44
92.84 69.19 98.49 95.22 99.42 96.88 94.65 96.51 96.87 93.34
92.96 68.25 98.34 95.96 98.96 97.10 95.08 96.84 97.16 93.41
92.91 58.52 98.40 95.95 95.14 97.16 94.59 97.55 95.63 91.76
115
Table 6: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9.
![Page 116: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/116.jpg)
WACV2021: Additional Experiments
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
85.30 52.85 29.64 94.30 0.00 10.36 80.77 1.47 81.59 48.48
76.14 61.49 40.19 96.51 0.00 16.49 79.05 1.76 77.68 49.92no linking
no linking
![Page 117: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/117.jpg)
PlotNet: Results
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89
92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77
91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87
91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34
91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88
92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19
92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39
92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44
117
Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9.
![Page 118: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/118.jpg)
PlotNet: Results
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89
92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77
91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87
91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34
91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88
92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19
92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39
92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44
118
Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9.
![Page 119: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/119.jpg)
PlotNet: Results
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89
92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77
91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87
91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34
91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88
92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19
92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39
92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44
119
Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9.
![Page 120: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/120.jpg)
PlotNet: Results
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89
92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77
91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87
91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34
91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88
92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19
92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39
92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44
120
Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9.
![Page 121: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/121.jpg)
PlotNet: Results
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89
92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77
91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87
91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34
91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88
92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19
92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39
92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44
121
Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9.
![Page 122: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/122.jpg)
● PlotNet performs better than all existing methods at all IOUs.
● At 0.9 IOU threshold, PlotNet improves upon its closest competitor by 16.22 absolute points.
Figure: Detected bounding boxes by PlotNet-v7 on an example plot from PlotQA dataset at an IOU threshold of 0.9.
122
PlotNet: Qualitative Analysis
![Page 123: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/123.jpg)
PlotNet: Comparison to other models
Figure: mAP (in %) v/s Inference Time per image (in ms) for different object detection models on PlotQA at an IOU setting of 0.9. (x, y) represents the tuple (mAP, time).
123
![Page 124: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/124.jpg)
PlotNet: Comparison to other models
Figure: mAP (in %) v/s Inference Time per image (in ms) for different object detection models on PlotQA at an IOU setting of 0.9. (x, y) represents the tuple (mAP, time).
124
![Page 125: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/125.jpg)
PlotNet: Comparison to other models
Figure: mAP (in %) v/s Inference Time per image (in ms) for different object detection models on PlotQA at an IOU setting of 0.9. (x, y) represents the tuple (mAP, time).
125
16.22pts
![Page 126: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/126.jpg)
Figure: mAP v/s IOU threshold for different object detection models.126
PlotNet: Comparison to other models
![Page 127: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/127.jpg)
Use-Case: Plot to Table Converter
(a) Input Image
(c) Ground-truth Table (d) Generated Table
(b) Predicted bounding boxes
Figure: Sample table generation using PlotNet's predictions 127
![Page 128: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/128.jpg)
Use-Case: Plot to Table Converter
(a) Input Image
(c) Ground-truth Table (d) Generated Table
(b) Predicted bounding boxes
Figure: Sample table generation using PlotNet's predictions 128
![Page 129: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/129.jpg)
129
Conclusion
Evaluated existing methods and exemplified the challenges
Proposed PlotNet addressing all the challenges
![Page 130: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/130.jpg)
130
High Recall Proposal Method End2End Training
Future Work
Image Source: Google Images
![Page 131: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/131.jpg)
Communicated:
Pritha Ganguly*, Nitesh Methani*, Mitesh M. Khapra and Pratyush Kumar, A Systematic Evaluation of Object Detection Networks for Scientific Plots., Under review at a Computer Vision Conference.
*the first two authors have contributed equally. 131
Visible Outcome
![Page 132: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/132.jpg)
Dr. Mitesh Khapra Dr. Pratyush KumarNitesh Methani(Research Scholar, IIT Madras) (Assistant Professor, IIT Madras) (Assistant Professor, IIT Madras)
132
Team
Pritha Ganguly(Research Scholar, IIT Madras)
![Page 133: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/133.jpg)
Nikita Suman
Siddhartha Preksha
TarunAnanya
Darwin Aakriti Madhura
JananiHimanshu
Shashank Nikhilesh
Priyesh
Deepak Shweta
133
Thank You!
![Page 134: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/134.jpg)
Image Source: Google Images134
Questions or Suggestions?
![Page 135: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/135.jpg)
135
Extra Slides
![Page 136: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/136.jpg)
PlotNet: Finding the Best Feature Extractor
136
![Page 137: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/137.jpg)
PlotNet: Finding the best feature extractor
IOU 0.9 0.75 0.5
Models\Class
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
mAP mAP mAP
R-10 89.46 37.63 93.64 78.12 95.22 94.06 88.10 66.95 83.86 80.78 96.74 97.57
R-22 91.37 24.08 97.03 81.01 98.79 90.47 81.99 51.79 47.92 73.83 97.01 98.08
R-50 87.64 15.72 74.57 41.87 98.92 81.60 54.21 43.35 35.67 59.28 93.91 97.67
R-22 FPN 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11
R-50 FPN 90.77 5.12 95.58 80.72 99.16 94.79 76.83 65.56 58.17 74.08 94.09 97.67
137
Table 3: Comparison of different variants of PlotNet on the PlotQA dataset by varying the number of layers in the ResNet(R)-50 architecture with mAP scores (in %) at IOUs of 0.9, 0.75, and 0.5.
![Page 138: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/138.jpg)
PlotNet: Finding the best feature extractor
IOU 0.9 0.75 0.5
Models\Class
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
mAP mAP mAP
R-10 89.46 37.63 93.64 78.12 95.22 94.06 88.10 66.95 83.86 80.78 96.74 97.57
R-22 91.37 24.08 97.03 81.01 98.79 90.47 81.99 51.79 47.92 73.83 97.01 98.08
R-50 87.64 15.72 74.57 41.87 98.92 81.60 54.21 43.35 35.67 59.28 93.91 97.67
R-22 FPN 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11
R-50 FPN 90.77 5.12 95.58 80.72 99.16 94.79 76.83 65.56 58.17 74.08 94.09 97.67
138
Table 3: Comparison of different variants of PlotNet on the PlotQA dataset by varying the number of layers in the ResNet(R)-50 architecture with mAP scores (in %) at IOUs of 0.9, 0.75, and 0.5.
![Page 139: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/139.jpg)
PlotNet: Finding the best feature extractor
IOU 0.9 0.75 0.5
Models\Class
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
mAP mAP mAP
R-10 89.46 37.63 93.64 78.12 95.22 94.06 88.10 66.95 83.86 80.78 96.74 97.57
R-22 91.37 24.08 97.03 81.01 98.79 90.47 81.99 51.79 47.92 73.83 97.01 98.08
R-50 87.64 15.72 74.57 41.87 98.92 81.60 54.21 43.35 35.67 59.28 93.91 97.67
R-22 FPN 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11
R-50 FPN 90.77 5.12 95.58 80.72 99.16 94.79 76.83 65.56 58.17 74.08 94.09 97.67
139
Table 3: Comparison of different variants of PlotNet on the PlotQA dataset by varying the number of layers in the ResNet(R)-50 architecture with mAP scores (in %) at IOUs of 0.9, 0.75, and 0.5.
![Page 140: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/140.jpg)
PlotNet: Finding the best feature extractor
IOU 0.9 0.75 0.5
Models\Class
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
mAP mAP mAP
R-10 89.46 37.63 93.64 78.12 95.22 94.06 88.10 66.95 83.86 80.78 96.74 97.57
R-22 91.37 24.08 97.03 81.01 98.79 90.47 81.99 51.79 47.92 73.83 97.01 98.08
R-50 87.64 15.72 74.57 41.87 98.92 81.60 54.21 43.35 35.67 59.28 93.91 97.67
R-22 FPN 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11
R-50 FPN 90.77 5.12 95.58 80.72 99.16 94.79 76.83 65.56 58.17 74.08 94.09 97.67
140
Table 3: Comparison of different variants of PlotNet on the PlotQA dataset by varying the number of layers in the ResNet(R)-50 architecture with mAP scores (in %) at IOUs of 0.9, 0.75, and 0.5.
![Page 141: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/141.jpg)
PlotNet: Finding the best feature extractor
IOU 0.9 0.75 0.5
Models\Class
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
mAP mAP mAP
R-10 89.46 37.63 93.64 78.12 95.22 94.06 88.10 66.95 83.86 80.78 96.74 97.57
R-22 91.37 24.08 97.03 81.01 98.79 90.47 81.99 51.79 47.92 73.83 97.01 98.08
R-50 87.64 15.72 74.57 41.87 98.92 81.60 54.21 43.35 35.67 59.28 93.91 97.67
R-22 FPN 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11
R-50 FPN 90.77 5.12 95.58 80.72 99.16 94.79 76.83 65.56 58.17 74.08 94.09 97.67
141
Table 3: Comparison of different variants of PlotNet on the PlotQA dataset by varying the number of layers in the ResNet(R)-50 architecture with mAP scores (in %) at IOUs of 0.9, 0.75, and 0.5.
![Page 142: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/142.jpg)
PlotNet: Qualitative Analysis
IOU 0.9 0.75 0.5
Models\Class
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
mAP mAP mAP
v0 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11
v1 92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77 97.74 98.24
v2 91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87 96.38 98.20
v3 91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34 96.97 98.26
v4 91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88 97.30 98.31
v5 92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19 97.70 98.18
v6 92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39 96.78 97.82
v7 92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44 97.93 98.32
142
Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9, 0.75 & 0.5.
![Page 143: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/143.jpg)
PlotNet: Qualitative Analysis
IOU 0.9 0.75 0.5
Models\Class
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
mAP mAP mAP
v0 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11
v1 92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77 97.74 98.24
v2 91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87 96.38 98.20
v3 91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34 96.97 98.26
v4 91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88 97.30 98.31
v5 92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19 97.70 98.18
v6 92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39 96.78 97.82
v7 92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44 97.93 98.32
143
Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9, 0.75 & 0.5.
![Page 144: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/144.jpg)
PlotNet: Qualitative Analysis
IOU 0.9 0.75 0.5
Models\Class
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
mAP mAP mAP
v0 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11
v1 92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77 97.74 98.24
v2 91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87 96.38 98.20
v3 91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34 96.97 98.26
v4 91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88 97.30 98.31
v5 92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19 97.70 98.18
v6 92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39 96.78 97.82
v7 92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44 97.93 98.32
144
Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9, 0.75 & 0.5.
![Page 145: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/145.jpg)
PlotNet: Qualitative Analysis
IOU 0.9 0.75 0.5
Models\Class
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
mAP mAP mAP
v0 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11
v1 92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77 97.74 98.24
v2 91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87 96.38 98.20
v3 91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34 96.97 98.26
v4 91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88 97.30 98.31
v5 92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19 97.70 98.18
v6 92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39 96.78 97.82
v7 92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44 97.93 98.32
145
Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9, 0.75 & 0.5.
![Page 146: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/146.jpg)
PlotNet: Qualitative Analysis
IOU 0.9 0.75 0.5
Models\Class
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
mAP mAP mAP
v0 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11
v1 92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77 97.74 98.24
v2 91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87 96.38 98.20
v3 91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34 96.97 98.26
v4 91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88 97.30 98.31
v5 92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19 97.70 98.18
v6 92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39 96.78 97.82
v7 92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44 97.93 98.32
146
Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9, 0.75 & 0.5.
![Page 147: Robert Bosch Centre for Data Science and Artificial Intelligence … Khapra.pdf · 2020. 9. 2. · 57 IOU = = 0.54 IOU = = 0.79 IOU = = 0.98 Image Source: Nitesh Methani, Pritha Ganguly,](https://reader035.vdocuments.us/reader035/viewer/2022071609/61475620afbe1968d379fe6c/html5/thumbnails/147.jpg)
PlotNet: Qualitative Analysis
IOU 0.9 0.75 0.5
Models\Class
bar dotline
legendlabel
legendpreview
plottitle
x-axislabels
x-axisticks
y-axislabels
y-axisticks
mAP mAP mAP
v0 91.02 31.69 97.08 81.57 99.36 96.06 85.33 82.00 90.95 83.89 97.21 98.11
v1 92.16 61.18 98.38 93.46 99.44 97.21 94.21 95.45 94.42 91.77 97.74 98.24
v2 91.79 41.86 93.74 94.64 98.29 83.11 85.69 89.32 49.36 80.87 96.38 98.20
v3 91.83 45.78 91.48 94.15 98.95 74.24 87.19 89.34 50.11 80.34 96.97 98.26
v4 91.88 61.44 96.44 95.58 99.27 97.19 90.64 97.55 87.66 90.88 97.30 98.31
v5 92.78 68.26 97.75 95.90 99.04 93.64 92.97 96.24 93.12 92.19 97.70 98.18
v6 92.74 59.12 94.87 95.58 92.26 94.46 94.12 95.36 76.85 88.39 96.78 97.82
v7 92.80 70.11 98.47 96.33 99.52 97.31 94.29 97.66 94.48 93.44 97.93 98.32
147
Table 5: Comparison of variants of PlotNet on the PlotQA dataset with mAP score(in %) at IOUs of 0.9, 0.75 & 0.5.