![Page 1: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/1.jpg)
Zero-shot Task TransferVineeth N Balasubramanian
Dept of Computer Science & EngineeringIndian Institute of Technology, Hyderabad
(Joint work with Arghya Pal, PhD student)
CVPR 2019 (Oral)
![Page 2: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/2.jpg)
Our Group’s Research
![Page 3: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/3.jpg)
Grad-CAM++: Generalized Visual Explanations
Chattopadhyay, Sarkar, Howlader, Balasubramanian, WACV 2018
![Page 4: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/4.jpg)
Grad-CAM++: Generalized Visual Explanations
Chattopadhyay, Sarkar, Howlader, Balasubramanian, WACV 2018
![Page 5: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/5.jpg)
Grad-CAM++: Generalized Visual Explanations
Chattopadhyay, Sarkar, Howlader, Balasubramanian, WACV 2018
![Page 6: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/6.jpg)
Causal NN Attributions
Chattopadhyay, Manupriya, Sarkar, Balasubramanian, arXiv 2019
Neural network as a SCM
![Page 7: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/7.jpg)
Causal NN Attributions
Chattopadhyay, Manupriya, Sarkar, Balasubramanian, arXiv 2019
![Page 8: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/8.jpg)
Causal NN Attributions
Chattopadhyay, Manupriya, Sarkar, Balasubramanian, arXiv 2019
![Page 9: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/9.jpg)
Zero-shot
Task Transfer
Zero-shot Task Transfer
![Page 10: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/10.jpg)
Tasks
❖ Vision tasks:■
■ Object recognition■ Depth■ Edge detection■ Pose estimation■ ...
Zamir et al., CVPR 2018
![Page 11: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/11.jpg)
Tasks
❖ Relation among vision tasks
Zamir et al., CVPR 2018
![Page 12: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/12.jpg)
Tasks
❖ Taskonomy CVPR 2018 (Best Paper)
➢ 26 Vision tasks
➢ Sampled set of tasks and not an exhaustive list
Zamir et al., CVPR 2018
![Page 13: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/13.jpg)
TasksVision tasks are often
related to each other. How to leverage?
Key Takeaway
![Page 14: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/14.jpg)
Zero-shot
Task Transfer
Zero-shot Task Transfer
![Page 15: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/15.jpg)
Zero-shot Classification: A Review
❖ Object recognition for a set of categories for which we have no
training examples
➢ 𝓨 = {y1, y2, … , ym} classes with training samples
➢ 𝓩 = {z1, z2, … , zn} classes with no training samples
➢ Learn a classification model: H : 𝓧 → (𝓩 union 𝓨)
![Page 16: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/16.jpg)
Zero-shot Classification: A Review
❖ For each class z ϵ 𝓩 and y ϵ 𝓨: ➢ attribute representations az , ay ϵ 𝓐
are available
![Page 17: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/17.jpg)
TasksVision tasks are often related to each other
Zero-shot classificationIf relation exists among classes,
new classes can be detected based on attribute representationwithout the need for a new training phase / ground truth
Key Takeaway
![Page 18: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/18.jpg)
Zero-shot Task Transfer: Motivation
● Vision tasks:○ Expensive ○ May require special sensors○ Lesser amounts of labeled data leads to poorly performing
models
Pal, Balasubramanian, Zero-shot Task Transfer, CVPR 2019
zero-shot classification → zero-shot task transfer
![Page 19: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/19.jpg)
Zero-shot Task Transfer ● Consider K tasks, i.e. 𝓣 = {𝓣1, 𝓣2, … , 𝓣K}
● Model parameters lie on a meta-manifold ℳθ
● On meta manifold; Task 𝓣 is equivalent to model parameter θ
Pal, Balasubramanian, Zero-shot Task Transfer, CVPR 2019
![Page 20: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/20.jpg)
Zero-shot Task Transfer
● Ground truth available for first m tasks○ 𝓣known = {𝓣1, 𝓣2, … , 𝓣m}○ Corresponding model parameters, {θ 𝓣 i : i = 1, … , m}, on meta
manifold ℳ known
● No knowledge of ground truth for the zero-shot tasks○ 𝓣zero = {𝓣(m+1), 𝓣(m+2), … , 𝓣K}
Pal, Balasubramanian, Zero-shot Task Transfer, CVPR 2019
![Page 21: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/21.jpg)
Zero-shot Task Transfer: Idea
○ Learn a meta-learning function Fw (·)○ Fw (·) regresses unknown zero-shot model parameters from
known model parameters
Pal, Balasubramanian, Zero-shot Task Transfer, CVPR 2019
![Page 22: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/22.jpg)
Task Transfer Net (TTNet)
Pal, Balasubramanian, Zero-shot Task Transfer, CVPR 2019
![Page 23: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/23.jpg)
Task Correlation Matrix
![Page 24: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/24.jpg)
More on Task Correlation
![Page 25: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/25.jpg)
Task Correlation Matrix● We get task correlation matrix from 30
annotators
● Annotators are asked to give task correlation label on a scale of {+3, +2, +1, 0, −1}○ +3 denotes self relation○ +2 describes strong relation○ +1 implies weak relation○ 0 to mention abstain○ −1 to denote no relation between two tasks
Note:Our framework is not limited to crowdsourced task correlation. Any other method to compute task correlation will work
![Page 26: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/26.jpg)
Results - Surface Normal EstimationTTNet6
Source Tasks: Autoencoding, Scene Class, 3D key point, Reshading, Vanishing Pt, ColorizationZero-Shot Task: Surface Normal
![Page 27: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/27.jpg)
Results - Depth Estimation
Ref: Arghya Pal, Vineeth N Balasubramanian, Zero-shot Task Transfer, CVPR 2019 Oral
TTNet6 (same model, only change in gamma values)
Source Tasks: Same as previousZero-Shot Task: Depth Estimation
![Page 28: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/28.jpg)
Results - Camera Pose Estimation
TTNet6 (same model, only change in gamma values)
Source Tasks: Same as previousZero-Shot Task: Camera Pose Estimation
![Page 29: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/29.jpg)
Why better than Supervised Learning?
![Page 30: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/30.jpg)
Zero shot to known task transfer
![Page 31: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/31.jpg)
How many source tasks do we need?
![Page 32: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/32.jpg)
Different Choices of Zero-shot tasks
Ref: Arghya Pal, Vineeth N Balasubramanian, Zero-shot Task Transfer, CVPR 2019 Oral
![Page 33: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/33.jpg)
Performance on Other Datasets:
Ref: Arghya Pal, Vineeth N Balasubramanian, Zero-shot Task Transfer, CVPR 2019 Oral
Object detection on COCO-Stuff dataset
![Page 34: Indian Institute of Technology, Hyderabad Dept of Computer ... · Indian Institute of Technology, Hyderabad (Joint work with Arghya Pal, PhD student) CVPR 2019 (Oral) Our Group’s](https://reader034.vdocuments.us/reader034/viewer/2022052013/602a092d8f3b27325131acba/html5/thumbnails/34.jpg)
Thank you!