detecting actions, poses, and objects with relational
TRANSCRIPT
![Page 1: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/1.jpg)
Detecting Actions, Poses, and Objects with Relational Phraselets
by Chaitanya Desai and Deva Ramanan
Presented by: Antonia CreswellDetecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva Ramanan
Wednesday, 5 November 14
![Page 2: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/2.jpg)
Problem
• Humans interact with objects in a variety of ways
• Interaction with objects leads to occlusions
• May be many people in one image
Wednesday, 5 November 14
![Page 3: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/3.jpg)
Interact in different ways:
Deva Ramanan, University of California, Irvine
Wednesday, 5 November 14
![Page 4: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/4.jpg)
Interaction lead to occlusions
Deva Ramanan, University of California, Irvine
Wednesday, 5 November 14
![Page 5: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/5.jpg)
Many people in one image
Deva Ramanan, University of California, Irvine
Wednesday, 5 November 14
![Page 6: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/6.jpg)
Motivation
• Articulated Skeletons
• Visual Phrases
• Poselets
http://www.urbiforge.org/index.php/Modules/UKinect2
Poselets and Their Applications in High-Level Computer Vision
Recognition using Visual Phrases Ali Farhadi, Mohammad Amin Sadeghi
Wednesday, 5 November 14
![Page 7: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/7.jpg)
Key Contributions/ Technical Ideas
• Identify phraselets
• Create a model as a composite of phraselets
• Apply relational constraints between phraselets
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva RamananWednesday, 5 November 14
![Page 8: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/8.jpg)
Identify PhraseletsPosition of part
Occluded or not?
Phraselet Label
Feature for part i in image n:
Cluster these to get the phraselets labelsKey Point: Occluded and non-Occluded parts are clustered separately: They have their own
set of labels!Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva Ramanan Deva Ramanan, University of California, Irvine
Wednesday, 5 November 14
![Page 9: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/9.jpg)
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva RamananWednesday, 5 November 14
![Page 10: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/10.jpg)
Relational Model
- E is the edge (or relation) between two parts - S is the score
encodes a prior acting as a compatibility measure
template tuned for mixture t(i)
HOG feature vector
springs that spatially constrain the parts i and j
deformation vector computed from the offset of pi&pj
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva Ramanan Deva Ramanan, University of California, Irvine
Wednesday, 5 November 14
![Page 11: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/11.jpg)
Learning this modelpart i
from class: t(2)
part jfrom class: t(1)
Edge label: I(z(i)| z(j))- Maximise Score S
- Find Max weight spanning tree
Location and types for all parts in n
Linear model
Learn Thetas to minimise:
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva Ramanan
Wednesday, 5 November 14
![Page 12: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/12.jpg)
Models learned with the tree structure:
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva RamananWednesday, 5 November 14
![Page 13: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/13.jpg)
Experimental Setup & Results
• Action Detection
• Action Classification
• Pose Evaluation considering occlusion
Wednesday, 5 November 14
![Page 14: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/14.jpg)
Action Detection
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva Ramanan
Wednesday, 5 November 14
![Page 15: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/15.jpg)
False False Positives
Top False PositivesFalse False Positives due to bounding box errors
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva RamananWednesday, 5 November 14
![Page 16: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/16.jpg)
Action Detection : Precision - Recall
Compares to visual phrase as a base line
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva Ramanan
Recognition using Visual Phrases Ali Farhadi, Mohammad Amin Sadeghi
Wednesday, 5 November 14
![Page 17: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/17.jpg)
Action Classification
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva Ramanan
Compare to DPM/VP, FMP, FMP + occ
Wednesday, 5 November 14
![Page 18: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/18.jpg)
Pose Estimation
• Should report location of all parts and any that have been occluded
• Novel scheme for evaluating models
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva RamananWednesday, 5 November 14
![Page 19: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/19.jpg)
Pose Scores:
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva Ramanan
F1 scores:Penalise for labelling occluded points as visible
Combines pose estimation with aspect estimation
Wednesday, 5 November 14
![Page 20: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/20.jpg)
Percentage of correct parts
• Reports on location of all parts including occlusions
• Suggests that this model predicts location of occluded parts well
Wednesday, 5 November 14
![Page 21: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/21.jpg)
Strengths & Weaknesses
• Relation between parts
• Ability to predict the location of occluded parts
• Separating clusters for occluded and non-occluded parts
Wednesday, 5 November 14
![Page 22: Detecting Actions, Poses, and Objects with Relational](https://reader034.vdocuments.us/reader034/viewer/2022042702/6265d712ee9ee71b5b726a38/html5/thumbnails/22.jpg)
Questions
Wednesday, 5 November 14