poselets lubomir bourdev and jitendra malik eecs, u.c ...yjlee/teaching/ecs289h... · bourdev,...
TRANSCRIPT
![Page 1: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/1.jpg)
PoseletsLubomir Bourdev and Jitendra MalikEECS, U.C. Berkeley and Adobe Systems
PRESENTED BY : SUGEERTH MURUGESAN
![Page 2: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/2.jpg)
Motivation•Pose has:•Appearance Space – pixel values•Configuration Space – 3D coordinates parts
Appearance space is function of: •Clothing •Viewpoint •Occlusion•Illumination
![Page 3: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/3.jpg)
MotivationChallenges in computer vision: •Detection •Joint Localization •Segmentation
![Page 4: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/4.jpg)
MotivationWhat if ?
Use Divide and Conquer ◦to detect “parts” to factor complexity
◦Need for:◦Introducing Poselets◦Algorithm to select Poselets
Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose annotations."
![Page 5: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/5.jpg)
Where is the bounding box?
![Page 6: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/6.jpg)
Poselets
Poselets capture part of the pose from a given viewpoint
![Page 7: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/7.jpg)
Poselets
Examples may differ visually but have common semanticsExamples may differ visually but have common semantics
![Page 8: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/8.jpg)
How do we train a poselet for a given pose configuration?
![Page 9: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/9.jpg)
H3D dataset
Pixel level labels
3D annotation of keypoints
[2] Taylor "Reconstruction of articulated objects from point correspondences in a single uncalibrated image."
[1] Maire, Michael, et al. "Using contours to detect and localize junctions in natural images."
Azimuth angles of frontal profiles and expected arm bending
![Page 10: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/10.jpg)
H3D dataset
Conditional probability masks Appearance queries
Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose annotations."
![Page 11: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/11.jpg)
Finding correspondences at training time
Given part of a human poseHow do we find a similar pose configuration in the training set?
Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose annotations."
![Page 12: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/12.jpg)
Finding correspondences at training time
We use keypoints to annotate the joints, eyes, nose, etc. of people
Left Hip
Left Shoulder
Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose annotations."
![Page 13: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/13.jpg)
Method (Finding Candidates)Distance from example s to example r as:
Where = [x, y, z] are normalized 3D coordinates of the i-th keypointof the example s. The weight is a Gaussian with mean at the center of the patch.
The term is a penalty based on the visibility mismatch of keypoint i in the two examples.
, ( )s rh i
( )sX i( )sw i
![Page 14: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/14.jpg)
Finding correspondences at training time
We use keypoints to annotate the joints, eyes, nose, etc. of people
Residual Error
Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose annotations."
![Page 15: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/15.jpg)
Training poselet classifiers
Residual Error:
0.15 0.20 0.10 0.350.150.85
1. Given a seed patch
2. Find the closest patch for every other person
3. Sort them by residual error
4. Threshold them
5. Use them as positive training examples to train a linear SVM with HOG features
Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose annotations."
![Page 16: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/16.jpg)
Examples
Example query regions (left column) and the corresponding closest matches in configuration space generated by H3D.
Top: 3D configuration space,Bottom: Appearance space
Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose annotations."
![Page 17: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/17.jpg)
Which poselets should we train?Choose thousands of random windows, generate poselet candidates, train linear SVMs
Select a small set of poselets that are:◦ Individually effective
◦ComplementaryBourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose annotations."
![Page 18: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/18.jpg)
Object localization
Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose annotations."
![Page 19: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/19.jpg)
Experiments:Torso Detection
![Page 20: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/20.jpg)
State of the art in Torso Detection
Frontal detector of Bourdev and Brandt [2]
Pedestrian detector of Dalal and Triggs [3]
The parts-based deformable detector of Felzenszwalb, Mcallester [5]Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose annotations."
![Page 21: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/21.jpg)
ExperimentsDetecting Keypoints
Detection rate of some keypoints conditioned on true positive detection. Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose annotations."
![Page 22: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/22.jpg)
Other Uses of PoseletsObject Segmentation (CVPR 2011)◦ Predict area using “soft masks”
◦ Deformation to match image edges
◦ Extends poselets to other objects, Birdlets
Activity Recognition (CVPR 2011)◦ Recognition from a single image
◦ Use which poselets fired and to what extent to predict the activity
Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors
trained using 3d human pose annotations."
![Page 23: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/23.jpg)
Some Poselets
![Page 24: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/24.jpg)
Conclusion & Future Work
Conclusion•The authors propose a two-layer classification/ regression model for detecting people and localizing body components.
•3D annotation guides the search for good parts.
Future work•Use H3D more widely
•Using poselets for recognizing complex tasks
![Page 25: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/25.jpg)
Poselets websitehttp://eecs.berkeley.edu/~lbourdev/poselets
The set of published poselet papers
H3D data set + Matlab tools
Java3D annotation tool + video tutorial
Matlab code to detect people using poselets
Our latest trained poselets
![Page 26: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/26.jpg)
Prior workDalal and Triggs ‘05◦Learn to classify pedestrians vs. background
◦HOG + linear SVM◦Doesn’t account for variations in body pose and viewpoint
Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose annotations."
![Page 27: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/27.jpg)
Which image has the poselet?
Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose annotations."
![Page 28: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/28.jpg)
Which poselets are discriminative for gender?
Preferred by human subjects
Preferred by our system
best
best
worst
worst
[Bourdev et al., ICCV11]
![Page 29: Poselets Lubomir Bourdev and Jitendra Malik EECS, U.C ...yjlee/teaching/ecs289h... · Bourdev, Lubomir, and Jitendra Malik. "Poselets: Body part detectors trained using 3d human pose](https://reader035.vdocuments.us/reader035/viewer/2022063011/5fc4e6676af887766f0ebe44/html5/thumbnails/29.jpg)
Selecting a small set of complementary poselets