watching unlabeled video helps learn new human actions...
TRANSCRIPT
Reading
Walking
Riding horse
Phoning
Running
Playing instrument Taking photo Using computer
Riding bike
Watching Unlabeled Video Helps Learn New Human Actions from Very Few Labeled Snapshots
Chao-Yeh Chen and Kristen Grauman The University of Texas at Austin
Problem Goal: learn actions from static images
Approach
Related Work
Represent body pose
Results
Recognizing activity in images
Conclusions
- Learn actions with discriminative pose and appearance features. e.g. [Maji et al. 2011, Yang et al. 2010, Yao et al. 2010, Delaitre et al. 2011] - Expand training data by mirroring images and videos. e.g. [Papageorgiou et al. 2000, Wang et al. 2009] - Synthesize images for action recognition and pose estimation. e.g. [Matikainen et al. 2011, Shakhnarovich et al. 2003, Grauman et al. 2003, Shotton et al. 2011,] - Ours: expanding the training set for “free” via pose dynamics learned from unlabeled data.
Representing body pose
Expand snapshots by pose dynamics learned from videos
Problems of static snapshots: - May have only few training examples for some actions. - Often limited to “canonical” instances of the action.
Hollywood Human Actions (unlabeled)
PASCAL VOC 2010 (11 verbs) Image
Stanford 40 Actions (10 selected verbs) Image
Recognizing activity in videos
- Augment training data without additional labeling cost by leveraging unlabeled video. - Simple but effective exemplar/manifold extrapolation strategies. - Significant advantage when labeled training examples are sparse. - Domain adaptation connects real and generated examples.
Unlabeled video data
Synthesize pose examples:
+ Few canonical snapshots Unlabeled video pool (generic human action)
Given training snapshot Infer pose before Infer pose after
+
Detected poselets PAV
P1
P2
P3 … P1 P2 P3
Assumptions: - Videos cover the space of human pose dynamics. - No action labels are given. - People are detectable and trackable.
Training snapshot Matched frame
Find
Nearby synthetic poses
Generate
1) Example based strategy
2) Manifold based strategy
- Poselet activation vector (PAV) by [Maji et al. 2011] - Each poselet captures part of the pose from a given viewpoint - Robust to occlusion and cluttered background
Poselet feature
Unlabeled video pool Training image
before after
t t+T t-T
Locally Linear Embedding [Roweis and Saul 2000]
Similarity function for temporal nearness and pose similarity:
Pose
Training image
Frames from video (source)
Static snapshots (target)
+
+
- -
-
-
- -
-
-
- Real
-
Overview Synthesize pose examples Few static snapshots
Unlabeled video data
Train with real & synthetic poses
Temporal
Train with real & synthetic poses
+
+
- -
-
-
- -
-
-
- +
+ +
+
Real Synthetic poses
-
Domain adaptation by [Daume III 2007]
Pose feature space Pose feature space
PASCAL(# train = 15) Stanford 10(# train = 17)
0.1
0.2
0.3
0.4
Accu
racy
(mAP
)
PASCAL Stanford 10 # of training images # of training images
15
0.1
0.2
0.3
0.1
0.2
17 20 30 37 50 20 25 33 40 50
Accu
racy
(mAP
)
Pose diversity among training images
Our
acc
urac
y ga
in
Most benefit when lack pose diversity Synthetic after Real trainings
Synthetic feature examples
Accu
racy
0.2
0.4
0.6
- Substantial gains when recognizing actions in video, since our method infers intermediate poses not covered in original snapshots.
Training Testing
Recognize action Our idea
Let the system: - Watch videos to learn how human poses change over time. - Infer nearby poses to expand the sparse training snapshots.
Synthetic before
Accuracy of video activity recognition on 78 testing videos from HMDB51+ASLAN+UCF data.
Generate Generate