1 MSRIUniversity of California Berkeley
Recovering Human Body Configurations using Pairwise Constraints between PartsRecovering Human Body Configurations using Pairwise Constraints between Parts
Xiaofeng Ren, Alex Berg, Jitendra MalikXiaofeng Ren, Alex Berg, Jitendra Malik
2 MSRIUniversity of California Berkeley
Finding PeopleFinding People
Challenges: Pose, Clothing, Lighting, Clutter, …
3 MSRIUniversity of California Berkeley
Previous WorkPrevious Work• Related Domains
– Tracking People– Detecting Pedestrians– ... …
• Localizing Human Figures– Exemplar-based:
[Toyama & Blake 01], [Mori & Malik 02], [Sullivan & Carlsson 02], [Shakhnarovich, Viola & Darrell 03], …
– Part-based:[Felzenswalb & Huttenlocher 00], [Ioffe & Forsyth 01], [Song, Goncalves & Perona 03], [Mori, Ren, Efros & Malik 04], …
– … …
• Related Domains– Tracking People– Detecting Pedestrians– ... …
• Localizing Human Figures– Exemplar-based:
[Toyama & Blake 01], [Mori & Malik 02], [Sullivan & Carlsson 02], [Shakhnarovich, Viola & Darrell 03], …
– Part-based:[Felzenswalb & Huttenlocher 00], [Ioffe & Forsyth 01], [Song, Goncalves & Perona 03], [Mori, Ren, Efros & Malik 04], …
– … …
4 MSRIUniversity of California Berkeley
Beyond “Trees”Beyond “Trees”
• A hard problem! More information is needed.
• Important cues that are NOT in the tree model:– Symmetry of clothing/color
– “V-shape” formed by the upper legs
– Distance/smooth connection between arms and legs
– ……
• A hard problem! More information is needed.
• Important cues that are NOT in the tree model:– Symmetry of clothing/color
– “V-shape” formed by the upper legs
– Distance/smooth connection between arms and legs
– ……
?
5 MSRIUniversity of California Berkeley
Our ApproachOur Approach• Preprocessing with Constrained Delaunay Triangulation
• Detecting Candidate Parts from Bottom-up
• Learning Pairwise Constraints between Parts
• Assembling Parts by Integer Quadratic Programming (IQP)
• Preprocessing with Constrained Delaunay Triangulation
• Detecting Candidate Parts from Bottom-up
• Learning Pairwise Constraints between Parts
• Assembling Parts by Integer Quadratic Programming (IQP)
6 MSRIUniversity of California Berkeley
Constrained Delaunay TriangulationConstrained Delaunay Triangulation
– Detect edges with Pb (Probability of Boundary)
– Trace contours with Canny’s hysteresis
– Recursively split contours into piecewise straight lines
– Complete the partial graph with Constrained Delaunay Triangulation
– Detect edges with Pb (Probability of Boundary)
– Trace contours with Canny’s hysteresis
– Recursively split contours into piecewise straight lines
– Complete the partial graph with Constrained Delaunay Triangulation
7 MSRIUniversity of California Berkeley
Detecting Parts using ParallelismDetecting Parts using Parallelism
(L1,1)
(L2,2)T
N
• Candidate parts as parallel line segments (Ebenbreite)
• (Scale-invariant) Features for parallelism:|Pb1+Pb2|/2, |1-2|, |L1-L2|/|L1+L2|, |(C1-C2)T|/|L1+L2|, |(C1-C2)N|/|L1+L2|
• Logistic Classifier
• Candidate parts as parallel line segments (Ebenbreite)
• (Scale-invariant) Features for parallelism:|Pb1+Pb2|/2, |1-2|, |L1-L2|/|L1+L2|, |(C1-C2)T|/|L1+L2|, |(C1-C2)N|/|L1+L2|
• Logistic Classifier
C1
C2
8 MSRIUniversity of California Berkeley
Pairwise Constraints between PartsPairwise Constraints between Parts
• Scale (width) consistency– Use anthropometric data as groundtruth
• Symmetry of appearance (color)
• Orientation consistency
• Connectivity– Short distance between adjacent parts
– “Smooth” connection between non-adjacent parts• short “gaps” on shortest path (on CDT graph)
• small maximum angle on the shortest path
• few T-junctions/turns on the shortest path
• Scale (width) consistency– Use anthropometric data as groundtruth
• Symmetry of appearance (color)
• Orientation consistency
• Connectivity– Short distance between adjacent parts
– “Smooth” connection between non-adjacent parts• short “gaps” on shortest path (on CDT graph)
• small maximum angle on the shortest path
• few T-junctions/turns on the shortest path
C1
C2
9 MSRIUniversity of California Berkeley
Learning Pairwise ConstraintsLearning Pairwise Constraints
15 hand-labeled images from a skating sequence
Empirical distributions of some pairwise features
For simplicity, assume all features are Gaussian (future work here as they are clearly non-Gaussian)
10 MSRIUniversity of California Berkeley
Assembling Parts as AssignmentAssembling Parts as Assignment
Candidates {Ci} Parts {Lj}
(Lj1,Ci1=(Lj1))
(Lj2,Ci2=(Lj2))
Cost for a partial assignment {(Lj1,Ci1), (Lj2,Ci2)}:
assignment
2
)22)(11( 2,1
2,1)22(),11(
kijij jj
k
jjk
ijijkf
H
11 MSRIUniversity of California Berkeley
Assignment by IQPAssignment by IQP• Suppose there are m parts and n candidates, the optimal
assignment minimizes a quadratic function• Suppose there are m parts and n candidates, the optimal
assignment minimizes a quadratic function
Q(x)=xTHxQ(x)=xTHxwhere x is a mn1 indicator vector and H is of size mnmn.where x is a mn1 indicator vector and H is of size mnmn.
• This is a well-formulated Integer Quadratic Programming (IQP) problem and has efficient approximate solutions.
• We choose an approximation scheme which solves mn linear programs followed by gradient descent.
• The approximate scheme produces a ranked list of torso candidates. We consider the top 5 torso candidates and solve the corresponding 5 IQP problems.
• We have m=9 and n~150; the total time is less than a minute.
• This is a well-formulated Integer Quadratic Programming (IQP) problem and has efficient approximate solutions.
• We choose an approximation scheme which solves mn linear programs followed by gradient descent.
• The approximate scheme produces a ranked list of torso candidates. We consider the top 5 torso candidates and solve the corresponding 5 IQP problems.
• We have m=9 and n~150; the total time is less than a minute.
12 MSRIUniversity of California Berkeley
13 MSRIUniversity of California Berkeley
14 MSRIUniversity of California Berkeley
15 MSRIUniversity of California Berkeley
ConclusionConclusion
• To find people under general conditions, we need to go beyond the traditional tree-based model;
• Most important constraints for the human body are between pairs of body parts;
• Pairwise constraints may be learned from a small set of training examples;
• Integer Quadratic Programming (IQP) efficiently finds optimal configurations under pairwise constraints.
• To find people under general conditions, we need to go beyond the traditional tree-based model;
• Most important constraints for the human body are between pairs of body parts;
• Pairwise constraints may be learned from a small set of training examples;
• Integer Quadratic Programming (IQP) efficiently finds optimal configurations under pairwise constraints.
16 MSRIUniversity of California Berkeley