enforcing constraints for human body tracking david demirdjian artificial intelligence laboratory,...
TRANSCRIPT
Enforcing Constraints for Human Body Tracking
David DemirdjianArtificial Intelligence Laboratory, MIT
WOMOT 2003
Goal Real-time articulated body tracking from
stereo accounting for constraints on pose
WOMOT 2003
Approach
Differential tracking: assuming the articulated body pose t-1 is known, estimate the pose t (or
equivalently the set of limb rigid motions i=(tii) between
poses t-1 and t) that minimizes the distance between the articulated model and the observed 3D data
tracking as a constrained optimization problem
WOMOT 2003
Approach
Differential tracking: assuming the articulated body pose t-1 is known, estimate the pose t (or
equivalently the set of limb rigid motions i=(tii) between
poses t-1 and t) that minimizes the distance between the articulated model and the observed 3D data
tracking as a constrained optimization problem– Solve unconstrained optimization problem– Project solution on constraint surface
WOMOT 2003
Projection-based approach
unconstrained optimum)
human motion manifold
WOMOT 2003
Approach
Estimate limb motions i=(tii) independently using standard multi-object tracking algorithm
Projection: find the closest body motion =(i’) to =(i) that satisfies human body constraints: – articulated constraints – other constraints: joint limit, …
WOMOT 2003
Previous work
Particle sampling: Sidenbladh & al. ECCV’00
Sminchisescu & Triggs CVPR’01
Differential tracking: Plankers & Fua ICCV’99
Jojic & al. ICCV’99
Delamarre & Faugeras ICCV’99
WOMOT 2003
Plan
Unconstrained problem Articulated constraints enforcing Other constraints Tracking results Application (Multimodal interface) Conclusion
WOMOT 2003
Multi-object tracking
Assuming the articulated body pose t-1 is known, estimate the set of limb rigid motions i=(tii) minimizes the distance between the (moved) limb and the observed 3D data
Consists in estimating limb motions i=(tii) independently:
– Estimate visible 3D mesh of each limb– Current implementation uses the ICP algorithm to
register each limb w.r.t 3D data
WOMOT 2003
Iterative Closest Point
3D registration– find the rigid transformation that maps shape St (limb model) to
shape Sr (3D data)
SrSt
),(minarg1
2*
n
i
rt SSGd
WOMOT 2003
Iterative Closest Point
Matching points• For all points in St, we search for the closest point in Sr by
computing the distance and keep the closest
SrSt
WOMOT 2003
Iterative Closest Point
Energy function minimization• Estimate the rigid transformation that minimizes the sum of
squared distances between corresponding matched points
SrSt
WOMOT 2003
Iterative Closest Point
Energy function minimization• Estimate the rigid transformation that minimizes the sum of
squared distances between corresponding matched points
SrSt
WOMOT 2003
Iterative Closest Point
Optimal rigid transformation (and uncertainty ) found by combining all the elementary displacements
WOMOT 2003
Plan
Unconstrained problem Articulated constraints enforcing Other constraints Tracking results Application (Multimodal interface) Conclusion
WOMOT 2003
ProjectionThe unconstrained optimal body motion is given by
=(1, 2 … N)With uncertainty
=(1, 2 … N)
)()( 12 TE
with =(1’, 2’ … N’) satisfying articulated constraints
Articulated constraints enforcement: find that minimizes the Mahalanobis distance:
WOMOT 2003
Articulated motion estimation
If Mij is a joint between objects i and j:
Mij joint
(Ri,ti)
(Rj,tj)obj. i
obj. j
Motion of point Mij
on limb i
Motion of point Mij
on limb j=
)(')(' ijjiji MM
WOMOT 2003
Articulated motion estimation
If Mij is a joint between objects i and j:
Mij joint
(Ri,ti)
(Rj,tj)obj. i
obj. j
Motion of point Mij
on limb i
Motion of point Mij
on limb j=
'''' jijjiiji tMRtMR
)(')(' ijjiji MM
WOMOT 2003
Articulated motion estimation
If Mij is a joint between objects i and j:
Mij joint
(Ri,ti)
(Rj,tj)obj. i
obj. j
Motion of point Mij
on limb i
Motion of point Mij
on limb j=
0'')''(][
0'']''[
')]'[(')]'[(
''''
jijiij
jiijji
jijjiiji
jijjiiji
ttM
ttM
tMItMI
tMRtMR
[.]x denotes skew-symmetric matrix
)(')(' ijjiji MM
WOMOT 2003
Articulated motion estimation
If Mij is a joint between objects i and j:
Mij joint
(Ri,ti)
(Rj,tj)obj. i
obj. j
Motion of point Mij
on limb i
Motion of point Mij
on limb j=
0'')''(][ jijiij ttM
[.]x denotes skew-symmetric matrix
)(')(' ijjiji MM
WOMOT 2003
Articulated motion estimation
If Mij is a joint between objects i and j:
Mij joint
(Ri,ti)
(Rj,tj)obj. i
obj. j
Motion of point Mij
on limb i
Motion of point Mij
on limb j=
0 ijS
)(')(' ijjiji MM
0'')''(][ jijiij ttM
=(1’, 2’ … N’)
WOMOT 2003
Articulated motion estimation
If Mij is a joint between objects i and j:
Mij joint
(Ri,ti)
(Rj,tj)obj. i
obj. j
Motion of point Mij
on limb i
Motion of point Mij
on limb j=
(Stack for all joints)0
)(')(' ijjiji MM
0 ijS
0'')''(][ jijiij ttM
=(1’, 2’ … N’)
WOMOT 2003
Articulated motion estimation
All the joint constraints can be written as a linear constraint:
0
is a linear combination of vectors in the nullspace of Therefore there exists a matrix V such that:
V
V can be estimated by SVD of
WOMOT 2003
Articulated motion estimation
)()( 12 VVE T
111 )( TT VVVVP
)()( 12 TE
unconstrainedmotion
articulatedmotion
Find minimum of E2 in nullspace of
P
…
(linear projection)
WOMOT 2003
Plan
Unconstrained problem Articulated constraints enforcing Other constraints Tracking results Application (Multimodal interface) Conclusion
WOMOT 2003
Other constraints
Constraints:– Static: Joint angle bounds, gravity law, …– Dynamic: Maximum velocity, …
Motivation:– Using more constraints to reduce local minima
and therefore increase tracking robustness– Application context can reduce tremendously
the dimension of the pose space
WOMOT 2003
Other constraints
)()( *1*2 TE
Pose constraints modeled by a (user-defined) function f, such that valid poses correspond to f()>0
ex: f()=min(g1(), g2(), … gN()) with g1() = angle(l-arm, l-forearm) – min_angle
g2() = max_angle - angle(l-arm, l-forearm)….
Constraints enforcement: find * that minimizes the Mahalanobis distance:
with * satisfying Ft-1(*)=f( *(t-1))>0
WOMOT 2003
Other constraints
)()( *1*2 TE
articulated motionarticulated constrained
motion
** V
)()(
)()(*1*2
*1*2
VVE
VVVVETT
T
with (local parameterization)
WOMOT 2003
Constrained optimization algorithm
Alternate between binary and stochastic searches
WOMOT 2003
Constrained optimization algorithm
Alternate between binary and stochastic searches
WOMOT 2003
Constrained optimization algorithm
Alternate between binary and stochastic searches
E2 = E0
WOMOT 2003
Constrained optimization algorithm
Alternate between binary and stochastic searches
E2 = E1
WOMOT 2003
Constrained optimization algorithm
Alternate between binary and stochastic searches
WOMOT 2003
TRACKING SEQUENCE
WOMOT 2003
Future work
Quantitative measurement(comparing results with tethered motion capture system)
Appearance/Shape information(learning color distribution + shape of limbs)
Motion/gesture(including dynamic constraints)
Learning human motion constraints (instead of giving them explicitly.. [ICCV’03])
WOMOT 2003
Applications Multimodal Human-Computer Interaction
(gesture + speech)
WOMOT 2003
WOMOT 2003
Conclusion
Projection-based approach for articulated body tracking– articulated constraints enforced by (linearly)
projecting unconstrained limb motion on articulated motion manifold
– other constraints enforced using a stochastic constrained optimization algorithm