![Page 1: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/1.jpg)
1
Hands and face tracking for VR applications
Adviser: Chih-Hung Lin
Date:2010/12/14
Speaker: Chin He Hsu
Javier Varona, Jose’ M. Buades, Francisco J. Perales Unidad de Gra´ficos y Visio´n por Ordenador, Dept. de matematiques i Informatica, Universitat de les Illes Balears (UIB), crta. Valldemossa km. 7,5, 07122 Palma de Mallorca, Spain
![Page 2: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/2.jpg)
2
Outline1.Introduction
2. Hands and face tracking algorithm
3.Visualization using H-Anim
4.Conclusion and future work
![Page 3: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/3.jpg)
3
1.Introduction
• In order to allow a user to navigate in a 3D-space
![Page 4: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/4.jpg)
4
Interactive 3D-space
![Page 5: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/5.jpg)
5
• system must detect a new user– entering into the system’s environment– analyse him to set parameters– tracking interesting regions
![Page 6: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/6.jpg)
6
2. Hands and face tracking algorithm
• tracking problem lies in identifying both hands and face in each image – detect skin-colour pixels– data association algorithm
![Page 7: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/7.jpg)
7
2.1. Skin-colour segmentation module
• skin-colour detection– necessary to model the actor’s skin-colour in a pre
vious step
![Page 8: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/8.jpg)
8
skin-colour sample
• transform these pixels from the RGB-space to HSL-space– hue and saturation values contain the chroma infor
mation
• two main problems– human skin hue values are near the red colour– saturation values are near 0
![Page 9: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/9.jpg)
9
![Page 10: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/10.jpg)
10
skin-colour distribution
•
• Gaussian model
•
•
1{ ,..., ,..., }, ( , )i n i i iX x x x x h s
1
1 n
ii
x xn
1
1( ), ( ) '
n
i ii
x x x xn
1
2
1 1( is skin) exp( ( )| |( )' )
2(2 ) | |p x x x x x
![Page 11: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/11.jpg)
11
![Page 12: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/12.jpg)
12
Contours of skin-colour blobs after the connected components process
![Page 13: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/13.jpg)
13
2.2. Data association module
•
s (p , w , )l l l l
p ( , ):position in the 2D image
w ( , ) : size of the limb in pixels
: angle in the 2D image plane
x yp p
w h
![Page 14: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/14.jpg)
14
Next linear scheme of prediction
•
• that an extreme limb will maintain the same velocity
p( ) p( ) p( 1)
p( ) p( ) p( 1)
t t t
t t t
![Page 15: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/15.jpg)
15
Set of hypothesis
•
•
{h }, 3,
Where
h (p , w , )
l
l l ll
H l
1{ ,..., ,..., }, : blob with labeli M iB b b b b
![Page 16: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/16.jpg)
16
• define an approximation to the distance from the x image pixel to the hypothesis h
• t=x p
n=R t',
where
cos sinR=
sin cos
![Page 17: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/17.jpg)
17
calculating the angle
• Normalized image pixel and the hypothesis centre
•
atan( / )x yn n
c ( , ),crossing point
cos
sin
x y
x
y
c c
c w
c h
![Page 18: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/18.jpg)
18
![Page 19: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/19.jpg)
19
• distance between an image pixel and a hypothesis
•
• if d( x ,h)<=0 , then x is inside the hypothesis h ,if d( x
,h)>0 , then x is outside the hypothesis h
(x,h) || n || || c ||d
![Page 20: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/20.jpg)
20
• a blob with empty intersection with all hypotheses
• a pixel x of a blob is inside a limb hypothesis
x , min{ (x,h)} 0h H
b d
x , x= iff { (x,h)} 0B l d
![Page 21: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/21.jpg)
21
Occlusion case solved using multiple labelling
![Page 22: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/22.jpg)
22
![Page 23: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/23.jpg)
23
2.3. 3D-point reconstruction
![Page 24: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/24.jpg)
24
Complete procedure: color segmentation, data association and 3D reconstruction
![Page 25: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/25.jpg)
25
3. Visualization using H-Anim
• H-Anim (humanoid animation)
• we use the H-Anim standard, this way we can collaborate with standard VRML (Virtual Reality Modeling Language ) models
![Page 26: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/26.jpg)
26
3D position
![Page 27: 1 Hands and face tracking for VR applications Adviser: Chih-Hung Lin Date:2010/12/14 Speaker: Chin He Hsu Javier Varona, Jose’ M. Buades, Francisco J](https://reader035.vdocuments.us/reader035/viewer/2022062519/5697c01e1a28abf838cd0d61/html5/thumbnails/27.jpg)
27
4. Conclusion and future work
• proposed a new system
• human–computer interaction
• future work