real time human posture detection with multiple depth sensors
TRANSCRIPT
![Page 1: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/1.jpg)
Contexte - IntroductionDOCTORAT DE L’UNIVERSITÉ DE TOULOUSE
Université Toulouse III - Paul Sabatier Systèmes embarqués et robotique
Real time human posture detection withmultiple depth sensors
JURY
Paul CHECCHIN Rapporteur
Alberto IZAGUIRRE Rapporteur
Mohamed AKIL Examinateur
Michel DEVY Examinateur
Frédéric LERASLE Directeur de thèse
Jean-Louis BOIZARD Directeur de thèse
Groupe RAP - Groupe N2IS
Wassim FILALI 07 Novembre 2014
![Page 2: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/2.jpg)
Theses background
Human posture recognition
2
Data acquisition
Learning
Real time reconstruction
Hardware integration
evaluations
Multiple depth sensors
Body parts detection
![Page 3: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/3.jpg)
Plan
![Page 4: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/4.jpg)
Kinect2Kinect
Mono sensor RGB-D multi sensor RGB-D
4
Mono sensor RGB
Multi sensor RGB
Introduction - Historic
![Page 5: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/5.jpg)
Depth sensor technology
Optical Diffractive Element
active RGB-D camera
Primesense - Patent
5
![Page 6: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/6.jpg)
Context - Application
Video games Videosurveillance(Health/Office)
6
![Page 7: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/7.jpg)
mono/multisensors RGB Approches
Humain model[Sundaresan et al. 2005]
Model
• Geometrical shapes adjustement
• Full model adjustment
Apparence Methods
• Images projection
• Adjusting the posture
3D Reconstruction Methods
• Voxellisation
• 3D Reconstruction
[Sigal et al. 2004]
Deformable surface[Li et al. 2011]
7
![Page 8: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/8.jpg)
mono RGB-D Approches : Advanatges and Disadvantages
Resolution
Random errorFor depth estimation
Scale
• Compensated by processingto avoid overlearning
Orientation
• Relative to the sensor
• Has Impact on learning
Auto occultations
• No solution
Précision
• Limits the field of view
[Shotton et al 2011a]
[Koshelham et al 2012]
8
2.5 D Descriptor
![Page 9: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/9.jpg)
multi Kinects Approches
[Zhang et al. 2012 ]
[Berger et al. 2011 ]
Particle filtering
Model adjustment
9
•No many examples of multi RGB-D in literature•No learning Approches
![Page 10: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/10.jpg)
multi RGB-D Approches – Advantages and disatvantages
Advantages Disadvantages Avoid interferences
Temporal multiplexing
Vibration
Correction
[Maimone et al. 2012]
10
![Page 11: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/11.jpg)
Our work on the Algorithmic
Our contributions
3D Descriptor for body
parts labeling
Free parameters
Database
Hardware architecture
New descriptor
Investigations on their
influence
Learning
Evaluations
Plateform
Example
11
![Page 12: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/12.jpg)
Plan
![Page 13: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/13.jpg)
Mocap in LAAS
Nombre de caméras Hawk 4
Résolution Hawk 640 x 480
Nombre de caméras Eagle 6
Résolution des caméras Eagle 2352 x 1728
Fréquence 200
13
![Page 14: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/14.jpg)
MOCAP system Operation
Temporal synchronisation
1) Chess for image calibration
2) Active camera
3) MOCAP
4) MOCAP calibration square
14
![Page 15: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/15.jpg)
Database - Recorded SequencesNSC13 IRSS35
Color views 3 3
Depth views 3 3
MOCAP cameras 10 10
MOCAP markers 13 35
Frequency 5 images / s 20 images / s
Nb sequences 5 8
Total Nb Postures 1 951 21 569
Sequences M2,
M3,
M4,
M5,
M6
Posture en T, mouvements bras
jambes, marche, course, saut,
pompes, break dance, natation
(bras), accroupis, chute arrière,
chute avant, équilibre, ping-
pong, volley ball, haltérophilie,
Tennis
C1, C2, C3,
C4, C5, C6,
C7, C8, C9
Posture en T, mouvements bras jambes genoux,
accroupis, bascule, haltérophilie, tennis, volley ball, ping-
pong, natation (bras), pétanque, lancement de poids,
volley ball, Pétanque, marche, course, assis debout, assis
par terre, saut, équilibre, étirement, boxe, bowling,
danse, chute avant, chute arrière, conduite, déplacer
chaise, s’asseoir, balayer assis, déplacer meuble, bouger
et filmer, jouer avec des balles, karaté, échauffement,
saut à la corde
Evaluation criteria15
p p
![Page 16: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/16.jpg)
Recorded sequences - Illustrations
MOCAP
Depth
Intermediate body parts
Central body parts(defined by MOCAP)
Centers of body parts
Application
16
![Page 17: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/17.jpg)
Plan
![Page 18: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/18.jpg)
Our approch
18
![Page 19: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/19.jpg)
Our approch (BPR) vs. [Shotton et al. 2011]
Segm
enta
tio
n
Ran
do
m f
ore
st 2
.5D
Mea
n s
hif
t 2
DM
ean
sh
ift
3D
Real dataset MOCAP
Sythetic dataset for learning
Ran
do
m f
ore
st 3
D
Free parameters study
19
Vo
xelli
sati
on
![Page 20: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/20.jpg)
Our 3D descriptor
XY
Z
(X1,y1,z1)
(X2,y2,z2)
(X3,y3,z3)
(X4,y4,z4)
(X5,y5,z5)
(0,0,0,1,1)
(1,0,1,0,1)
1 Postur 7 0 K Voxels
T2(X2,y2,z2)
T3(x3,y3,z3)
T4(X4,y4,z4)
T5(x5,y5,z5)
T1(x1,y1,z1) Crossing the decision tree
20
![Page 21: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/21.jpg)
Decision Tree generation
T2(X2,y2,z2)
T3(x3,y3,z3)
T4(X4,y4,z4)
T5(x5,y5,z5)
T1(x1,y1,z1)
Φ Ensemble de vecteurs candidats
75M, 90K
αS0
21
Descripteurs tirés
![Page 22: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/22.jpg)
Decision forest
x log(x)
Entropy
Information gain
22
Trees Forest
Ponderation
Vote
![Page 23: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/23.jpg)
Plan
![Page 24: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/24.jpg)
Descriptors size
0.453
0.666
0.7680.800 0.786 0.786
0.77755.1%
68.1%
73.1% 74.2% 73.3% 72.1% 71.3%
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
0.400
0.450
0.500
0.550
0.600
0.650
0.700
0.750
0.800
0.850
0.900
0.1 0.2 0.4 0.7 1 1.5 2
clas
sifi
cati
on
%
me
anA
vera
geP
reci
sio
n
Valeur maximale de la norme des Vecteurs (m)
Taille de la fenêtre des vecteurs descripteurs - UniNorm
mAP
Classif
24
![Page 25: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/25.jpg)
Number of Trees
0.792
0.836
0.902
73.5%
88.3%
70.0%
75.0%
80.0%
85.0%
90.0%
95.0%
0.700
0.750
0.800
0.850
0.900
0.950
1 2 3 4 5 7 9 12 16 20
Cla
ssif
icat
ion
me
anA
vera
geP
reci
sio
n
Nombre d'Arbres (N)
Nombre d'arbres (N)
mAP
Classif
25
![Page 26: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/26.jpg)
Quantitative Evaluations
0,875
0,39
0,161
0,159
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
<0,01 <0,02 <0,03 <0,04 <0,05 <0,10 <0,15 <0,20 <0,30 <0,50
Me
an a
vera
gep
reci
sio
n
Seuil du calcul du "Mean average precision" en (m)
Comparaison BPR vs. OpenNI(Séquence : IRSS35-C3)
BPR
ONI0
ONI1
ONI2
26
![Page 27: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/27.jpg)
Qualitative Evaluations
27
![Page 28: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/28.jpg)
Plan
![Page 29: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/29.jpg)
Our work on the Hardware level
Analysis of requirements
Architectural exploration
GPU
FPGA
Comparative evaluation
Conclusion
Functional alaysis
CPU
GPU
FPGA
CPU
29
Solutions catalogue
![Page 30: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/30.jpg)
Functionnal analysis – Modelisation SysML
30
640x480x16bit
Box : 500K Voxels100K Full Voxels
1000 postures25M VoxelsTree of 700K nodes
Voxellisation
![Page 31: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/31.jpg)
Hardware solution catalog
PC
GP GPU
Embedded
Processors
Servers
microcontrollers
ARM
Dédiés
DSP
FPGA
ASIC
Cloud
PIC12F/ 8bits / 30MHz / 2mW / 1$
i7-5960X / 8Cores / 3.5GHz / 140W / 1000$
Tesla K40 / 2880 Cores / 235W / 5500$
100x(16 Cores/ 104GB) => $140/h
Virtex-7 / 2M LC / 6.8 BT/20-40W/$17K-$40K
31
![Page 32: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/32.jpg)
Architectures evaluation on the Background detection function
- principle
32
Image Background
![Page 33: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/33.jpg)
CPU - Plateform
Xion Pro Live
ServerHP Z800
• Display• Calibration
Capture « multi thread »Background detection3D Geometry
•Cameras•Rays•Voxellisation,…
Decision forest
bpr
Capture platform
Benefits
Algorithms evaluation platform
ASIC PS-1080
33
Performance - 10 to 30 ms
Learning time : 1h to 10h
Prediction time of one full posture 70 ms
![Page 34: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/34.jpg)
GPU – Background detection
Relatively quick handling
Parallelisation / Acceleration x30
DisatvantagesAvantages
High power consumption
CPU dependency
Memory copy Host/GPU
34
Performance - 1 to 2 ms
![Page 35: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/35.jpg)
FPGA – Components
Demosaicing Line Fifo
Start of PacketEnd of Packet
Generation
I2C Control
Frame Writer
Fifo
Counter
@Data
Frame Reader
Fifo
Counter
@Data
Memory write Memory read
Pixel Fetcher
Fifo
Data OutData In = @Fifo
Memory read
Reusable components library
Benefits Distorsion correction Rotation
Images fusionHomography
35
![Page 36: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/36.jpg)
FPGA – Background detection
Hardware blosck for the background de tection
Optimised model
36
Image
fond
Image fond
![Page 37: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/37.jpg)
FPGA – Integration in the SOPC
Ressource Usage Usage %
Logic elements 7 619 11%
Total logic
registres5 218 8%
Total LAB 630 15%
Total Internal
memory usage
(bit)
739 840 64%
Total memory
bloc usage188 75%
PLLs 2 50%
Global clocks 16 100%
37
Performance - 3 ms
Altera Cyclone IV 115K
![Page 38: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/38.jpg)
Plan
![Page 39: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/39.jpg)
Architectures Comparision
CPU GPU FPGA
Runtime - - Xeon One Thread
10 ms to 30 ms
Quadro FX48001 to 2 ms
Altera Cyclone IV3 ms
Details - Depends on the number of pixels to
process
4 ms for 4 channles Time to read the image from the memory. Can be
concatenated with other functions.
Avantages •Flexibility•Development platform
•Average learning curve •Highly parallel architecture•Reduced processing time•Reduced consumption
Disadvantages •Processing time•Processing / power
•High consumption•CPU dependency•Bottlenecks
•Long learning curve•Important development time•Limited precision processing (fixed/floating point)
39
![Page 40: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/40.jpg)
Repartition
Fonction
Cap
ture
De
pth
pro
cess
ing
Bac
kgro
un
d
de
tect
ion
Blo
bs
sele
ctio
n
Vo
xelli
sati
on
Lab
elin
g
Me
an s
hif
t
Solution Ressource
Console
Kinect – Sensor
Kinect – PS1080
Console – Processor
Console – GPU
PC
Xtion – Sensor
Xtion – PS1080
Processor
GPU
FPGA
External Sensor
Specific Module
Soft-core
40
![Page 41: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/41.jpg)
Plan
![Page 42: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/42.jpg)
Conclusions
42
![Page 43: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/43.jpg)
Perspectives
Temporal filteringMulti Kinect : fusion of reconstructions
Synthetic datasetEnrichir la base de
données
Learning algorithm parallelisation
Servers/ Cloud / GPU Learn bigger database
Enhance labeling quality
Hardware integration
Integrate all functionalities
Prototype compact à faible consommation
Mono Kinect : pixels labeling
Fall detection
Human activities recognition
Human machine interaction
43
![Page 44: Real Time Human Posture Detection with Multiple Depth Sensors](https://reader035.vdocuments.us/reader035/viewer/2022062419/55a23d0a1a28ab1f6e8b45a0/html5/thumbnails/44.jpg)
Thanks