![Page 1: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/1.jpg)
The Future of Natural User Interaction NUI: a great new discipline in the making
Baining Guo Microsoft Research Asia
![Page 2: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/2.jpg)
Kinect from Insiders’ Perspective
• An interview with Kinect contributors
![Page 3: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/3.jpg)
![Page 4: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/4.jpg)
Kinect from Insiders’ Perspective
• Kinect: A big collaboration project – Involving Xbox division, MSR Redmond, MSR Cambridge, MSR
Silicon Valley, and MSR Asia
• Kinect :1st mass market product, 20 millions customers NUI is not science fiction
• NUI is a new engineering discipline in the making – NUI is far from where it should be; needs bigger “foundation &
pillars”
– “Foundation and pillars”are yet to be invented
![Page 5: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/5.jpg)
Core Technologies & Problems
• Kinect is all about connecting you and your avatar – without a controller
![Page 6: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/6.jpg)
Kinect, You and Your Avatar
You
Your Avatar
![Page 7: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/7.jpg)
Kinect, You and Your Avatar Tracking you • Your identity • Your facial expression
![Page 8: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/8.jpg)
Kinect, You and Your Avatar Tracking you • Your identity • Your facial expression
Controlling your avatar • Pose correction & tagging • Face animation
![Page 9: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/9.jpg)
Core Technologies
• Tracking
– identity tracking,
– facial features tracking,
– head pose tracking
– …
• Gesture control (“gesture building”)
• Digitization
![Page 10: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/10.jpg)
Tracking Track you and your movement
![Page 11: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/11.jpg)
Kinect Identity (identity tracking)
• A core technology of Kinect for robustly binding each player with his avatar
• An essential part of Kinect & used by all Kinect games as it is a Kinect game certification requirement
• See more info in the article – Tommer Leyvand, Casey Meekhof, Yi-Chen Wei, Jian Sun, and Baining Guo, “Kinect
Identity: Technology and Experience”, IEEE Computer, April 2011
– This article made it to the list of the most read articles of IEEE Computing Now
![Page 12: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/12.jpg)
Kinect Identity
• In Kinect, each skeleton a game character / a player profile
• When skeleton tracking fails & resumes or player leaves & comes back
– Which skeleton to use? A new player or an existing payer?
![Page 13: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/13.jpg)
Solution? 1 3 2
5
4
6 7
8
9
1
0 1
1
1
2 1
3
1
4 1
5 #24 #25 #77 #78 #RGB frame
![Page 14: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/14.jpg)
Body dimension
• Body tracking is unstable for recognition
1 3 2
5
4
6 7
8
9
1
0 1
1
1
2 1
3
1
4 1
5 #24 #25 #77 #78
initialized skeletons from individual frames
#RGB frame
![Page 15: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/15.jpg)
Challenges and our solution
• Challenges: robustness and performance
– instable skeleton tracking, varying lighting …
– 2ms/frame buffer for xbox game
• Our solution: fusion of multiple visual signatures
– facial, clothing, body dimension
– robust and efficient feature extraction
– efficient recognition: decision tree and NN classification
![Page 16: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/16.jpg)
![Page 17: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/17.jpg)
Tracking Head Pose (head orientation)
• Roll : (-45, 45)
• Yaw : (-60, 60)
• Pitch : (-60, 60)
• Our measurement
![Page 18: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/18.jpg)
Training Kinect to track head pose
head MoCap tracker
Training data: video and head MoCap data
![Page 19: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/19.jpg)
roll
yaw
pitch
Head MoCap Data
![Page 20: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/20.jpg)
neutral
mouth open
smile
dark & far dark & near bright & far bright & near
![Page 21: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/21.jpg)
![Page 22: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/22.jpg)
Challenge: fast head pose tracking
• Hard constraint: computing time/frame < 5 ms
– Has to be accurate too
• A classical regression problem w/ a big input feature
space (dim = 7100)
• PCA helps but not sufficient (dim = 2500)
– Accuracy cannot be compromised
![Page 23: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/23.jpg)
From image to pose – and only pose!
• Identity
• Pose
• Lighting
• Expression
![Page 24: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/24.jpg)
Manifold embedding by Multi-class LDA
• Linear Discriminant Analysis (LDA)
– Quantize pose space into discrete pose classes
– Find optimal subspace projection that maximizes between-
class variation and minimizes within-class variation (track
only pose & nothing else, dim = dozens)
|A1-A2|^2
S1^2+S2^2 2-class LDA: find subspace projection that maximizes
![Page 25: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/25.jpg)
Manifold embedding by Multi-class LDA
• Linear Discriminant Analysis (LDA)
– find optimal subspace projection that maximizes between-class variation and minimizes
within-class variation
𝑐𝑜𝑣𝑏𝑒𝑡𝑤𝑒𝑒𝑛 =1
𝐶 (𝑢𝑖 − 𝑢)(𝑢𝑖 − 𝑢)
𝑇𝐶
𝑖=1 𝑐𝑜𝑣𝑤𝑖𝑡ℎ𝑖𝑛 =
1
𝐶 𝑐𝑜𝑣𝑖𝐶
𝑖=1
𝑆 =𝑤𝑇𝑐𝑜𝑣𝑏𝑒𝑡𝑤𝑒𝑒𝑛𝑤
𝑤𝑇𝑐𝑜𝑣𝑤𝑖𝑡ℎ𝑖𝑛𝑤
(𝑐𝑜𝑣𝑤𝑖𝑡ℎ𝑖𝑛)−1𝑐𝑜𝑣𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝐶 − 1
𝑢𝑖: mean of samples in class i
𝑢: mean of all samples
pose space dim eigen-space of
• compact but keeps essential pose info
![Page 26: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/26.jpg)
Head Pose Tracking
• Shipped in Feb 2012 in Kinect SDK, NUI API
– Available to all Kinect developers worldwide
• Kinect as a “publication venue”
![Page 27: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/27.jpg)
Facial Feature Tracking
• Avatar Kinect: a Kinect service for people to chat & interact via their Xbox Live avatars in virtual chat rooms
– Chat & interact: A new digital lifestyle
• Shipped to all Kinect users in July 2011 with Kinect Fun Labs
• Collaboration w/ MSRA & MSR Redmond (Zhengyou Zhang)
![Page 28: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/28.jpg)
Avatar Kinect (video)
![Page 29: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/29.jpg)
Research Issues
Estimated shape
𝑆 = {𝑥𝑖 , 𝑦𝑖}
![Page 30: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/30.jpg)
Research Issues
• Requirements: accurate, robust, efficient
![Page 31: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/31.jpg)
Existing Best Approach – AAM (Active Appearance Model)
2
Cost Function
Greedy Optimization:
Reconstruction Error
Model generated Input Face
![Page 32: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/32.jpg)
Drawbacks of AAM-based Approaches
• Cost Function – Bad generalization on unseen
person
• High computation cost – Local minimum; sensitive to
initialization
• Parametric model – Not adaptive in the iterative
optimization
The standard theoretical framework for the past 20 year!
![Page 33: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/33.jpg)
AAM vs Explicit Shape Regression
• Cost Function – Bad generalization on unseen
person
• High computation cost
– Local minimum; sensitive to initialization
• Parametric model – Not adaptive in the iterative
optimization
• Cost Function – A repressor learned from a very
large training data
• Super fast (2-10ms) – Two level cascade and multiple
initialization
• Non-parametric model – Adaptive coarse-to-fine shape
constraint
AAM-based approaches Explicit shape regression (CVPR ’12 oral)
![Page 34: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/34.jpg)
Gesture Control Control how your avatar moves
![Page 35: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/35.jpg)
Understanding the player’s movement
From raw data to high-level knowledge
Depth map skeleton gesture
• Depth map: low-level raw data
• Skeleton: intermediate representation
• Gesture: high-level knowledge for controlling avatars
Can we define gesture using the skeleton?
• skeleton is not reliable at critical moments (arms
crossing, legs crossing, body turning sideways)
![Page 36: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/36.jpg)
Pose Correction & Tagging
Depth
Image Background
Removal Skeleton
Extraction
Skeleton
Correction
![Page 37: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/37.jpg)
Skeleton/Pose Tagging
•The corrected skeletons are tagged w/ numerical values
•The numerical values are used to drive the avatar
![Page 38: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/38.jpg)
Context-Based Pose Correction & Tagging
• Context == the activity that the user is doing
• For the given context, gather ground truth data
• manually labeled skeleton with tags
• From the ground truth data, train a random forests regressor
for automatic pose correction & tagging (in this context!)
![Page 39: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/39.jpg)
The Gesture Component of Kinect SDK
• Shipped in Feb 2012 Kinect SDK, NUI API
– Official name: Kinect gesture builder
– Available to all Kinect developers world wide
• More details in
– "Exemplar-Based Human Action Pose Correction and Tagging", W. Shen, K. Deng, X. Bai, T. Leyvand, B. Guo & Z. Tu, IEEE Computer Vision and Pattern Recognition, 2012
![Page 40: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/40.jpg)
Digitization Bring physical objects into cyber space
![Page 41: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/41.jpg)
Object Digitization
• Simple inputs – Front and back snapshots of objects
• Good 3D reconstruction results – From noisy input to smooth outputs
• Fast – Using both CPU and GPU
![Page 42: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/42.jpg)
Object Digitization • Shipped with the Kinect Fun Labs in July 2011
– The object capture lab
– Available to all Xbox Live members on Kinect
![Page 43: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/43.jpg)
![Page 44: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/44.jpg)
Major Research Issues • Dealing with noisy Kinect data: Poisson geometry processing
– Mesh editing with Poisson-based gradient field manipulation, Y Yu, K Zhou, D Xu, X Shi, H Bao, B Guo, HY Shum, ACM Siggraph 2004
– Laplacian surface editing, O Sorkine, D Cohen-Or, Y Lipman, M Alexa, C Rössl, HP Seidel, Eurographics SGP, 2004
– Poisson surface reconstruction, M Kazhdan, M Bolitho, H Hoppe, Eurographics SGP, 2006
“it preserves surface details and produces visually pleasing results by distributing errors globally through least-squares minimization”,
-- K Zhou, J Huang, J Snyder, X Liu, H Bao, B Guo & H Shum (2005)
(See “Large deformation using volumetric graph Laplacian”, Siggraph 2005)
![Page 45: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/45.jpg)
Major Research Issues
• Making it fast: data-parallel octree
– Can we build geometry octrees on the GPU?
– Data-parallel octrees for surface reconstruction, K Zhou, M Gong, X Huang, B Guo, IEEE TVCG 2011
![Page 46: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/46.jpg)
Face Digitization & Animation (“creating my avatar that looks me & moves like me”)
Temporal resolution
Spatial resolution
Image based
3D scanner
Video based
Structured lighting
Motion capture
![Page 47: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/47.jpg)
High-Fidelity Facial Animation
• Traditional motion capture
– Realistic motion details, but lacks geometry details
• Our approach: FaceMocap+
– Realistic motion details just like motion capture
– Plus geometry details as in laser scans
– Paper published in Siggraph‘11
![Page 48: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/48.jpg)
![Page 49: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/49.jpg)
![Page 50: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/50.jpg)
Face Digitization • Generate personalized 3D avatar based on Kinect input
– on-going research w/ MSRA + MSR Redmond (Zicheng Liu)
Temporal resolution
Spatial resolution
3D scanner
Image based
Our system
![Page 51: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/51.jpg)
Image-based Approach
…
Face priors
Morphable Model
Image 2D feature points
![Page 52: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/52.jpg)
Kinect-based 3D Face Modeling
Laplacian Deformation
Morphable Model
…
Face priors
![Page 53: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/53.jpg)
Comparison
Image-based Our approach Photo
![Page 54: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/54.jpg)
What about hair?
![Page 55: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/55.jpg)
![Page 56: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/56.jpg)
Our Vision
• Is to do vision, graphics & multimedia
• Making technologies disappear
![Page 57: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/57.jpg)
Our Vision
• Is to do vision, graphics & multimedia
• Making technologies disappear
![Page 58: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/58.jpg)
Our Vision
• Is to do vision, graphics & multimedia
• Making technologies disappear
![Page 59: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/59.jpg)
Our Vision
• Is to do vision, graphics & multimedia
• Make technologies disappear
![Page 60: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/60.jpg)
Future of Kinect: turning Sci-Fi into reality
![Page 61: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/61.jpg)
Beyond Kinect: Cloud + NUI
• Old era of “PC + GUI”
– PC has processing powers
– GUI allows easy access to the processing power
• New era of “Cloud + NUI”
– The cloud has knowledge & data
– NUI allows easy access to the knowledge & data
![Page 62: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/62.jpg)
Where are we today?
Kinect is the“Kitty Hawk”of NUI
![Page 63: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/63.jpg)
Emerging Research Themes
• Tracking
– identity tracking,
– facial features tracking,
– head pose tracking
– …
• Gesture control
• Digitization
![Page 64: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/64.jpg)
To Recap …
• Kinect from insiders’perspective
• Core technologies of Kinect
• Emerging research themes in NUI
• A great new engineering discipline -- huge opportunities, huge challenges
![Page 65: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/65.jpg)
Thank You!
![Page 66: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/66.jpg)
Facial signature
• Face detection
• Face alignment
• Signature extraction
• <5 ms
![Page 67: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/67.jpg)
Facial signature
• Signature is learned from data
Learning-based local
descriptor
Concatenated patch
histogram
PCA dimension
reduction
Zhimin Cao, Qi Yin, Xiaoou Tang, and Jian Sun. Face Recognition with Learning
based Descriptor. CVPR 2010.
![Page 68: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/68.jpg)
• Top #1 in LFW face recognition benchmark
• Microsoft FaceLibrary (http://toolbox/Facelib/)
Facial signature
state-of-the-art in face recognition
Implementation is adapted for Kinect
– Kinect camera
• lighting, resolution…
![Page 69: The Future of Natural User Interactiontranslectures.videolectures.net/site/normal_dl/tag=703425/icme2012… · NUI: a great new discipline in the making Baining Guo Microsoft Research](https://reader033.vdocuments.us/reader033/viewer/2022060521/604fd2583233ad1e672267c5/html5/thumbnails/69.jpg)
Algorithm: training
1. LBP feature extraction (dim = 7139)
2. PCA (dim = 2500)
3. Multi-class LDA (dim = dozens)
4. Clustering to find a small number of
exemplars (cluster centers)
• Model size: a few megabytes