real-time facial motion capture and animation on mobile€¦ · real-time facial motion capture and...
TRANSCRIPT
![Page 1: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/1.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Real-Time Facial Motion Capture & Animation on Mobile
Emiliano Gambaretto , Director of Research
Stefano Corazza, CEO
![Page 2: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/2.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Character Animation and the Power of Storytelling
![Page 3: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/3.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Over 100m people go watch animated movies/year
![Page 4: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/4.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Animating Characters in 3D is hard
High-end Keyframe Animation
![Page 5: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/5.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Animating Characters in 3D is hard
Using Motion Capture
![Page 6: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/6.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
What if it was as easy as doing a video chat?
![Page 7: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/7.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Video chat using a 3D character?How fun would that be?
![Page 8: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/8.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Face Plus - Technology Breakthrough
![Page 9: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/9.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Face Plus: Movie Quality
• Award Winning Animated Short “Unplugged”• Real-Time, Works with a standard webcam
![Page 10: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/10.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Face Plus: Movie Quality
• Award Winning Animated Short “Unplugged”• Real-Time, Works with a standard webcam
![Page 11: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/11.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Machine Learning is the Enabler
![Page 12: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/12.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Computer Vision vs Computer Graphics
• Computer Graphics
• Computer Vision
Expression & Emo-ons
Geometry Texture
Texture Geometry
Expression & Emo-ons
![Page 13: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/13.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Machine learning applied to facial animation
Learning a map between a face appearance and the expression/emo-on it conveys
![Page 14: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/14.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
FaceplusEngine
At run-me the trained algorithm :• Detect faces • Extract facial expressions and emo-ons• Provide the rendering engine with the informa-on required to animate a character
Machine learning applied to facial animation
![Page 15: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/15.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Faceplus processing pipeline
Retargeting/animationFace DetectionViola Jones framework1,2
Face Tracking Expression Mapping
1Robust real-‐-me object detec-on. Paul Viola, Michael Jones. IJCV 20012Find a Face. Alexander Lyashevsky . AMD Fusion developer summit 2011
![Page 16: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/16.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Faceplus processing pipeline
Retargeting/animationFace DetectionViola Jones framework1,2
Face Tracking Expression Mapping
1Robust real-‐-me object detec-on. Paul Viola, Michael Jones. IJCV 20012Find a Face. Alexander Lyashevsky . AMD Fusion developer summit 2011
![Page 17: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/17.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Faceplus: face tracking
Global Models
Deformable model that aims at modeling the appearance of the en-re face
The model parameters are determined by matching the model to the input image
Pros Accurate
Well suited for GPU accelera-on
Cons
Inverse problem subject to local minima
Require ini-aliza-on / re-‐ini-aliza-ons
Computa-onally intensive
Geometrical Component
Ac5ve Appearence Models. T.F. Cootes, G.J. Edwards, C.J. Taylor. ECCV 1998
![Page 18: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/18.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Faceplus: face tracking
Global Models
Deformable model that aims at modeling the appearance of the en-re face
The model parameters are determined by matching the model to the input image
Pros Accurate
Well suited for GPU accelera-on
Cons
Inverse problem subject to local minima
Require ini-aliza-on / re-‐ini-aliza-ons
Computa-onally intensive
Texture Component
![Page 19: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/19.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Face global modeling
Putting the global model together
![Page 20: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/20.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Faceplus: face tracking
Local Models
Face modeled as a set of rigid 2D patchesLocal features are detected separately
Local features loca-ons are used to reconstruct the facial expression
Pros Robust
Doesn’t require any ini-aliza-on
Highly parallelizable / Scalable
Cons
Noisy reconstruc-on
Computa-onally intensive
Tricky SIMD implementa-on
Texture Component
Geometric Component
• Each patch is associated with a distribution model that
determines is location with respect to the detected face
![Page 21: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/21.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
How to invert these models?
![Page 22: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/22.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Inverting the face global model
Obtaining a residual to be minimized in a least square sense
ResidualInput video and superimposed
geometric model
(inversely)
Warped image
Texture model
W-‐ =
Current parameter
![Page 23: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/23.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Inverting the face global model
Inverting the model
ResidualInput video and superimposed geometric
model
(inversely)
Warped image
Texture model
W
-‐ =
Current parameter
• Generate the geometric model (ver-ces loca-ons) and the texture model (image) from the model parameters
• Each working item processes one pixel in the texture space:
• Computes the pixel loca-on in the video input space
• Samples the video input at the computed loca-on input video has to be an image buffer
• Subtracts the corresponding texture model
• Residual norm computed through reduc-on
• Images in the texture space don't need to undergo sampling opera-ons → Memory Buffers
![Page 24: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/24.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
• The registration is achieved by minimizing the residual in a least square sense.
• The GPU takes care of the most computationally intensive and highly parallel tasks (computation of the residual, Jacobian and some linear algebra)
• The CPU takes care of tasks that are harder to parallelize. ( solution of small, constrained linear systems )
• AAM inverse problem is notably slow on CPU. Most methods presented in literature to speed it up involve approximations that a!ects the robustness and e"ciency of the method.
• Alternatively the AAM inverse problem can be solved very e"ciently on heterogeneous systems.
Inverting the face global model in an heterogeneous system
![Page 25: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/25.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
• ( CPU ) The geometric model is used to determine the possible location of patches given the location of the detected face.
• ( GPU ) Features detectors are used to located the patches in the input image.
• ( CPU ) A clusterization algorithm takes care of grouping multiple detections.
• ( CPU ) A robust reconstruction algorithm estimates the geometric model (yellow line) given the detected patches( yellow dots ).
• ( GPU ) The global model is used to re#ne the model estimation ( blue line ).
Inverting the face local model in an heterogeneous system
![Page 26: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/26.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Implementation
• Faceplus is a library.
• Faceplus is portable: it is written in C++ and
OpenCL.
• Faceplus scales with the hardware.
• Faceplus can be used within 3rd party software:
Unity3d, Motion Builder.
![Page 27: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/27.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Conclusions
• Animation Creation reaching consumer• Very computationally intensive• Modern platforms such as the Tegra-K1
enable it on mobile• Potential for real-time applications such
as video chat 2.0
![Page 28: Real-Time Facial Motion Capture and Animation on Mobile€¦ · Real-Time Facial Motion Capture and Animation on Mobile Author: Emiliano Gambaretto Subject: 3D Animation is the art](https://reader036.vdocuments.us/reader036/viewer/2022063023/5ffd2e5cf597573f6d4e1863/html5/thumbnails/28.jpg)
GTC 2014 - Real-Time Facial Motion Capture & Animation on Mobile
Thank You!