hand tracking - m tech seminarpratikm/projectpages/...art technology hand tracking pratik kalshetti...
TRANSCRIPT
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Hand TrackingM Tech Seminar
Pratik Kalshetti
IIT Bombay
November 22, 2016
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Outline1 Introduction
Immersive Virtual EnvironmentChallenges in Hand TrackingAim of Seminar
2 ContributionElements of a Hand Tracking System
Sensor TypeHand ModelReal-time MeasurementsCandidate Evaluation
Pipeline of a Hand Tracking System
3 ImplementationSensorHand ModelForward Kinematics for AnimationExperimental SetupResults
4 Conclusion
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Immersive Virtual Environment
• Computer-generated scene
• First person point of view
• Manipulating virtualobjects using human hand
• Hand needs to be trackedfor communicating withthe virtual world Figure: Hologram
Image Source: [4]
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Challenges in Hand Tracking
High DoF
Figure: Human Hand.
Image Source: [2]
Sensor Limitation
Figure: Fingers behindare occluded.
Robustness
Figure: Robustnessdecreases withdistance.
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Aim of Seminar
• Understand various hand tracking strategies in recent era
• Provide a holistic view of the major stages in a handtracking system
• Implement a hand tracking system to test the state of theart technology
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Elements of a Hand Tracking System
Figure: Elements of Hand Tracking System.
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Sensor Type
Marker-based Tracking
Figure: Tracking using Color Glove
Image Source: [8]
Vision-based Tracking
Figure: Tracking using DepthSensor
Image Source: [6]
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Hand Model
Complex Mesh Model
Figure: Linear blend skinnedmodel. The numbers near eachjoint indicate the allowed DoF.
Image Source: [1]
Simple Mesh Model
Figure: From left to right: Thecylinder model used for tracking,the skeleton, the BVH skeletonexported to Maya to drive therendering, the rendered handmodel
Image Source: [7]
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Real-time Measurements
Accurate measurements help in recovery from tracking loss
Figure: Hand Segmentation using skin silhouettes.
Image Source: [5]
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Candidate Evaluation
• Appearance-based Method - Fast; Fails when Occlusion
• Model-based Method - Slow; Works for Partial Occlusion
Figure: Evaluation Strategies
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Pipeline of a Hand Tracking System
Figure: Overall Pipeline.
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Sensor Type
Leap Motion Sensor
(a) Hand Tracking usingLeap Motion Sensor.
(b) Inside Leap MotionSensor.
Figure: Hand Tracking Hardware
Image Source: [3]
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Hand Model25 DoF Model made of cylinders and quad in OpenGL
Figure: Front View of implemented Hand model.
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Forward Kinematics for Animation
Forward Kinematics
Q: n - vector of joint anglesX : vector defining endpoint’s position and orientationX = F(Q)
(a) Step 1 (b) Step 2 (c) Step 3 (d) Step 4
Figure: Animation of bending Index Finger using Forward Kinematics
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Experimental Setup
Software: Leap Motion SDK; OpenGL API
Figure: Experimental Setup.
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Results
(a) Pose 1. (b) Pose 2.
Figure: Successfully Tracked Poses
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Results
(a) Occlusion. (b) Fast motion.
Figure: Incorrectly Tracked Poses
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
Conclusion
• Open new doors to the world of human computerinterfaces
• Modular division of stages will help focus on each elementindependently
• Implemented model suggest the persisting problems withcurrent hand tracking systems
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
References I
Luca Ballan et al. “Motion capture of hands in actionusing discriminative salient points”. In: EuropeanConference on Computer Vision. Springer. 2012,pp. 640–653.
Anatomy Human Body. Diagram Hand Bones AnatomyHand And Wrist Diagram Human Anatomy Diagram.url: http://www.anatomy-diagram.info/diagram-hand-bones-anatomy/diagram-hand-bones-anatomy-
hand-and-wrist-diagram-human-anatomy-diagram/
(visited on 11/17/2016).
Mike Hord. Leap Motion Teardown. url:https://learn.sparkfun.com/tutorials/leap-
motion-teardown (visited on 11/08/2016).
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
References II
Jason Mick. Microsoft CEO: HoloLens V1 Will Debut forDevs in 2016, See 5 Year Evolution. url:http://www.dailytech.com/Microsoft+CEO+
HoloLens+V1+Will+Debut+for+Devs+in+2016+See+5+
Year+Evolution/article37456.htm (visited on11/17/2016).
Juan Manuel Ramirez-Cortes et al. “Shape-based handrecognition approach using the morphological patternspectrum”. In: Journal of Electronic Imaging 18.1 (2009),pp. 013012–013012.
Toby Sharp et al. “Accurate, robust, and flexible real-timehand tracking”. In: Proceedings of the 33rd Annual ACMConference on Human Factors in Computing Systems.ACM. 2015, pp. 3633–3642.
HandTracking
PratikKalshetti
Introduction
ImmersiveVirtualEnvironment
Challenges inHand Tracking
Aim of Seminar
Contribution
Elements of aHand TrackingSystem
Sensor Type
Hand Model
Real-timeMeasurements
CandidateEvaluation
Pipeline of aHand TrackingSystem
Implementation
Sensor
Hand Model
ForwardKinematics forAnimation
ExperimentalSetup
Results
Conclusion
References
References III
Andrea Tagliasacchi et al. “Robust Articulated-ICP forReal-Time Hand Tracking”. In: Computer GraphicsForum. Vol. 34. 5. Wiley Online Library. 2015,pp. 101–114.
Robert Y Wang and Jovan Popovic. “Real-timehand-tracking with a color glove”. In: ACM transactionson graphics (TOG). Vol. 28. 3. ACM. 2009, p. 63.