Hand Tracking forVirtual Object Manipulation

Thibaut Weise

Project Supervisor:Second Marker:

Professor Guang-Zhong YangDr Duncan Gillies

2/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

MotivationVirtual and Augmented Environments

Robotic ControlSurgerySpace, Deep Sea

3/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

State of the Art

EM-Tracking

Acoustic Tracking

Data Glove

Optical

4/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Optical Approaches

Appearance-based Approach

Model-based ApproachReconstruct 3D Hand Model

Two methods:Minimize Model against FeaturesReconstruct from Features

5/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Requirements

Focus: Grab Gesture

Thumb and Index FingerReal-timeResponsiveAccurate

6/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

System Overview

Calibration

Hand Tracking

Virtual Environment

Marker-based

Marker-less

7/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

ReconstructionTriangulation

Requires Mapping between Pixels and 3D World Coordinates

8/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Calibration

Optimize known World Coordinates against detected Image Coordinates

Planar Approach => Chessboard

OpenCV ImplementationStereo Optimization

9/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Calibration AccuracyCorner Detector Accuracy

Why sub-pixel accurate?

0

2

4

6

8

10

12

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14

Corner Detector Error Std (view angle)

Re

co

ns

tru

cti

on

Err

or

(mm

)

10/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

X-Corner DetectorSolution: Improved X-Corner Detector

Saddle Point

Product of Eigenvalues minimised: λ1λ2= fxxfyy-fxy2

Gradient zero:0 = fx + s fxx + t fxy

0 = fy + s fxy + t fyy

Recalculate using weighted derivatives of neighbourhood

11/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Corner Detector Comparison

Solution: Improved X-Corner Detector

0

0.04

0.08

0.12

0.16

0.2

0.24

0.28

0 0.05 0.1 0.15 0.2

Gaussian White Noise (std)

RM

S D

etec

tio

n E

rro

r (p

ixel

)

XCD

IXCD

OpenCV

12/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Calibration Automation

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

40 60 80 100 120 140 160

Number Chessboard Points

Rec

on

stru

ctio

n E

rro

r (m

m)

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

4 9 14 19

Number of Views

Rec

on

stru

ctio

n E

rro

r (m

m)

Improved Chessboard Detector

13/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Calibration Tool

14/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Hand Tracking Marker-based Marker-less

15/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Marker Tracking3 Markers

Thumb, Index Finger and Hand

Uses Improved X-Corner DetectorFastAccurateSimple

16/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Hand ReconstructionFind all potential Markers in ImagesMatch Points using Epipolar Constraints

17/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Hand ReconstructionReconstruct all possible hand configurations

Different markersValidate hand constraints

18/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Hand ReconstructionUse best match based on temporal information

Minimise Euclidean distance to predicted position

19/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Hand ReconstructionInitialisation based on Orientation Heuristic

20/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Marker-less Tracking2 FingertipsReconstruct Hand Position

Hand LocalisationSkin Colour Detection

21/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Hand Localisation

Background Subtraction

-

Skin + Background

+

22/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Hand Localisation

Find Hand Contour

23/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Hand ReconstructionFind Peaks

Calculate k-curvature:[C(i)-C(i-k)]T[C(i+k)-C(i)]

Peaks where angle > 0o and < 90o

24/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Hand Reconstruction

Match peaks using Epipolar Constraints

find possible Finger Tips

Calculate Finger Axis

25/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Hand ReconstructionReconstruct hand position

Validate hand constraints

Use best hand based on temporal information

26/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Evaluation

Marker-based Marker-less

27/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Virtual EnvironmentVirtual Tower of Hanoi

Virtual Hand => GrabberFixed Dimensions

Grab and Push Objects

PerceptionColour CodingShadow

28/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Implementation

ApplicationThread

Video / Tracking Thread

Virtual Environment

Hand UpdateDraw / Settings

Settings

29/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Demo

30/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

ConclusionAccurate semi-automatic Calibration

Marker-based Tracking

Marker-less Tracking

Virtual Environment

31/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Future WorkMarker-based Tracking

Handle Occlusion

Marker-less TrackingRefine Skin Colour DetectionUse Gradient

32/32Visual Information Processing Visual Information Processing Department of ComputingDepartment of Computing Thibaut WeiseThibaut Weise

Questions

?Who wants to try?