National Research Council CanadaConseil national de recherches Canada

Canada

In Proc. IEEE Conf. on Automatic Face and Gesture Recognition, Washington DC, May 21-22, 2002

Dmitry O. Gorodnichywww.cv.iit.nrc.ca/~dmitry

Institute for Information TechnologyInstitut de technologie de l'information

Computational Video GroupGroupe Vidéo Informatique

((22)) Face-Tracking Based User Interfaces Face-Tracking Based User Interfaces 1. Replacing cumbersome track-ball (track-stick) on laptops.2. Extra degree of control (e.g. to switch the focus of attention).3. Hands free control (e.g. for handicap users).4. Interactive games: more physical, entertaining, 3D control, multiple-user

Fig.1. A user plays an aim-n-shoot Bubble-Frenzy game aiming the turret by pointing with her nose. (slight rotation of head allows to aim precisely in 180o range)

((33))

Fig. 2. Two users play a virtual ping-pong game, bouncing the ball with their heads. Image-based tracking allows one to track heads, however it doesn’t allow one to pin-point with head.

Key Issues and ApproachesKey Issues and Approaches 1.Speed (in real time). 2.Affordability (with cheap easy-to-install, but low-quality USB cams)

3.3.RobustnessRobustness (to normal head motion).4.PrecisionPrecision (with pixel precision)

Image-based Face Tracking: - Uses global facial cues: skin colour, head shape, head motion- Doesn’t require high-quility images, robust, but not precisenot precise

((44))

Should be used for Should be used for preciseprecise tracking. However, it’s tracking. However, it’s not robustnot robust.. [ Bradsky, Toyama, Gee, Cipolla, Zelinsky, Matsumoto, Yang, Baluja, Newman, …]

…”still not ready for practical implementation”

- Feature f is associated with vector Vf (obtained by centering a mask on the feature)- Features are tracked by template matching with Vf in the local area of interest (calculated with image-based cues)

- The pixel u=(i,j) which has the largest score s(Vu, Vf) is returned

Feature-based Face TrackingFeature-based Face Tracking

Fig. 3. Tracking eyes(from [Gorodnichy97]).

Question: What features to use?Question: What features to use?Proposition 1: Robust and precise tracking can be achieved by designing an invariant to head motion feature template.

((55))

Features are conventionally thought of as visually distinctive (ie with large I(f) ).

Hence, the commonly used features are edge-based, such as corners of brows, eyes, lips, nostrils etc.

They however are • not robust

• not always visible

Desired feature properties:1. Uniqueness: s(Vf, Vu) min

2. Robustness: s(Vft=0, Vf

t) max3. Continuity (for sub-pixel accuracy): the closer a pixel u in an image is to the

pixel corresponding to f, the larger the score between Vu and Vf

(Then evidence-based convolution can be applied to refine feature position u)

Edge-based Features Edge-based Features – not good

((66)) Convex-shape features Convex-shape features – much betterDefinition 1: Convex-shape feature is defined as an extremum of a convex-shape surface

• Shape-from-Shading theory shows, that these features exhibit the desired properties (for the fixed camera-user-light configuration)

Nose featureNose featureDefinition 2: Nose feature is the extremum of the 3D nose

surface curvature defined as z=f(x,y) in camera centered coordinate system.

Thus defined, Nose feature is • Very robust• Can be detected with sub-pixel precision • PLUS, It is always visible!

((77)) NouseNouse TM TM Face Tracking TechnologyFace Tracking Technology

• Based on tracking the convex-shape nose feature.• Enables precise hand-free 2D control in a) joystick or b) mouse modes.

• Allows aiming and drawing with the nose.Just think of your nose as a chalk or a joystick handle!

NB: Left/Right head motion is very natural and can be easily applied for control, provided it can tracked precisely.

• Affordable and downloadable. Uses a generic USB camera!

Zero intialization of Nouse Using Nouse for Painting

((88)) Performance: Robustness & PrecisionPerformance: Robustness & PrecisionThe range of head motion tracked

‘No’ motion

‘Yes’ motion

Robustness to rotation Robustness to scale

Test: The user rotates his head only! (the shoulders do not move)

((99)) On Importance of Two Cameras On Importance of Two Cameras For humans: it is much easier to track with two eyes than with one eye.

Not only extends tracking from 2D to 3D,but also makes tracking more precise and robust!

For computers however: … 1. The relationship between “eyes” is not known. 2. Tracking of features is not robust (to rotation and scale)

StereoTracker from CVG NRC: Tracks face in 3D with two USB cams to control a virtual man, by using 1) Projective Vision Theory

and 2) robust Nose Feature Tracking

((1010)) StereoTracking with USB webcamsStereoTracking with USB webcamsStage 1: Self-calibration• The relationship between the cameras is represented using the Fundamental Matrix F: (uleft, F uright)=0• F can be found automatically for any two cameras by observing the same scene using www.cv.iit.nrc.ca/research/PVT:

find cornersmatchingfilteringrobust solution with 7 selected corners (RANSAC)F

Stage 2: Feature selection and calibration verification• Select features in one image• Verify that the epipolar line

passes thru each feature in the second image

• Use nose tip feature and two other common features (eg brow corners)• More at www.visioninterface.org/vi2002

((1111)) Using Nose for StereoTrackingUsing Nose for StereoTrackingProposition 2 : With F known, the tracked 3D feature is the one that minimizes the epipolar error defined by

Proposition 3 : First detect convex-shape nose feature. Then use rigidity constraint to find other features.

((1212)) ConclusionsConclusions

AcknowledgementsAcknowledgements• Nouse TM is trademark of Computational Video Group IIT NRC• Work done with Gerhard Roth, Shazad Malik• BubbleFrenzy game is provided by www.extendedreality.com

• Nose is a very unique feature. Humans are lucky to have it!• Nose allows us to track a face very robustly and precisely. • Pointing with Nose is natural.

This makes 2D perceptual user interfaces a reality!• Nose helps recovering other facial features. • Two cameras (even bad webcams) make tracking more robust.

This makes 3D face tracking affordable, precise and robust. • Use your Nose as MouseUse your Nose as Mouse! – Use NouseUse Nouse!

NouseTM is open for public evaluation atwww.cv.iit.nrc.ca/research/Nouse