JN 04/19/23

Knowledge Systems Lab

A Method for Temporal Hand Gesture Recognition

Joshua R. NewKnowledge Systems Laboratory

Jacksonville State University

JN 04/19/23

Knowledge Systems Lab

Outline

• Terminology

• Motivation

• Current Research and Applications

• System Overview

• Implementation Approach

• Demonstration

JN 04/19/23

Knowledge Systems Lab

Terminology

Image Processing - Computer manipulation of images. Some of the many algorithms used in image processing include convolution (on which many others are based), edge detection, and contrast enhancement.

Computer Vision - A branch of artificial intelligence and image processing concerned with computer processing of images from the real world. Computer vision typically requires a combination of low level image processing to enhance the image quality (e.g. remove noise, increase contrast) and higher level pattern recognition and image understanding to recognize features present in the image.

JN 04/19/23

Knowledge Systems Lab

Motivation

JN 04/19/23

Knowledge Systems Lab

Motivation

• Gesturing is a natural form of communication– Gesture naturally while talking– Babies gesture before they can talk

• Interaction problems with the mouse– Have to locate cursor– Hard for some to control (Parkinsons or

people on a train)– Limited forms of input from the mouse

JN 04/19/23

Knowledge Systems Lab

Motivation (2)

• Interaction Problems with the Virtual Reality Glove– Reliability– Always connected – Encumbrance– Sanitation

JN 04/19/23

Knowledge Systems Lab

System Overview

JN 04/19/23

Knowledge Systems Lab

System Overview

StandardWeb Camera

Rendering

User InterfaceDisplay

Han

d

Movem

ent

User

Gesture Recognition

System

ImageCapture

Update Object

Image Input

JN 04/19/23

Knowledge Systems Lab

System Overview (2)

• System:• OpenCV and IPL libraries (from Intel)

• Input:• 640x480 video image• Hand calibration measure

• Output:• Rough estimate of centroid• Refined estimate of centroid• Number of fingers being held up• Manipulation of 3D skull in QT interface in response to

gesturing

JN 04/19/23

Knowledge Systems Lab

System Overview (3)

• Hand Calibration Measure:• Max hand size in x and y orientations in # of pixels

JN 04/19/23

Knowledge Systems Lab

System Overview (4)

Saturation Channel Extraction (HSL space):

Original ImageOriginal Image Hue

Lightness

Saturation

JN 04/19/23

Knowledge Systems Lab

Proposed Approach

JN 04/19/23

Knowledge Systems Lab

Proposed Approach

JN 04/19/23

Knowledge Systems Lab

Proposed Approach (2)

JN 04/19/23

Knowledge Systems Lab

Proposed Approach (3)

)(*19.0 HandSizeYHandSizeXRadius

• The finger-finding function sweeps out a circle around the rCoM, counting the number of white and black pixels as it progresses

• A finger is defined to be any 10+ white pixels separated by 17+ black pixels (salt/pepper tolerance)

• Total fingers is number of fingers minus 1 for the hand itself

JN 04/19/23

Knowledge Systems Lab

Proposed Approach (4)

• Temporal Recognition System for hand movement patterns

• Input feature-vector creation: based on calculations from 12 centroid locations (hand movement during 3 seconds – average length of temporal gesture)

• Learning system training and recognition: SFAM classification used for ability to interactively add new, user-defined gestures

JN 04/19/23

Knowledge Systems Lab

Proposed Approach (5)

• Temporal Gesture Recognition System

JN 04/19/23

Knowledge Systems Lab

Proposed Approach (6)

Input Feature-Vector Creation• Twelve centroids collected (x,y)• Find top-left centroid so that gestures are not start-point

dependent (a square is a square whether you start at the top left or the bottom right)

• Compute centroid differences to recognize movement, not position (a square whether the hand is at the precise 12 points or in between those points)

• Normalize using the perimeter to recognize percentage of total movement since users are inaccurate in repeating a gesture (a square whether large or small)

Note: System still sensitive to clockwise vs. counter-clockwise square (undo-like feature)

JN 04/19/23

Knowledge Systems Lab

Proposed Approach (7)

Learning System Training and Classification1. Feature vector is formatted for use by SFAM2. Movements of the hand were recorded and assembled into

one file for training the SFAM system- 25 examples each of circle, square, left/right, up/down

3. System classification of hand gesture every 3 seconds4. Train New Gesture button provided, stores gesture under the

label entered in the box (5 is the default since 1-4 are already taken)

JN 04/19/23

Knowledge Systems Lab

Proposed Approach

JN 04/19/23

Knowledge Systems Lab

Demonstration

SystemConfiguration

SystemGUI Layout

JN 04/19/23

Knowledge Systems Lab

Demonstration (2)

Gesture to Interaction Mapping

Number of Fingers:

2 – Roll Left3 – Roll Right

4 – Zoom In5 – Zoom Out