7/27/2019 2013-10-28 Progress report

1/2

Human-Robot Interaction in Unstructured EnvironmentsProgress ReportOctober 28, 2013Kevin DeMarco

Gesture Recognition

Last week I began researching state-of-the-art methods for gesture recognition without a depth sensor. Thereare many examples of individuals using a single web camera to detect a users hand and extracting somesimple pose information. However, most of the examples involve an individual with their hand in frontof a background that contrasts well with the hand. The common methods within OpenCV for hand poserecognition involve forming a convex hull around the hand, where the vertices of the hull imply the positionsof the finger tips. While most of the examples end at this phase of the detection, the next step in gesturerecognition involves detecting hand and finger movements in space. In this phase of detection, Hidden MarkovModels (HMM) or Support Vector Machines (SVM) can be used for training the gesture models. I found anopen source project, called the Gesture Recognition Toolkit (GRT), that uses SVMs for gesture recognition,which I plan on testing ( http://www.nickgillian.com/software/grt ).

Most of the gesture recognition methods involve two steps: segmentation of the hand from the environmentand pose or gesture recognition using machine learning. It seems that the hand segmentation aspect witha single non-depth camera is the most difficult aspect, since once the hand and finger poses are known,the machine learning algorithms can easily match the gestures to pre-trained models. I am interested inresearching knowledge-based methods of hand segmentation since the human eye and brain are so adept atdetecting hands in optical images. We should be able to develop a computer program that can extract thehands pose in arbitrary lighting and background conditions.

OpenCV Workbench

When I was working with OpenCV during the first phase of my sonar-based diver detection research, I foundthat I spend a considerable amount of time writing custom C++ code to trim videos down to size andcapturing new test videos. Thus, I decided to write my own OpenCV Workbench GUI application, as shownin Figure 1, to assist in development.

The OpenCV Workbench facilitates jumping to and examining specific frames in the video sequencethrough the use of the horizontal frame slider and a number box that lets the user enter the exact frameto examine. Also, the Workbench has the capability to change the playback framerate. Not only can theOpenCV Workbench read all of the commonly used video formats (e.g., AVI, WMV, MP4, MPG, and MOV)and a standard web camera, but it can also read the sonar files from a BlueView 2D Imaging Sonar. Thisallows the user to use a single interface when prototyping computer vision algorithms.

The OpenCV Workbench also has the capability of exporting newly recorded and edited video streams tonew files on disk. The export video dialog window is shown in Figure 2. The Qt GUI framework provides arobust framework that enables error checking on user forms before invalid data entry can cause the programto crash.

The next major feature that I will be implementing will be a visual interface for enabling and disablingcomputer vision algorithms (e.g., threshold, edge detection, face detection, background subtraction, etc.) and

a GUI for arranging the order in which the algorithms execute.

TODO

Test out the Gesture Recognition Toolkit (GRT) with OpenCV

Implement interface to enable and disable computer vision algorithms

Research the use of knowledge-based computer vision for improved tracking of hand posture.

1

7/27/2019 2013-10-28 Progress report

2/2

Figure 1: The OpenCV Workbench GUI

Figure 2: The OpenCV Workbench GUI: Export Video Frames

2