Implementing Gesture Recognition

The first question that aroused my curiosity was, ‘What exactly is a ‘gesture?’ According

to a dictionary, gesture is a sequence of postures that are connected by motions over a

short period of time. I did not find that really helpful. I’ll explain it in an easier way—a

gesture is a form of non-verbal or non-vocal communication. So your messages are

communicated by visible bodily actions like waving, swiping, facial movements or even

pointing to the sky.

What is gesture recognition This technology enables devices to interpret human gestures using sensors, whose data

is then processed using mathematical algorithms. The system is built to identify specific

human gestures, and then use them either to convey information or to control a device.

The gestures used have to be intuitive, simple and universally acceptable to ensure they

are easily adopted by users.

Gesture recognition makes computers more accessible for the physically-challenged and

also makes interaction more natural in the 3D virtual world or in gaming. In today’s

world, gesture recognition is an evolving technology that can be seen all around us. The

latest phones, computers, gaming consoles, and even TVs now feature it.

How it works The human body’s movements are read by an imaging sensor like a camera, which then

sends that data to a computer. The computer uses the gestures captured as inputs,

processes this input to understand what was gestured, and then sends the answer as a

command to the currently running application or control device.

Let us look at an example of a person clapping his hands in front of a camera. When this

clapping gesture data is processed by the connected computer, it could produce the

sound of cymbals, as programmed. The specialty of this technology is that users do not

have to wear any special equipment or attach any device to their bodies, as is the case

with some alternative technologies.

Sensing the gestures When you gesture at your computer or mobile device, it obviously needs a way to ‘see’

or ‘sense’ what you are doing. This is where sensors come in. There are a number of

sensing technologies that can be considered, but we’ll stick to the popular ones here. Accelerometers. Also known as inertial or acceleration sensors, these are used to detect

the changes in force resulting from the fall, motion, tilt, shock, positioning or vibration of

the body they are connected to, with respect to time. These measure linear acceleration,

so if the device is in a free fall, the accelerometer will give a ‘0’ value for acceleration. Application. In the case of a smartphone, the direction of the screen changes as soon as

we tilt or turn the phone. The direction in which we hold the phone (horizontal or

vertical) is sensed by an accelerometer and the information in the form of electrical

signals is sent to the screen, which automatically adjusts itself either in landscape or in

portrait mode. So, here, contact gesture recognition plays an important role. Accelerometers have paved the way for many games to be developed for smartphones,

especially for some of those sports simulation games that let you use the phone as a

steering wheel or a fishing pole. To get an idea of the range of accelerometers that you

can select from, refer to Table I.

Gyroscopes. Gyroscopes, also popularly known as gyros, have had a major impact in the

world of gesture recognition. Their ability to measure the orientation of a device enables

them to measure both vertical and horizontal rotation. This has brought smoothness and

preciseness to the operation of a mobile device. The next time you hold a phone on its

side, and the screen transforms into landscape mode automatically, you know which

component to thank. Application. In the case of a smartphone or a tablet, adding a gyro and an accelerometer

allows the device to sense motion on six axes including roll, pitch and yaw—which helps

you fly your airplane in a flight simulator game on a phone. So if accelerometers have

given more business to game developers, gyros have helped them be more innovative.

Bend sensors. These are also known as flex sensors. As we know, even finger

movements are considered as gestures, so bend sensors help us to identify those

gestures by recognising finger postures. They work on the principle that resistance

increases with an increase in the bend. Data gloves. Gesture recognition technology has not only been implemented in

smartphones and tablets, but also in television sets and computers. In order to

implement hand gesture recognition on these devices, data gloves have been developed.

These gloves identify the finger and hand gestures in real time scenario. These are wired

either to a TV set or to a computer and then worn by the person operating the device.

These act as an input device and eliminate the use of a mouse or a keyboard.

Application. The bend sensors are placed on the joints of all the fingers in the data glove,

so the more the fingers are bent, the more is the resistance which is introduced in the

device for further action.

Infrared proximity sensors. These have the ability to detect the presence of nearby

objects without any physical contact. They emit an electromagnetic field and then

identify the changes either in the field or in the return signal, which they detect after

reflection from an object that has got close to the sensor. There are many types of

proximity sensors; however, infrared proximity sensors play a very important role in

gesture recognition. As soon as these infrared sensors recognise the movement of a

body in front of them, they pass a signal to the device for further action.

Application. Many of us have seen a wash basin with an automated tap. The tap turns on

as soon as it senses some hand movement under it. This is because of the presence of

infrared proximity sensors that detect your hand movement. Here’s a fun thing to try

out—if you keep your hand under the tap without any movement, the tap would not

allow the water to fall. This is because if you keep your hand immobile, the sensor does

not sense a changing field and thus thinks that your hand is not in front of it any more.

Capacitive proximity sensors. They support gesture recognition enabling robust, easy-to-

use and feature-rich user touch interfaces on a device—these are what make up your

capacitive touch screen. Such sensors allow users to operate the device just by the touch

of their skin or finger rather than by applying pressure. Some of the newer models have

superior sensors that do not even require a touch—hovering above the screen is enough

to get their attention.

Application. This has helped to bring about a new era of devices with superior touch

screens that allow multiple points of input, and better durability. What’s more, capacitive

screens allow the device to have a glass or some other rigid protection on top of the

touch screen, thus improving the device’s ergonomics.

Touch-sensing controllers, such as the CapSense from Cypress, have specific technical

improvements like water tolerant capacitive touch buttons, which eliminates false

touches under moist conditions. “They also allow capacitive proximity sensing for up to

30cm, which is instrumental in enabling features such as ‘wake-on approach,’ and can

even replace the IR sensors mentioned in the previous section,” explains Anbarasu

Samiappan, product apps manager, Cypress Semiconductor.

Microsoft’s Digits prototype is part of an effort to bring gesture-based

control in a mobile device. This wrist-worn sensor uses IR laser, LEDs and

camera to recognise the fingers movement (Image courtesy: Microsoft) What’s coming up in the future I am pretty sure we will see something like Tom Cruise’s sorcery in ‘Minority Report’

within a couple of decades. But until that happens, the field is taking great strides

forward. Researchers have been working on new kinds of hardware to compute gestures,

including FPGAs. “Apart from basic components to capture the images, an FPGA-based

gesture recognition system handling large computational complexity of optical flow to

perform parallel processing is a low-cost solution. Using an FPGA system, gesture

recognition can be implemented at 30 frames per second, and the system software can

subsequently schedule all tasks during processing,” explains P. Chow Reddy, manager

(R&D), Power Division, ICOMM Tele Pvt Ltd. Apart from the focus in consumer electronics, gesture recognition seems to have

matured enough to be noticed and implemented in industrial settings. The BMW Group

has demonstrated a gesture-driven system that allows a quality assurance worker to

“…examine and document flaws in a component simply by pointing.” Ford had earlier

announced its own gesture-based ambitions in the form of ‘a complete virtual factory.’

Industry aside, even Microsoft Office 2013 has gesture recognition built into it. At Electronics Rocks‘13 conference, one of our speakers Hemant Surale showcased a

cool smart glove product that he had designed. Hemant says that the present scenario in

gesture recognition seems that development in this area is at its peak. “Most of the

devices comes with their own proprietary SDKs for application developers. In addition

there are many emerging startups developing effective way to interact with day to day

equipments. But, Internet of things will present new challenges with respect to

integration of such systems into day to day life of users.” Hemant is also an active

member of the IoTBLR community. IoTBLR is a community based in Bangalore, which is

excited about connecting disparate devices to one another via the web to create Internet

of Things products. Any engineer can find them on facebook at

facebook.com/groups/IoTBLR

The field of gesture recognition seems to be gearing up for quite a future, and design

houses working on this technology better get ready for the competition that comes with

any high-profile industry.