© Encore Event Technologies
G E S T U R E C O N T R O L
H O W T O U S E M O V E M E N T A N D T O U C H
T O E N G A G E Y O U R A U D I E N C E
Gesture control is an exciting new tool in the AV world that allows multimedia content, including
video, to be controlled and manipulated in real-time by presenters, performers, and even the
audience themselves. As a field, it includes everything from projected interactive computer
animations that are altered by a presenter’s movements, through to sound effects built into
props that play when a person is detected in close proximity. In uniting the tactile with the
creative, the movements and positions of both people and objects can be used to trigger any
kind of multimedia content – the trick is in knowing how and when this technique is best used,
and how to go about creating the right kind of content to best exploit it.
E V E N T I N S I G H T S
W H I T E P A P E R S E R I E S T O
H E L P Y O U T R A N S F O R M
Y O U R E V E N T S
ENCORE-ANZPAC.COM/EVENT-INSIGHTS
EVENT INSIGHTS
E V E N T I N S I G H T S W H I T E P A P E R
© Encore Event Technologies
ENCORE-ANZPAC.
COM/EVENT-
INSIGHTS
LEARN MORE
T H E T O O L K I T
– S E N S O R S
Firstly, let’s look at the practicalities of
how to create the gesture control signal
itself. Gesture control relies on electronic
devices that can react to movement and
then send a signal to another device,
usually a computer. Devices that can do
this include iPhones, Microsoft Kinect
sensors, pressure sensors, infrared sensors,
magnetic sensors, and more.
I P H O N E S
iPhones are constantly generating data
on their position in space, how fast and in
which direction they’re moving, and the
way they’re being held; it’s how the screen
reorients so quickly when you move it from
vertical to horizontal. All of this data can be
easily streamed from the phone over WiFi
to any other device on that network, which
in a gesture control application, would be
a computer. Now think of the iPhone as
a sensor that can be placed anywhere on
stage, including inside an object or hidden
in clothing, that is constantly sending
information on where it is and how it is
moving. This data can be used as trigger
points for an action – when the phone is
moved from a specific location a computer
animated video is projected. When the
phone moves quickly around this point
(for example, being swung around by the
presenter) the animation is manipulated in
real-time to mirror the phone’s movement.
K I N E C T S E N S O R S
Microsoft Kinect sensors were originally
manufactured as motion capture devices
for the Xbox gaming console. They are
a camera and microphone designed to
track the movements of the human body
and capture speech. They can facilitate
real-time 3D motion capture, facial and
voice recognition. Thanks to an infrared
laser depth sensor, they can even work
in low-light environments, which are
often encountered on stage when
using projections. Kinect sensors can
be used to capture data on the position
and movement of any performer or
presenter and send that to a computer to
manipulate visuals in response. They can
also capture the image and movement
of the performers as video and send
that in real-time to be manipulated and
projected, creating amazing effects where
the performer is interacting with a virtual
version of themselves.
O T H E R S E N S O R S
Any device that can send some kind of
signal can be used for gesture control. A
simple infrared sensor, for example the
kind that sounds a shop’s ‘someone’s here’
bell when a customer walks through the
door, can easily be used to cue any kind of
event when an audience member breaks
its beam. Magnetic sensors send a pulse
when the magnet and the metal surface
it is adhering to are separated – a door
being opened could be used as the trigger
action. Pressure sensors can be inserted
into any surface and activated when stood
on, making an AV cue location specific.
T H E B U S I N E S S
E N D –
S O F T W A R E
& H A R D W A R E
The spectacular effects of gesture control
have been made possible by advances in
computer technology that make real-
time manipulation of video an affordable
possibility. This was previously the domain
T H I N K O F T H E
I P H O N E
A S A S E N S O R T H A T C A N B E P L A C E D A N Y W H E R E O N S T A G E
E V E N T I N S I G H T S W H I T E P A P E R
E V E N T I N S I G H T S W H I T E P A P E R
© Encore Event Technologies
ENCORE-ANZPAC.
COM/EVENT-
INSIGHTS
LEARN MORE
of supercomputers and not even remotely
achievable in a live setting.
These developments in control and
processing capabilities enable designers
and technicians to take control inputs
from any kind of source and use them to
trigger complex and continuous changes
to video that is being generated and
projected.
The most impressive form of gesture
control is the real-time manipulation of
large–format projected video in response
to presenters or performers. This can
mean anything from a cloud of particles
whirling around dancers in response to
their movements, to animated versions
of presenters that have been altered in
a computer to look like science-fiction
avatars. Taken further, it can even be
used visual techniques that look like
‘holograms’, otherwise known as the
‘Pepper’s Ghost’. These projections
require multiple large projectors working
together, co-ordinated by computer.
There also needs to be the stage space for
large projection screens, or in the case of
holograms, transparent scrims.
Peppers Ghost and gesture control enables dancer
performances to be interactive
C O N T E N T –
W H A T Y O U
N E E D T O G E T
R I G H T
Like any other creative tool, gesture
control is only effective when it helps
you communicate with your audience, as
opposed to just being an effect for effect’s
sake. Any viable idea for gesture control
will originate in the meaning and purpose
of your message. Is there something in
your presenter’s speeches that can be
better explained with gesture control? Is
there a feature of your new product that
can be expressed through movement?
Your AV company’s content creators will be
experienced in making the kind of content
that works best with gesture control, and
will help you make the right choices for
your event.
E X A M P L E S I N
A C T I O N
Staging Connections use of gesture
control at an award winning event for
Asian Paints:
Asian Paints had teamed up with artists
in India to transform old buildings by
painting stunning artworks and murals.
At a major international incentive event,
the company’s President talked about the
project. Using motion sensors to track the
President’s movements, the team gave
him a paint brush which he used to make
painting movements over the projected
building images. The new building photos
were ‘painted’ in real-time to reveal the
artworks. The audience was delighted and
clapped and cheered!
Gesture control technology following speaker’s
movements to deliver a live reveal
A C C E L E R O M E T E R , V I D E O P R O J E C T I O N
At the same event for Asian Paints, the
President presented a paint tin to the
audience while a 3D animated version of it
was projected onto the screen behind. The
movement of both tins, real and animated,
were synced perfectly.
T R I G G E R C O M P L E X A N D C O N T I N U O U S C H A N G E S T O
V I D E O
T H A T I S B E I N G G E N E R A T E D A N D
P R O J E C T E D
E V E N T I N S I G H T S W H I T E P A P E R
© Encore Event Technologies
ENCORE-ANZPAC.
COM/EVENT-
INSIGHTS
LEARN MORE
When the real paint tin was vigorously
‘tipped’, its 3D counterpart ‘poured’ out a
vibrant animation.
Motion sensing technology following speaker’s
movements to deliver a dynamic presentation
Real-time movement capture and
manipulation, projection as hologram:
Customer - Australian Hotel Association
Using a UV camera, we bought a live feed
of dancers into our software to trace an
outline of the dancers in real-time, and
projected the result onto a ‘Pepper’s Ghost’
screen, creating a three-dimensional
illusion that interacted with the real
dancers, and changed colour with the light
show.
Dancers movements perfectly aligned with the
visuals using motion sensing technology
O T H E R
E X A M P L E S
G E S T U R E S T O C O N T R O L L I G H T I N G
Other innovations we can offer to make
an event stand out is to use sensor
technology to control lighting. Lighting
can now be controlled by tracking the
hand movements of presenters. When
presenters wave or gesture with their
hands, the lighting can mimic and follow
the movement. Such as circling their
hands in the air, the lighting will also
project in a circular motion. Sounds can
also be controlled by motion such as by
raising arms higher, the music levels can
follow suit. Even the closing of fists can
be calibrated to mean ‘off’ and switch off
sound and lighting. With gesture control,
the presenter can truly become the
Maestro of their presentation with virtually
limitless opportunities to surprise and
delight your audiences.
G E S T U R E T O C O N T R O L S O U N D E F F E C T S
One of the simplest forms of gesture
control we have used is to set up magnetic
sensors on doors. This has been used to
create a ‘Haunted House’ to play spooky
sound effects on entering a room. We’ve
also rigged a large ‘Magic Mirror’ with
a pressure sensor that played random
comments when attendees walked past it.
[ CONTACT ]
Speak to your local Encore representative or
get in touch by emailing
or call 1800 209 099 (AU) or
+64 9 259 0025 (NZ)
T H E P R E S E N T E R C A N T R U LY
B E C O M E T H E
M A E S T R O
O F T H E I R P R E S E N T A T I O N
