kinect audio-runner: audio feedback for improving...

Kinect Audio-Runner: Audio Feedback for Improving Performance

in Long-Distance Running

J O R D I B O L Í B A R

Master of Science Thesis Stockholm, Sweden 2012

Kinect Audio-Runner: Audio Feedback for Improving Performance

in Long-Distance Running

J O R D I B O L Í B A R

DT212X, Master’s Thesis in Music Acoustics (30 ECTS credits) Degree Progr. in Electrical Engineering 270 credits Royal Institute of Technology year 2012 Supervisor at CSC was Roberto Bresin Examiner was Sten Ternström TRITA-CSC-E 2012:062 ISRN-KTH/CSC/E--12/062--SE ISSN-1653-5715 Royal Institute of Technology School of Computer Science and Communication KTH CSC SE-100 44 Stockholm, Sweden URL: www.csc.kth.se

Kinect Audio-Runner: Audio feedback for

improving performance in long-distance running

Abstract

Unlike many other sports, running is perceived as a sport that can be easily practiced by

everybody without any need for a proper technique or previous training. Moreover, once a

running pose is adapted, it becomes fairly difficult to change it for good. However, audio

feedback is becoming a popular method for aiding people to improve their performance in

sports. Hence, the aim of this project is to help a runner to improve his technique on a treadmill

by sending him feedback by means of music and auditory icons. The movements of the runner

are tracked with a non-intrusive method using Kinect. Three different elements of the running

technique are taken into account and a specific training can be set for every runner by setting

different thresholds as goals to achieve during the training. Therefore, the training becomes

more intuitive and pleasant for the runner. All these features have been assembled in a program

which has been called Kinect Audio-runner.

Kinect Audio-Runner: Ljudåterkoppling för

träning av långdistanslöpare

Sammanfattning

Till skillnad från många andra sporter, löpning uppfattas som en sport som lätt kan praktiseras

av alla, utan något behov av en korrekt teknik eller tidigare utbildning. Dessutom, när man har

lärt en teknik, blir det ganska svårt att ändra den i framtiden. Å andra sidan är ljudåterkoppling

på väg att bli en populär metod för att hjälpa människor att förbättra sina resultat inom idrotten.

Därför är syftet med detta projekt att hjälpa en löpare att förbättra sin teknik på ett löpband

genom att ge honom återkoppling genom musik och ljud. Löparens rörelser analyseras med en

Kinect kamera. Tre olika delar av löpteknik tas hänsyn till och en särskild träning kan ställas in

för varje löpare genom att sätta olika tröskelvärden som mål att uppnå under träning. Därför blir

träningen mer intuitiv och trevlig för löpare. Alla dessa funktioner har samlats i ett program som

har kallats Kinect Audio-Runner.

Acknowledgments

I would like to thank everyone who helped me along this project. Daniele Cardinale, from the

Sport Physiology Laboratory, which is part of the Elite Performance Centre of the Swedish

Sports Confederation. Thanks for letting me use the sports lab to carry the tests and for all the

advice and feedback. My friends Etienne, Martin and Juan for putting up with many hours of

running on a treadmill, testing and improving the program. Roberto Bresin, my advisor, for all

the advice, help and material he has provided me. This work was supported by the Swedish

Research Council, Grant Nr. 2010-4654.

Table of contents

Introduction ................................................................................................................................... 1

Background ............................................................................................................................... 1

Goals ......................................................................................................................................... 2

Thesis content ............................................................................................................................ 2

Related work ................................................................................................................................. 3

Sonification and sports engineering .......................................................................................... 3

Biomechanics ............................................................................................................................ 3

Psychology of perception .......................................................................................................... 4

Problem formulation ..................................................................................................................... 5

Kinect and motion tracking ........................................................................................................... 6

Kinect functionalities ................................................................................................................ 7

Body tracking ............................................................................................................................ 8

Biomechanics and physics in running ........................................................................................... 9

The running cycle ...................................................................................................................... 9

What determines a good technique? .......................................................................................... 9

The tilt angle of the upper body .......................................................................................... 10

The distance between the landing foot and the projection to the ground of the centre of

gravity ................................................................................................................................. 11

The vertical displacement of the runner’s centre of gravity ................................................ 11

Methodology ............................................................................................................................... 12

Installing the drivers ................................................................................................................ 13

Pure data program ................................................................................................................... 13

Interface................................................................................................................................... 14

Audiorun ................................................................................................................................. 16

Record data .............................................................................................................................. 20

Skeleton and calculations ........................................................................................................ 21

Vertical displacement .......................................................................................................... 22

Step distance ........................................................................................................................ 27

Tilt of the torso .................................................................................................................... 30

Sonification ............................................................................................................................. 31

Sonification of the vertical displacement ............................................................................ 32

Sonification of the tilt .......................................................................................................... 35

Sonification of the step distance .......................................................................................... 36

Overall ................................................................................................................................. 38

Stats ..................................................................................................................................... 38

Results ......................................................................................................................................... 40

Set-up ...................................................................................................................................... 41

Perception tests ........................................................................................................................ 43

Vertical displacement perception ........................................................................................ 43

Tilt perception ..................................................................................................................... 44

Step distance perception ...................................................................................................... 44

Combined perception: Vertical displacement + tilt ............................................................. 45

Combined perception: Vertical displacement + tilt + step distance .................................... 46

Video ....................................................................................................................................... 46

Conclusions and further work ..................................................................................................... 47

Conclusions ............................................................................................................................. 47

Further work ............................................................................................................................ 47

Bibliography ................................................................................................................................ 49

Appendix ..................................................................................................................................... 51

List of used terms .................................................................................................................... 51

Table of figures

Figure 1 - Microsoft's Kinect (http://xboxconsole.blogspot.se) .................................................... 7

Figure 2 - Depth image from Kinect ............................................................................................. 8

Figure 3 - Tracked skeleton with Kinect ....................................................................................... 8

Figure 4 - The running cycle

(http://csmres.jmu.edu/biology/Bio490/Biomechanic%20Webposter/background.htm).............. 9

Figure 5 - Runner leaning forward and the produced resulting forces (edited from

http://wakatennis2011.blogspot.se) ............................................................................................. 10

Figure 6 - OSCeleton calibration pose (http://www.kinecthacks.com/guides/bvh-motion-

capture-guide/) ............................................................................................................................ 14

Figure 7 – Kinect audio-runner program structure. ..................................................................... 14

Figure 8 - User interface with its different buttons and sliders. It consists in four steps: 1-

Analysis of the current running technique, 2- Setting the desired thresholds for the training, 3-

Training with audio feedback and 4- Display of the results. ....................................................... 16

Figure 9 - Incoming Kinect OSC messages. A message is forwarded to different patches

depending on its header. .............................................................................................................. 17

Figure 10 - Skeleton plot and visualisation settings. ................................................................... 18

Figure 11- Input and output of the audio feedback. A local music file can be used or the line in

of the sound card can be directly read. ........................................................................................ 18

Figure 12 – Sound played once the program is started. .............................................................. 19

Figure 13 - Reset connections. Depending on what wants to be repeated different values will

need to be cleared. ....................................................................................................................... 19

Figure 14 - Record and play data Pd patch. The OSC messages are changed to fit the format of

the qlist function. ......................................................................................................................... 21

Figure 15 - draw-skeleton Pd patch. It creates the skeleton in the 3D space and also computes

the three analysed parameters of the training. ............................................................................. 22

Figure 16 - Filtering and de-normalization for the vertical displacement of the CG .................. 23

Figure 17 - Example of absolute vertical displacement with noise ............................................. 23

Figure 18 - Finding minimum values of the vertical displacement ............................................. 24

Figure 19 - Peak detection points ................................................................................................ 24

Figure 20 – Example of relative vertical displacement with noise ............................................. 25

Figure 21 – Peak detection for the vertical displacement ........................................................... 25

Figure 22 - Example of vertical displacement with step maximum values ................................. 25

Figure 23 - Transition detection for the vertical displacement ................................................... 26

Figure 24 - Example of vertical displacement with detected transitions..................................... 26

Figure 25 – Error corrector for vertical displacement ................................................................. 27

Figure 26 - Example of vertical displacement processed for sonification .................................. 27

Figure 27 – Filtering and de-normalization of the step distance data ......................................... 28

Figure 28 – Maximum feet depth detection ................................................................................ 28

Figure 29 - Example of maximum foot depth detection ............................................................. 29

Figure 30 - Average foot distance values calculation ................................................................. 29

Figure 31 - Filtering and de-normalization of the torso tilt data ................................................. 30

Figure 32 - Tilt angle calculation ................................................................................................ 30

Figure 33 - Tilting patch.............................................................................................................. 31

Figure 34 - Sonification patch ..................................................................................................... 32

Figure 35 – Vertical displacement sonification patch. The vertical displacement values are

averaged and sent to the twang patch. ......................................................................................... 33

Figure 36 - Frequency adjustment on the twang patch ............................................................... 34

Figure 37 - Volume adjustment of the twang patch .................................................................... 35

Figure 38 - Tilt sonification patch. The tilt angle determines the cutting frequency of the low

pass filter. .................................................................................................................................... 36

Figure 39 - Step distance sonification patch ............................................................................... 37

Figure 40 - Overall performance sonification patch. If all the required conditions match for a

while, an auditory icon is played to tell the runner to keep on doing like that. ........................... 38

Figure 41 - Statistics patch .......................................................................................................... 39

Figure 42 - Test in the Sport Physiology Laboratory, from the Swedish Sports Confederation . 41

Figure 43 - Kinect tracking a runner (edited from

http://www.truefitness.com/galleries/37/cs550-treadmill-images) ............................................. 42

Tables

Table 1 - Vertical displacement test parameters ......................................................................... 43

Table 2 - Tilt test parameters ....................................................................................................... 44

Table 3 - Step distance walking test parameters ......................................................................... 45

Table 4 - Step distance running test parameters .......................................................................... 45

Table 5 - VD + tilt test parameters .............................................................................................. 45

Table 6 - Complete test parameters ............................................................................................. 46

Kinect Audio-Runner: Audio feedback for improving performance in long-distance running

Jordi Bolíbar

1

Introduction

Background

In the last years, auditory display has become quite popular together with visualization in order

to present data or send feedback in different contexts. Psychological studies have concluded that

it is really intuitive for the human brain to react to sounds; therefore, the concept of sonification

has started to be used in sports and other activities that require a good body coordination and

technique. Another positive aspect about sonification is that in most sports, the sportsperson

must focus his sight on many different elements of the sport he is practicing, which makes it

almost impossible for him to pay attention at the same time to visual feedback. However, audio

feedback is way less intrusive in that sense, and most sports can be performed normally even if

the runner must be able to hear certain sounds.Sonification has already been applied in a wide

variety of fields such as: physical exercise, games, physiotherapy, etc.[1]. On the other hand,

many runners are eager to find a good technique which allows them to improve their

performance and reduce the chances of injuries at the same time; or they simply learnt by

themselves and developed some bad running habit which later became impossible to change. By

reading books or just taking some advice it is not so easy to correct your own technique, since it

requires a lot of outsight and not just focusing on a single part of the body but the whole as a

pack. That is why runners have recently been looking at new technologies that help them enjoy

running even more. A remarkable example is the new product developed by Nike and Microsoft

called “Nike+ Kinect Training”, which through the help of a motion capture sensor (the same

used in this project), the sportsperson is assisted in his training through visual and audio

feedback [21]. On the other hand, new philosophies and techniques have appeared in running,

claiming to have found the “perfect” way to run. However, there is a lot of controversy

concerning that, since experts do not seem to agree about these philosophies. Many say that they

are mere commercial products, with a clear aim focused on making money and being different

than their rivals. The two main ones are Pose running and Chi running, which share many

characteristics.In any case, there are some clear facts that will be presented later in detail, that

can simply be analysed by physics and that have been accepted by many different professionals

[2][3].


Jordi Bolíbar

2

Goals

In this project, a method is presented for aiding a long-distance runner to correct and improve

the technique in real time. By tracking and analysingthe runner’s movements and sending audio

feedback to him,a much more intuitive way to correct the pose can be achieved; thus,the runner

can adjust the pose in real time. Normally, the tracking of a runner is performed in some kind of

intrusive manner, whether placing sensors to the runner’s body or by changing somehow the

normal running procedure. In this project, the runner is tracked with Kinect, a motion sensing

camera by Microsoft which can track more than 14 joints of a human body. By using this, the

runner can run as always on a treadmill and it is still possible to track the body movements and

extract all the desired meaningful data.

Thesis content

This report is structured in the following way. After the introduction, an overview of the

previous work done is presented. Since this project assembles extremely different topics such as

psychology, biomechanics or engineering, the related work has been separated into topics.

Then, a problem formulation is stated, setting the scope and goal of the project. After this, a

series of introductory chapters explain all the necessary background about the motion sensor

used and the biomechanics of running. After this, the methodology used in this project is

explained. How the Kinect Audio-Runner program is structured, how each single problem has

been faced and how all the different parts of the software interact together. Finally, the results of

this project are shown, explaining the tests carried in the laboratory and later on, some

conclusions and useful thoughts for future work are drawn.


Jordi Bolíbar

3

Related work

Sonification and sports engineering

At the Music, Speech and Hearing department of Kungliga Tekniska högskolan, sonification

has been applied in different projects. Firstly, sonification was used for improving a runner’s

technique by analysing the vertical displacement of a runner with a smartphone’s accelerometer

[4]. Another project, focused on the improvement of performance in rowing [5]. Different

sensors were placed in the boat and the oars in order to compute different information like the

horizontal acceleration and displacement and send audio feedback to the rowers in order to

improve their technique. On the other hand, there are other projects that have worked with

sonification with runners. A study by Barrass and colleagues [6] studied the preferences by

casual runners for different types of audio feedback. This was achieved by asking runners to

carry a probe logging their preferences among six sonification models. In another study, music

was used in three types of sonification of the degree of motoric synchronization in active music

listening [7]. The interesting contribution of this study was that the sonification itself was music

that changed in different ways depending on how coordinated the movements were. Thomas

Hermann, Andy Hunt and John G. Neuhoff edited a handbook with contributions from several

authors about an introduction to sonification and auditory display. The book gathers information

concerning sonification from a wide range of fields and explains different techniques and useful

applications for it [1]. Recently, Nike and Microsoft have developed a videogame for Xbox 360

which helps people to train at home with the aid of Kinect. The game gives different challenges

which the user must complete by performing certain movements that are tracked and analysed

by Kinect [21].

Biomechanics

In the field of biomechanics and running analysis, Branko Skof and Stanko Stuhec carried a

thorough analysis of Jolanda Ceplak’s running motion[8]. She is an elite runner, holder of a

world record for 800m. The angles, displacements, speeds and accelerations of many of her

limbs and joints were captured, plotted and studied. Finally, some conclusions are extracted

about what determines her good technique and makes her run that fast. Jessica Gonowon

analyzed The Physics of Efficient Running in a web project [9]. Different aspects on how

physics affected the running technique were studied, such as forces, pendulums, the center of

mass or the pelvic rotation. Tom F. Novacheck wrote a thorough analysis about the

Biomechanics of running [10]. He summarizes most of the literature available so far about the

running gait, and compares it to walking and sprinting. Moreover, he also analyses kinetic and

kinematic data, including times of each phase, position of the different limbs or EMG in order

to know the activity of the muscles involved. Steve Magness posted a long article in his “The

Science of Running” website about how to run with proper biomechanics [11].


Jordi Bolíbar

4

He describes with a high amount of detail the process of running and gives lots of

recommendations on what to take into account when trying to improve your technique. In an

instructional video made by the Okemos Playmakers running store, some key elements of the

running technique are explained in a very intuitive way. The elements that have been taken into

account are extremely relevant for this project [12].

Psychology of perception

In psychology, it has been proved that there is a strong relationship between sounds and actions,

and it can be used to ease and help people in the process of learning something. Sonification is

basically using this in order to use audio as information. In [13], E. Kohler et. al. studied the

neuronal reactions of a set of monkeys after observing certain actions involving a sound. They

concluded that the same neurons from the monkeys discharged when they performed the action

or they heard the related sound. This enhances the idea of this project of using only sound

feedback as information. When the runner starts performing the training, he will hear sounds

that will relate to the motion of his body. After a while, once he becomes familiar with them, he

will learn how to adjust his pose in order to make them sound in an attractive way (correct

technique). After some training, if the sound feedback is removed, the runner will still be able to

run as he was taught since he will be able to anticipate the sounds that he would hear just like if

he was doing the training.


Jordi Bolíbar

5

Problem formulation

In order to carry this project, some boundaries and requirements have been specified to set its

scope. The goal is to give audio feedback to both casual and professional runners to adjust their

pose and technique in real time. There have been similar projects with sonification with

running, but they never gathered all the features included in this project. They are the following

ones:

The motion capture and body tracking are performed with a non-intrusive method

which is explained later thoroughly. Neither sensors nor any especial equipment must

be worn by the runner.

The audio feedback takes into account more than one parameter of the runner, so the

runner must interpret the multiple changes in the feedback in order to know what to

change.

Due to constraints on the distance where the tracking can be performed, the runner must

run on a treadmill. The technique is trained and learnt on the treadmill and later applied

when running on a real track.

The motion of the arms is not taken into account. Since the runner is tracked from

behind, when the arms swing in front of the chest they remain hidden. Thus, the

information provided is not reliable, plus the arms are not as important as the rest of the

analysed parameters.

The training is not based on any specific threshold values of the three analysed

elements. The program gives total flexibility to the runner and trainer to set the desired

thresholds to achieve. Therefore, it can be applied to achieve any desired running

technique or philosophy.

The training is built on background theory of long-distance running energy economy.

This means that it is aimed at average speed running, such as marathon running. It

would not apply for sprinting, where energy consumption is not the main concern and

other parameters have to be taken into account.


Jordi Bolíbar

6


Jordi Bolíbar

7

Kinect and motion tracking Before going into the details of the project, an explanation about the motion tracking system

that has been used will be given. Then, the methodology for tracking the motion of the runner’s

body will be assessed.

Kinect functionalities

Kinect (see figure 1) is a motion sensing camera, designed by Microsoft for the Xbox 360

console. It features an RGB camera, a 3D depth sensor, a motor to adjust the tilt of the camera, a

multi-array microphone and a LED which can be controlled. After some time, Microsoft

released the Kinect Software Development Kit (SDK), which allowed users to develop their

own applications in C++/CLi, C# or Visual Basic .NET. Later on, open source drivers appeared,

and PrimeSense released a motion tracking middleware called NITE. OpenNI packed the

binaries, the middleware and the hardware drivers into an installation pack, which is

multiplatform (Linux, Windows and Mac) unlike the Microsoft SDK, which was only for

Windows.

Figure 1 - Microsoft's Kinect (http://xboxconsole.blogspot.se)

The Kinect sends the tracked skeleton, RGB image and depth map (see figure 2)through the

USB cable to the computer. The different messages are treated by the OpenNI framework in

order to produce meaningful data. However, in this project, a program called OSCeleton [14]

has been used; which works as a proxy that broadcasts the skeleton data from the Kinect

middleware as OSC messages to a certain port. These messages are the ones directly read by the

software designed for this project, which will analyse the runner’s pose and send the audio

feedback. The input interface of an application called OSCeleton has been used in order to

process these messages and record and plot them.

http://xboxconsole.blogspot.se/


Jordi Bolíbar

8

Figure 2 - Depth image from Kinect

Body tracking

For this project, only the tracked skeleton information has been used. It is structured in 14 joints

(head, neck, torso, right and left shoulders, right and left elbows, right and left hands, right and

left hips, right and left knees and right and left feet) as it can be seen in figure 3. Each of these

contains its 3D position (X, Y, Z) with the Kinect as the origin of the 3D space (0, 0, 0).Once

the Kinect starts sending messages, first it will send a message when a new user (person in its

view range) is detected. After this, it will send another message when the skeleton starts being

tracked, meaning that it will start sending its coordinates soon. Finally, it will send a message

when a user is lost.

Figure 3 - Tracked skeleton with Kinect

The Kinect is able to track each of the skeleton’s joints if they are directly visible. This means

that if any joint is hidden behind an object or even the body itself, the Kinect will lose track of it

and will mostly produce a random or determined position. This has presented some troubles

when implementing the program as it will be explained in detail later. Moreover, more than one

user can be tracked at the same time; but this possibility has not been used in this project, hence

it will not be further analysed.


Jordi Bolíbar

9

Biomechanics and physics in

running In order to know what are the characteristics of a proper long-distance running technique,

answers must be found in biomechanics and physics. There are many philosophies or techniques

that are aimed to be sold as a commercial product, by selling a training pack with an infallible

method which claims to be the “perfect running technique”. These kinds of approaches do not

have a unanimous response among the elite running community, however, there are some facts

explained but physics that are widely accepted [2]. The aim of this project is not to cover all the

aspects of a good technique, but to focus on three of them and try to correct them in an intuitive

and pleasant way.

The running cycle

First of all, a little introduction to what the running cycle is will be presented, in order to

understand its different phases. The running cycle is formed by four different phases: stance,

float, swing and float again. They are depicted in Figure 4.

Figure 4 - The running

cycle(http://csmres.jmu.edu/biology/Bio490/Biomechanic%20Webposter/background.htm)

In each one of these phases there is a key element to take into account and to correct. In the

following chapter the characteristics of a good running technique that have been used for this

project are explained. These are the criterions that are applied when determining the audio

feedback that the runner receives.

What determines a good technique?

When running, ideally all the used energy must be spent in moving forward. When negative

horizontal accelerations are produced, the runner is slowed down. Since energy is also spent in

lifting the runner’s body, it is extremely important to have a pose that minimizes this energy

consumption in order to invest it on moving forward [9].

http://csmres.jmu.edu/biology/Bio490/Biomechanic%20Webposter/background.htm


Jordi Bolíbar

10

All these factors are explained by physics, and three very important ones have been chosen for

this project as the parameters to be analysed and sonified. The reason for choosing these three

and not others is that they are extremely key elements of the running cycle and at the same time

they can successfully be tracked by the Kinect. There are other interesting elements that would

be interesting to track, such as the swing of the arms. However, as it has been explained before,

the arms are hidden behind the chest thus making it impossible for the Kinect to track them.

The three parameters that are analysed are:

The tilt angle of the upper body

The angle of the body of a runner is a key aspect on how the forces are distributed in the

horizontal axis. The more the runner leans forward the more weight is put in front of the centre

of gravity. For the case of the human body, the centre of gravity is exactly the same as the

centre of mass, so from now on these two terms will be used to refer to the same thing. It is

obvious that in reality it does not happen this way, but if the weight of the torso is put before the

centre of gravity the upper body is pulled by gravity and it tends to fall, thus producing a

movement forwards (see force vectors depicted in figure 5). The runner must try to use this in

his favorin order to reduce energy consumption. If the upper body were tilted backwards, the

resulting effect would be the opposite: gravity would pull the runner backwards thus slowing

him down[9]. It is important to state that the tilt of the body must come from the ankle; it is not

enough to just run with the legs straight and tilt the torso. On this project the analysed angle

comes from the torso due technical reasons from the tracking, but unless the runner runs in a

very awkward way the angle of the torso should be the same as the angle of the whole body.

Figure 5 - Runner leaning forwardand theproduced resulting forces(edited from

http://wakatennis2011.blogspot.se)

The further the runner leans forward the faster he will be able to run. As the distance to the

centre of gravity increases, the force produced by gravity also increases. This resulting

magnitude can be calculated using equation 1:

𝐹𝑔𝑟𝑎𝑣𝑖𝑡𝑦 =𝑔 ∙ 𝑀 ∙ 𝑚

𝑑 (1)

http://wakatennis2011.blogspot.se/


Jordi Bolíbar

11

Where: g is the gravitational constant (9.81 𝑚/𝑠2 on Earth), M is the mass of the Earth, m is the

mass of the runner and d is the distance between them.

This statements stands since the more the runner leans forward, the closer his centre of gravity

becomes to the floor (smaller d) and therefore he is pulled with a greater force [9].

The distance between the landing foot and the projection to the ground

of the centre of gravity

Another extremely important aspect on the running cycle, closely related with the previous one,

is the distance between the landing foot and the projection to the ground of the centre of gravity

during the stance phase. If the foot lands in front of the centre of mass, it represents an opposing

force to the forward movement, thus slowing down the runner. Landing behind the centre of

mass can be quite awkward and can make the runner lose balance. The best thing to do is to

land the foot right under the centre of mass or as close as possible to it. This makes the whole

stance phase much natural and helps to relax the ankle and calf muscles, since they do not have

to hold and stop the runner’s weight for that long [9][12]. The legs must flow under the torso of

the runner, following the motion and inertia.

The vertical displacement of the runner’s centre of gravity

A key aspect when running, is trying to minimize the vertical displacement of the runner. The

energy must be spent in moving forward, not upward. However, the goal is not to run with a

completely null vertical displacement. Every time the foot lands on the ground, a force is

produced when the leg extends, thus pushing the runner forward. As if it were a cannonball,

there is a trajectory that allows reaching the furthest point possible. This means that the vertical

and horizontal component of the stride must be optimized for every gait in order to achieve the

longest possible distance [3].If the vertical displacement is reduced, this also means that the

impact suffered by the feet and legs is mitigated thus decreasing the chances of getting injured.

As it was shown in a previous study from the same department in KTH as the one from this

project, there is a direct relationship between the vertical displacement of the runner’s centre of

gravity and the energy consumption [4]. Therefore, this represents an extremely important

element in running economy.


Jordi Bolíbar

12


Jordi Bolíbar

13

Methodology

Installing the drivers

First of all, in order to get Kinect working on a PC, a series of drivers and middleware must be

installed. They have been developed by PrimeSense and they consist on OpenNI and NITE. It is

possible to install them separately from the OpenNI website, but for this project an auto installer

was used which included all of them in the same installer [20]. This version is called

“BrekelOpenNIKinect Auto Installer – Developer Editionv1.5.4.0.exe”and after installing it the

Kinect is ready to be used. Moreover, a software developed by the same people was

installed;the “BrekelKinect Setup v0.50.exe”. It gives a visual interface where the depth image

from the Kinect, the RGB image and the tracked skeleton can be seen at the same time. It also

provides you with functionalities for recording audio and tracked movements. Nonetheless,

these were not used for this project and this software was mainly used to make sure that the

Kinect was correctly placed and the runner did not go out from the screen while running during

the initial set-up.

Pure data program

In this section the different steps followed in the development of the program called Kinect

Audio-Runner will be shown in a structured way. This project has been developed in Pure data

(Pd) [15], a visual programming language aimed at creating multimedia works. Pd is structured

in patches and sub-patches, which separate the different parts of a program and help to structure

and interconnect these different parts. The patches will be explained following their hierarchy.

Many of the patches used in this project have not been developed by me, since they belong to a

project called OSCeleton which has been used and further expanded with the motion

recognition and sonification algorithms. Before executing the Pd code and starting to track the

runner, the OSCeleton proxy must be executed in order to send the Kinect messages to the port

7110 of the computer. Once OSCeleton has been executed, it will start looking for users. In

order to correctly track a user, the runner must adopt the psi pose, as it is shown in Figure 6.

http://www.brekel.com/release/Brekel%20OpenNI%20Kinect%20Auto%20Installer%20-%20Developer%20Edition%20v1.5.4.0.exe

http://www.brekel.com/release/Brekel%20Kinect%20Setup%20v0.50.exe


Jordi Bolíbar

14

Figure 6 - OSCeleton calibration pose (http://www.kinecthacks.com/guides/bvh-motion-capture-guide/)

Once the runner is detected, he will be able to start running normally and all the skeleton

information will be correctly received by the Pd code. The Pd code is composed by different

patches, structured in the following hierarchy depicted in Figure 7.

Figure 7–Kinect audio-runner program structure.

Interface

This patch is the interface which the user uses in order to interact with Kinect Audio-Runner. It

consists on different buttons, toggles and sliders which control the different sections of the

program and display some information in a graphical way. It also displays the instructions in

different steps in order to carry on with the sonification training for a runner.

Inte

rfac

e

Audiorun

sonification

vd_sonification twang

step_sonification

tilt_sonification

stats

overall

draw-skeleton

vertical_disp

tilting

step_dist

connection

joint

record data

results

http://www.kinecthacks.com/guides/bvh-motion-capture-guide/


Jordi Bolíbar

15

The different steps are:

1. Running analysis: The runner runs for a desired time and the three different parameters

of his running technique are analysed and stored. No audio feedback will be given

during this test period. The test button must be pressed again in order to stop the test.

2. Setting thresholds: Once the mean value of the three different parameters has been

computed, a threshold of improvement can be set in order to perform the sonification

training. The lower the threshold is for the vertical displacement and the step distance,

the stricter the sonification will become. For the tilt angle case, the higher the threshold

is set, the more demanding will be the training. When changing this value all the

sonification thresholds are adjusted in consequence.

3. Start the sonification training: Once the thresholds have been set, the runner can start

the training. It is possible to select which of the three elements want to be trained in

order to train any combination of them. The music will start as well as the vertical

displacement audio feedback and the good technique auditory icon. While this mode is

selected, the runner will receive audio feedback depending on these movements and the

previously specified thresholds.

4. Display results: After concluding the training, a series of results are displayed in the

interface for the runner to check on his performance and in order to see how much he

improved.

Moreover, it also allows displaying or hiding the tracked skeleton, checking the three analysed

parameters with sliders, to play, stop or adjust the volume of the music file being played and to

reset all the variables in order to sonify again. The user interface is shown in figure 8.


Jordi Bolíbar

16

Figure 8 - User interface with its different buttons and sliders. It consists in four steps: 1- Analysis of the

current running technique, 2- Setting the desired thresholds for the training, 3- Training with audio

feedback and 4- Display of the results.

Audiorun

This patch is responsible to gather all the other patches and sub-patches. In some ways it is

similar to the Interface but without buttons and toggles and presented in way it would not be

understood by the end user.

However, its main purpose is to take the incoming OSC messages broadcasted by OSCeleton

from the Kinect and send them to the draw-skeleton patch and to the record data patch in case

the movements of the runner want to be recorded for further use. The OSC messages are

received through the port 7110. Once a new user has been discovered and the body is already

being tracked, the received messages have the following structure:

Address pattern: "/joint"

Type tag: "sifff"

s: Joint name, check out the full list of joints below.

i: The ID of the user.

f: X coordinate of joint in interval [0.0, 1.0]

f: Y coordinate of joint in interval [0.0, 1.0]

f: Z coordinate of joint in interval [0.0, 7.0]

For example:

/joint r_foot 1 0.577044 0.812729 2.79039


Jordi Bolíbar

17

It can be seen that the coordinates, for the X and Y values, go from 0.0 to 1.0 and for the Z

value from 0.0 to 7.0. The reason for this is that the space where the Kinect can track a user is

much deeper than wide or tall. A user will be able to move roughly around 6 m2, between a

distance of 0.7-6 meters, whilst the horizontal field at minimum viewing distance would be

around 0.8 meters and for the vertical field around 0.6 meters [16]. That is why for the Z

component it is logical to have a wider range of values.

The Pd code responsible for this is shown in Figure 9.

Figure 9 - Incoming Kinect OSC messages. A message is forwarded to different patches depending on its

header.

The route function routes the /joint and /new_skel messages to different paths. The /joint

messages are sent to the draw-skeleton patch, where the different nodes are plotted and also

where the three parameters to analyse the technique are computed. The /new_skel messages are

used to create a new skeleton and plot it together with the previous one.

On the other hand, it is also responsible for plotting the runner’s movements on a separate

screen using a 3D space representation from Pd called Gem. Different parameters are specified

when creating the Gem window, such as: the lighting, perspective, title of the window, rotation

of the different joints, etc. The section responsible for this is shown in figure 10.


Jordi Bolíbar

18

Figure 10 - Skeleton plot and visualisation settings.

In addition, the audiorun patch also takes the incoming audio and adjusts the volume. It is

possible to use music from the input line of the computer’s sound card (for instance an iPod) or

a WAV file stored in the computer.

After this, the two audio channels are sent to the sonification patch where the music is filtered

and processed in order to give the necessary audio feedback. Once the music has been

processed, it is sent to the output line of the computer in order to be played. Moreover, it is also

possible to record the output sound into a WAV file in order to listen to it after the training. The

block is shown in figure 11.

Figure 11- Input and output of the audio feedback. A local music file can be used or the line in of the

sound card can be directly read.


Jordi Bolíbar

19

Once the program is loaded, a sound is played as an introduction. In Figure 12 it can be seen

how it is executed.

Figure 12–Sound played once the program is started.

Every time a new test or sonification training is started, all the stored values in the program

must be erased in order to start from scratch. These orders must be centralised in order to be

easily applied throughout the code as it is shown in figure 13. Once a new test is started, all the

values must be cleared. In case of wanting to run a new sonification training, only certain values

will be cleared, and the average ones which set the thresholds will be kept.

Figure 13 - Reset connections. Depending on what wants to be repeated different values will need to be

cleared.


Jordi Bolíbar

20

Record data

This patch is used in order to record the incoming messages from Kinect and also it allows

playing them once they have been stored in a .txt file.The messages are edited before storing

them, adding a time stamp to each of them thus fulfilling the format asked by the qlist function.

The messages are stored in the .txt file with the following format:

29.0249 osceleton /joint head 1 0.544386 0.0754202 3.33552;

0 osceleton /joint neck 1 0.547206 0.194634 3.27663;

0 osceleton /joint l_shoulder 1 0.606797 0.203392 3.32487;

0 osceleton /joint l_elbow 1 0.606797 0.366595 3.32487;

0 osceleton /joint l_hand 1 0.606798 0.529799 3.32487;

0 osceleton /joint r_shoulder 1 0.487615 0.185877 3.22838;

0 osceleton /joint r_elbow 1 0.487352 0.349077 3.22998;

0 osceleton /joint r_hand 1 0.503657 0.510319 3.2492;

0 osceleton /joint torso 1 0.545305 0.31582 3.23652;

0 osceleton /joint l_hip 1 0.58308 0.442837 3.22854;

0 osceleton /joint l_knee 1 0.576513 0.675919 3.05872;

0 osceleton /joint l_foot 1 0.559341 0.908464 2.93515;

0 osceleton /joint r_hip 1 0.503727 0.431176 3.1643;

0 osceleton /joint r_knee 1 0.499874 0.673481 3.31297;

0 osceleton /joint r_foot 1 0.494751 0.878755 3.4167;

The first number means the delay with respect the previous sample in milliseconds. Then it is

followed by “osceleton”, in order to be received by the atoms waiting for the symbol

“osceleton”. After this, comes the normal message originally sent by the Kinect.

The qlist function allows reading .txt files chosen from a path, and they can be played using the

time stamps (see figure 14). It also allows stopping and rewinding the set of messages. This part

was already included in the OSCeleton program and it has been really useful in order to test the

program outside the lab. Runners were recorded during the lab tests and those recordings were

later used to test the algorithms before running the proper tests again in the lab.


Jordi Bolíbar

21

Figure 14 - Record and play data Pd patch. The OSC messages are changed to fit the format of the qlist

function.

Skeleton and calculations

The draw-skeleton patch is responsible for plotting the skeleton using Gem and for computing

the three different parameters of the running technique that are analysed during the sonification

training. A 3D object is created for each joint by the joint patch, which is plotted in the 3D

space. Moreover, a line is drawn between the different joints using the connection patch. All

these patches that plot the skeleton come from the previously mentioned OSCeleton project. So

the detailed explanations will be focused on the calculations of the running technique

parameters, which are the ones created in this project.


Jordi Bolíbar

22

The original OSCeleton code with the three new blocks added is shown in figure 15.

Figure 15 - draw-skeleton Pd patch. It creates the skeleton in the 3D space and also computes the three

analysed parameters of the training.

Vertical displacement

The vertical_disp patch takes values of the joint of the torso in order to analyse the vertical

displacement of the runner. That joint is the closest one to the runner’s centre of gravity;

therefore it is the most reasonable body joint to us for this calculation. First the joints from the

torso are filtered at the input of this patch and then the vertical (Y) component is taken and

filtered with a certain cut-off frequency. The reason for doing this is to reduce some of the

tracking noise produced by the Kinect. Using a low pass filter allows to eliminate the high

frequency movements which are closely related to noise. A cut-off frequency of 5 Hz has been

used for all the tests, but it can be easily changed.

After that, since the values of the coordinates are between 0.0 and 1.0 for the X and Y

components and between 0.0 and 7.0 for the Y component, they have to be de-normalized [17].

The de-normalization functions applied are the following ones:

x = 1280 – x∙2560

y = 960 - y∙1920

z = z∙1280


Jordi Bolíbar

23

All the resulting distances are in millimeters.

The code responsible for doing this is shown in figure 16.

Figure 16 - Filtering and de-normalization for the vertical displacement of the CG

After this, in order to calculate the vertical displacement of the runner, the difference between

the maximum and minimum values of the torso joint must be computed. Since these may vary at

every step, they must be computed in a way that they update themselves automatically.

Moreover, the algorithm must be sure to be looking at an absolute minimum value of a period,

since noise can alter the original shape of the motion and can create relative minimum values

that do not coincide with the absolute one. An example of this can be seen in figure 17, where

the sinus-like shape of the vertical displacement can be seen, but it still has some noise after

being filtered with a 5Hz low pass filter.

Figure 17 - Example of absolute vertical displacement with noise

The solution to this problem is to use a hysteresis window which leaves a margin of time to

check if that value is really an absolute minimum value. For this case, four different values in

different time intervals have been taken and all of them are checked in order to be sure that that

value fulfills the requirements (see figure 18).

0

20

40

60

80

100

120

140

160

1 16 31 46 61 76 91 106 121 136 151 166 181 196 211

Ve

rtic

al d

isp

lace

me

nt

(mm

)

Samples


Jordi Bolíbar

24

Figure 18 - Finding minimum values of the vertical displacement

This method is widely used in this project with both maximum and minimum values, so it will

be presented more in detail by means of pseudo-code. A combination of five different points is

searched in the incoming values. Once a match is found that fulfills the requirements, that value

is taken as a maximum or minimum. In order to make the pseudo-code easier to understand, the

five points shown in figure 19 will be used.

Figure 19 - Peak detection points

For detecting maximum values:

if 1 is smaller than 2 and 2 is smaller than 3 and 4 is smaller than 3

and 5 is smaller than 4

3 is a maximum value

For detecting minimum values:

If 1 is bigger than 2 and 2 is bigger than 3 and 4 is bigger than 3

and 5 is bigger than 4

3 is a minimum value


Jordi Bolíbar

25

Once the minimum value of a period is known, it is subtracted from the rest of the incoming

values so all the positions are relative to the minimum vertical displacement. After this, the

same operation carried in order to find the minimum values is carried for the maximum values.

The incoming data looks like in figure 20.

Figure 20 – Example of relative vertical displacement with noise

Therefore, if the peaks are extracted, the maximum displacement for each period is known.

When a maximum value is found, the value is held until another maximum value is found (see

figure 21).

Figure 21–Peak detection for the vertical displacement

After detecting the maximum values and holding them, the output looks like in Figure 22.

Figure 22 - Example of vertical displacement with step maximum values

0

20

40

60

80

100

1 9

17

25

33

41

49

57

65

73

81

89

97

10

5

11

3

12

1

12

9

13

7

14

5

15

3

16

1

16

9

17

7

18

5

19

3

20

1

20

9

21

7

Ve

rtic

al d

isp

lace

me

nt

(mm

)

Samples

0

20

40

60

80

100

1 81

52

22

93

64

35

05

76

47

17

88

59

29

91

06

11

31

20

12

71

34

14

11

48

15

51

62

16

91

76

18

31

90

19

72

04

21

12

18

Ve

rtic

al d

isp

lace

me

nt

(mm

)

Samples


Jordi Bolíbar

26

After having the maximum values, these still must be treated a bit in order to make them more

reliable. First of all, there are small transitions between the values that are held, so the single

values need to be taken in order to process them in a more comfortable way. In order to do this,

a transition is detected using the following condition shown in Figure 23.

Figure 23 - Transition detection for the vertical displacement

The output for this module is shown in Figure 24.

Figure 24 - Example of vertical displacement with detected transitions

It can be seen that there is still some noise present in the signal, like in the peak of 200 mm. But

since all the values will be averaged using windows in the sonification patch, this kind of errors

will be mitigated. After this, these values are inserted to the following module, which avoids the

insertion of noise into the data by reducing extreme values which differ too much from the rest.

For instance, the two peaks that can be seen in Figure 15 of around 200 mm will be reduced

since they clearly represent noise and not the movement of the runner. The total number of

samples has been reduced since only the transitions are accepted. These samples are processed

with the blocks presented in Figure 25. The aim of these functions is to avoid huge differences

between vertical displacement values. Since noise produces sometimes extreme values, they are

cancelled by producing just a small increase or decrease in the next incoming value, depending

on if the value is higher or lower.

0

50

100

150

200

250

1 7

13

19

25

31

37

43

49

55

61

67

73

79

85

91

97

10

3

10

9

11

5

12

1

12

7

13

3

13

9

14

5

15

1

Ve

rtic

al d

isp

lace

me

nt

(mm

)

Samples


Jordi Bolíbar

27

Figure 25 – Errorcorrector for vertical displacement

Therefore, the values sent to the sonification patch which have already been adjusted will look

like in Figure 26.

Figure 26 - Example of vertical displacement processed for sonification

Step distance

The step_dist patch calculates the distance between the landing foot and the centre of gravity of

the runner and averages the distance for both feet every cycle. First of all, the coordinates for

the right and left feet and the torso are separated. Then, the values for the Z position of the feet

and the Z position of the torso are low pass filtered as always in order to reduce incoming noise

from the Kinect tracking. After this, the values are de-normalized as it has been done previously

with the vertical displacement (see figure 27).

0

20

40

60

80

100

120

140

1 7

13

19

25

31

37

43

49

55

61

67

73

79

85

91

97

10

3

10

9

11

5

12

1

12

7

13

3

13

9

14

5

Ve

rtic

al d

isp

lace

me

nt

(mm

)

Samples


Jordi Bolíbar

28

Figure 27 – Filtering and de-normalization of the step distance data

Initially, the intention was to calculate the step distance by checking when the foot was at a very

close distance to the floor, and then compare the depths of the landing foot and the CG.

However, this proved to be extremely noisy, since the Kinect has a maximum error of around 3-

4 cm, which made that every time the foot swung down, a wide range of values were interpreted

as foot contact. In order to solve this and use a more robust method, the maximum depth of the

feet has been computed. This value of course is slightly superior as the foot contact, since the

foot always swings back a little bit before touching the floor, but it is completely proportional to

the step distance and therefore is still totally useful for this application.

A maximum peak detector has been applied to the difference between the depth of the feet and

the depth of the CG. The maximum depth values of both feet are averaged and sent to the

sonification patch. Moreover, the total average value is computed in order to be used for the test

part and threshold selection. The corresponding code can be seen in figure 28.

Figure 28–Maximum feet depth detection


Jordi Bolíbar

29

The difference between the foot depth and the CG depth with the peak detection applied looks

like in figure 29.

Figure 29 - Example of maximum foot depth detection

The average of both feet and the total average value are computed as it is shown in figure 30.

Figure 30 - Average foot distance values calculation

-800

-600

-400

-200

0

200

1

25

49

73

97

12

1

14

5

16

9

19

3

21

7

24

1

26

5

28

9

31

3

33

7

36

1

38

5

40

9

43

3

45

7

48

1

50

5

52

9

55

3

57

7

60

1

62

5

64

9

Dis

tan

ce (

mm

)

Samples

Step distance - CGStep distance


Jordi Bolíbar

30

Tilt of the torso

The tilting patch calculates the tilt of the upper body with respect the floor. The coordinates of

the torso and neck joints are taken and they are filtered and de-normalized as it has done for the

previous patches (see figure 31).

Figure 31 - Filtering and de-normalization of the torso tilt data

Once the values of the height and depth of the torso and neck joints have been de-normalized

and filtered, the difference in height and depth is computed. These two values represent two

sides of a triangle, and the angle that they form is the angle between the vertical projection from

the floor and the runner’s back (see figure 32). With these two values, the tangent of the angle is

calculated, and from this, the inverse tangent is applied in order to know the resulting angle in

radians. Then this angle is converted to degrees and sent to the sonification patch.

Figure 32 - Tilt angle calculation


Jordi Bolíbar

31

As it has been done in the step distance patch, the overall average angle is computed in order to

use it for running test. The code responsible for this is depicted in figure 33.

Figure 33 - Tilting patch

Sonification

The sonification patch takes the incoming audio signal chosen for the training and changes it

accordingly to the received data produced by the runner’s movements in order to create the

audio feedback. First of all, the patch has four main different sonifications: the step distance, the

vertical displacement, the tilt of the upper body and the overall performance. These four parts

work together sharing certain information and especially they use the same time reference,

which is the frequency of the incoming messages For each one of them a sub-patch has been

created in order to ease the navigation and structure of the program. Moreover, a sub-patch for

the calculation of statistics of the runner’s performance has been included too.

In Figure 34 the structure of the sonification patch is presented. On the left there is a timer based

on the input samples coming from the Kinect. It was developed due to delays and non-stability

of the Pd metronomes when running the whole program. The time is referenced depending on

the incoming Pd messages, which makes the synchronization easier. It is used in order to signal

when to restart the current average value used for the sonification. By default it is set to 100

samples, which means around 4 seconds (since most samples are sampled at 25 Hz). This means

that every 100 samples will be averaged and used as the current value for the sonification,

which helps to reduce the processing time of the program. If the cutting frequency of the filters

were to be changed in average 25 times per second it would require a much higher processing

time than if it is performed around every 4 seconds. The cut-off frequency for the low pass filter

can go from 100 to 20000 Hz. For the high pass filter, it goes from 5 to 5000 Hz.


Jordi Bolíbar

32

The tilt and step distance sonification patches are connected to their respective filters and the

vertical displacement one is connected to the twang patch which generates the bouncing sound.

Moreover, if the specific button for each one of the three sonified parameters is not pressed, the

values are blocked before reaching the filter in order not to apply them.

Figure 34 - Sonification patch

Each sub-patch will be analysed separately, focusing on the element that it is sonifying.

Sonification of the vertical displacement

The vd_sonification sub-patch (see figure 35) receives the computed values of the vertical

displacement from the sub-patch inside the draw-skeleton patch and filters the incoming music

file accordingly. On the left side of the patch, the total average value of the vertical

displacement is computed. This value is used for the test that the runner does before starting the

actual training. With this average value a threshold will be set which will determine the training.

When the test button is pressed and the sonification one is not, then the average value is stored

for setting the thresholds. Once the sonification starts and the test button is not pressed and the

sonification one is, the average value is not modified. Moreover, every time that the retest

button or another test is run, the average value is erased in order to compute it once again from

scratch.


Jordi Bolíbar

33

Figure 35–Vertical displacement sonification patch. The vertical displacement values are averaged and

sent to the twang patch.

On the right side of the patch, the incoming values are averaged every 100 samples using the

time flags sent from the sonification patch. This averaged value is held until the next flag

comes, and it is sent to a transition detector.

It is actually needed because the transitions between average values are not instantaneous, so

there are some values in between which need to be ignored.

Besides this, a condition is also set for the value to be sent to the twang patch only if the

sonification button is pressed, the test button is not pressed and the sonification of the vertical

displacement has been chosen on the interface. If all these conditions are fulfilled, the value is

sent to the twang patch which will produce the bouncing sound feedback as it is explained in the

next section. Otherwise, a zero value is sent which will be stopped by the spigot box.

Twang

This patch has been taken from the code examples from the “Designing sound” textbook,

included in a compilation in The MIT Press website [18]. When an inlet is banged, a boingy

sound is produced. The volume, frequency and vibrato of it can be adjusted through different

inputs. Since this patch has not been developed for this project, all further explanations will be


Jordi Bolíbar

34

avoided and they will only be focused on explaining how the incoming vertical displacement

values are scaled in order to produce the audio feedback.

First of all, the frequency of the boingy sound is adjusted depending on the values of the vertical

displacement. The higher the vertical displacement is the higher will be the frequency of the

sound that the runner will hear. This helps to intensify the feeling that he is bouncing too much.

In order to do this, a function has been created in order to scale the frequency.

This function takes into account the threshold that has been set for the training. A constant value

is multiplied by the current vertical displacement value received and the threshold. This will

mean that if the runner goes over that threshold the constant value will be increased, and if he

goes under it, the constant value will be reduced. For values different to zero, the frequency will

be determined by formula 2.

𝑓 = 600 𝑣𝑑

𝑣𝑑𝑡𝑕 (2)

Where 𝑣𝑑is the current vertical displacement and 𝑣𝑑𝑡𝑕 is the threshold value for the vertical

displacement.

On the top left of Figure 36, it can be seen that when a message arrives to the inlet, it triggers

the boingy sound.

Figure 36 - Frequency adjustment on the twang patch

The same procedure done with the frequency is applied to the volume. Before all the samples

reach the sound output, they are multiplied by a value between 0 and 1 in order to adjust the

volume. In this case, the function is divided into two different ranges of values. If the current

vertical displacement value is over the specified threshold, the sound will be played at

maximum volume (1). For the values lower than the threshold, a function is applied where the

bigger is the difference between these two values, the smaller the volume will get. This is done

by dividing the input value by the subtraction of itself with the threshold. This kind of function

helps to make the transition over the threshold really obvious for the runner. As soon as he gets

close to the threshold the volume will increase a lot and vice versa. If more intermediate values

were used, it could be quite confusing for the runner, since he is supposed to listen to many

different sorts of audio feedback at the same time. Therefore, if the response of the audio

feedback becomes obvious and easy to understand, the chances for it to be useful to the runner

will clearly increase.


Jordi Bolíbar

35

Formula 3 determines the volume of the auditory icon.

𝑣𝑜𝑙𝑢𝑚𝑒 =

𝑣𝑑𝑡𝑕 − 𝑣𝑑 < 0 ; 1

𝑣𝑑𝑡𝑕 − 𝑣𝑑 > 0 ; 𝑣𝑑

𝑣𝑑𝑡𝑕 − 𝑣𝑑 ∙ 200 (3)

Where 𝑣𝑑 is the current vertical displacement and 𝑣𝑑𝑡𝑕 is the threshold set for the vertical

displacement.

The part of the program that applies this formula is shown in figure 37.

Figure 37 - Volume adjustment of the twang patch

With these two parameters sonified, the boingy sound is played over the music file in a periodic

fashion, aiding the runner to understand his vertical displacement.

Sonification of the tilt

The tilt of the upper body is sonified by using a low pass filter that cuts the high frequencies of

the song when the runner does not tilt his body enough compared to the specified threshold.

The tilt sonification averages the incoming values in a periodic way as it is done with the

vertical displacement sonification patch. For every averaged value, a function is applied to it in

order to turn the angle in degrees into a value in hertz which will change the cutting frequency

of the filter. This function uses the same idea as for the one used for the volume of the boingy

sound. If the angle is over the threshold, the cutting frequency is set to 20000 Hz, which is the

maximum frequency that a human ear can hear. However, if the angle is under the threshold, the

maximum frequency (20000 Hz) will be reduced depending on the difference between the

current angle and the specified threshold. This type of function will help to make the transitions

really abrupt. Therefore, as soon as the runner has an angle under the threshold, he will hear that

the music changes dramatically.


Jordi Bolíbar

36

In 4 the formula responsible for converting the angle into a cutting frequency for the filter can

be seen.

𝑓𝑐 =

𝛼𝑡𝑕 − 𝛼 < 0 ; 20 ∙ 103

𝛼𝑡𝑕 − 𝛼 > 0 ;2000

𝛼𝑡𝑕 − 𝛼

(4)

Where 𝛼 is the current tilt angle of the body and 𝛼𝑡𝑕 is the threshold for the tilt angle of the

body.

The sonification patch for the tilt can be seen in figure 38.

Figure 38 - Tilt sonification patch. The tilt angle determines the cutting frequency of the low pass filter.

Sonification of the step distance

The step distance between the foot and the projection of the centre of gravity is sonified using a

high pass filter that cuts the low frequencies of the song whenever the runner performs over the

specified threshold.

Once again, for this sub-patch (see figure 39) the values are also averaged periodically, but this

time a counter has been used which helps to produce a new average value when four values of

foot distance have been received. However, the applied function to turn the step distance values

into hertz works in the exact same way too.


Jordi Bolíbar

37

Figure 39 - Step distance sonification patch

In 5 the formula that converts the distance in mm from the step into the cutting frequency of the

filter is shown.

𝑓𝑐 =

𝑠𝑑𝑡𝑕 − 𝑠𝑑 + 10 < 0 ; 5 ∙ 103

𝑠𝑑𝑡𝑕 − 𝑠𝑑 + 10 > 0 ;3 ∙ 103

𝑠𝑑𝑡𝑕 − 𝑠𝑑 + 10

(5)

Where 𝑠𝑑 is the current step distance and 𝑠𝑑𝑡𝑕 is the threshold for step distance of the training.

If the step distance surpasses the threshold for more than 1 cm, the cutting frequency will be set

to 5000 Hz, which is the maximum frequency for this filter. This will mean that all frequencies

under 5000 Hz (mid-low and low frequencies) will be removed from the song. Once the step

distance goes under the threshold, a constant value is divided by the difference between this

current value and the threshold. This will produce the same effect as for the other two

sonification patches. As soon as the runner goes under the threshold, the sound will improve

dramatically. However, if he gets close to it, the sound will be quickly filtered and only high

frequencies will remain on the song.


Jordi Bolíbar

38

Overall

The overall patch (see figure 40) is responsible for telling the runner if he has been performing

correctly according to the specified threshold for a certain amount of time. Depending on which

of the three elements have been chosen for the sonification training, if the sonification button is

pressed and the chosen elements are fulfilling the specified thresholds, a counting will start. If

10000 samples are received (a little bit more than 10 seconds) under these conditions, a sound

will be played over the music to tell the runner than he is performing in the correct way.

Figure 40 - Overall performance sonification patch. If all the required conditions match for a while, an

auditory icon is played to tell the runner to keep on doing like that.

This is intended to ease the training for the runner. Once the runner is performing in the right

way, he will hear the music correctly with no artefacts or anomalies; but after doing it right for a

while, this auditory icon will tell him that he is on the right track and he just must keep up the

current pose to eventually memorise it.

Stats

This sub-patch calculates the percentage of time that the runner is fulfilling the desired

thresholds. However, an error threshold can be set in order to take into account also values that

approach the desired threshold. For example, if an error threshold of 10% is chosen, even if the

runner performs 10% more over the chosen sonification threshold it still count as if he were

performing correctly. In order to do it, the current values of the vertical displacement, the step

distance and the tilt are compared to the specified thresholds, and if they fulfil them they are


Jordi Bolíbar

39

counted as successful samples. Then, these samples are divided over the total amount of

samples, thus having the ratio of correct samples.

All these calculations are sent to the results patch, where they are displayed in a more intuitive

way. The patch is shown in figure 41.

Figure 41 - Statistics patch


Jordi Bolíbar

40


Jordi Bolíbar

41

Results In this chapter, the different experiments and tests that have been performed with runners will

be exposed, as well as the set-up used. All the tests have been done in the Sport Physiology

Laboratory in Elite Performance Centre of the Swedish Sports Confederation, in Lidingö

(Stockholm).

Set-up

For all the tests, the following set-up has been used (also shown in figure 42):

The Kinect is mounted on a tripod with the help of a tripod mount. It must be perfectly

aligned to the runner’s axis. This means that it must be completely horizontal with

respect the floor and forming a 90º angle when facing the treadmill. This will assure

that the coordinates fulfil the program’s requirements. Moreover, it must be placed at

the rear of the treadmill, at a distance of approximately 2 meters from the runner (the

runner will move forward and backward while running anyway). However, for a totally

correct placement it is better to check first the RGB image of the Kinect in order to be

sure that the runner fits completely in the screen.

The Kinect is connected to a power source and to a laptop by means of a USB cable.

The laptop has stereo 40W speakers connected to it and it is placed on a table next to

the Kinect. The speakers are aimed at the runner at a considerably high volume in order

to be heard over the treadmill noise.

The room where the tests are run must be lit with artificial light, and sunlight from the

windows must be avoided. Covering the windows with blinds is the best solution [19].

Figure 42 - Test in the Sport Physiology Laboratory, from the Swedish Sports Confederation


Jordi Bolíbar

42

Once everything is set up, the Pd program can be started by opening the Interface patch. After

this, it is recommended to press the skeleton button and then to open the OSCeleton application

in order to start tracking the runner. To do so, he must stand in front of the Kinect facing it, in

the same spot he will run. Moreover, he must adopt the psi posed as it has been previously

explained until OSCeleton says that the user is being tracked correctly. In order to check that the

tracking works fine, the user’s skeleton is shown in the skeleton window.

After this, the runner must start running in the same way as he normally does, in order to

analyse his current technique. However, it is highly recommendable to start running with the

legs slightly more separated than normal for the Kinect to track the movements better. After 5-

10 seconds of proper tracking, the runner can run as always and the Kinect will still track the

legs correctly. Once the runner is already running, the test button must be pressed and all

average values for the three sonified parameters will be computed. After a certain amount of

time (determined by the person who runs the test) which should be enough to have a meaningful

set of values, the test button can be pressed again in order to finish the test. Then, the average

values are displayed on the interface; and using a slider, they can be modified in order to set the

desired thresholds for the three analysed parameters that the runner should achieve during the

training.

Once the test part has concluded, the runner can start the training and he will hear the audio

feedback that will react depending on his position compared to the desired thresholds. The set-

up looks like in figure 43.

Figure 43 - Kinect tracking a runner(edited from http://www.truefitness.com/galleries/37/cs550-

treadmill-images)

During the training the runner should try to focus on making the music sound well. This means

that the audio feedback will tell him that he is running in the desired way when he will not hear

any artefacts or changes in the music. In order to finish the training, the sonification button must

be pressed again, which will end the audio feedback and will stop the computation of results.

http://www.truefitness.com/galleries/37/cs550-treadmill-images

http://www.truefitness.com/galleries/37/cs550-treadmill-images


Jordi Bolíbar

43

After this, the results button can be pressed, which will show the percentage of time the runner

has been complying with every one of the analysed elements. Furthermore, the overall

percentage of time that he has been fulfilling the three of them at the same time is also

displayed.

Perception tests

A series of tests were carried, for checking if the runner was able to understand the audio

feedback to the movements of his body and get to know where the threshold was in order to

change the sound in a controlled manner. The runners were asked to be able to go over and

under the threshold whenever they wanted in order to change the music and auditory icons at

will. The goal was not to actually test if Kinect Audio-Runner was able to improve someone’s

technique but to see if it could successfully interact with a runner. Each one of the three sonified

elements were tested separately and then in a combined way. The vertical displacement and the

torso tilt were tested by three different casual runners and the step distance by one (due to some

problems on the implementation on the early tests). Each one of the tests will be commented

separately.

Vertical displacement perception

For this test, the runners were asked to run for a while on the treadmill in order to measure their

average vertical displacement using the test mode of the program. Then, the threshold was set

slightly underneath their average vertical displacement and they were asked to sense at which

point the threshold was and to be able to run for as long as they wanted both under and over the

threshold. It did not prove difficult for the runners, except for some very specific situations

where the tracking did not work as it was supposed to. At the beginning, they had to adapt to the

fact that the boingy sound takes some time to adapt to the current vertical displacement of the

runner, since the values are averaged in order to make it more robust. However, all of them

were very pleased with the way it worked and were perfectly capable of controlling the boingy

sound at will. An example of a recorded test is shown in the demonstration video.

The parameters and thresholds used were:

Table 1 - Vertical displacement test parameters

Treadmill speed Max.vertical displacement Max. step distance Min. torso tilt

8 km/h 10 cm - -


Jordi Bolíbar

44

Tilt perception

The tests for the tilt angle were the most successful ones. The accuracy of the reaction of the

sound was really good and the runners had absolutely no problems when trying to make the

music sound well or not.

As with the vertical displacement, the reaction was not extremely fast, since the values were

averaged and it took a few seconds for the sound to adapt, but that did not prove as a problem.

An important thing to mention is that the runners had to be told not to tilt only their upper body;

the tilt should come from the ankle [12]. If a large angle was set, the only way to achieve it

correctly was increasing the speed. At low speeds it was not possible for the runners to tilt that

much in a correct way, they could only do it by tilting only the torso. An example of a recorded

test can also be found in the demonstration video.


Table 2 - Tilt test parameters


8 km/h - - 15º

Step distance perception

In order to analyse the sonification of the step distance, two different tests were carried by the

third runner. First, he had to walk on the treadmill trying to make strides of different lengths. He

was asked to be able to go over or under the set threshold whenever he wanted. The results

turned to be successful, and the runner was able to make the music change (cutting the high

frequencies) whenever he made long strides over the determined threshold (6cm). Compared to

the other two sonifications, it took a slightly longer time for the audio feedback to adapt due to

averaging computations of the step distances. However, it hardly ever took more than 3-4

seconds to adapt, which did not produce any troubles to the runner. After the walking test, he

was asked to do the same but running.

For the running case, the distance between the vertical projection of the CG and the landing foot

was much smaller than for the walking case. This is due to the fact that the runner tilts forward

in order to run, thus making the distance to the landing foot much shorter. When the runner was

asked to walk normally the step distance was quite similar to the when he ran, but he was able

to make much longer strides while walking which could easily surpass any threshold used when

running. Therefore, the thresholds for the step distance were different than the ones from the

walking test as it can be seen in the tables 3 and 4.


Jordi Bolíbar

45

Table 3 - Step distance walking test parameters


4,5 km/h - 6 cm -

Table 4 - Step distance running test parameters


9 km/h - 4 cm -

Combined perception: Vertical displacement + tilt

This test was run after the runners had tried both of the sonifications individually. All the three

runners coincided in the way of facing it and they tried to deal with the sonified elements one by

one. Since it is quite hard to tilt with a higher vertical displacement, they first tried to reduce

their vertical displacement by making the boingy sound disappear while the music could not be

heard correctly. After finding the threshold and getting to know at which maximum vertical

displacement they were allowed to run, they focused on tilting forward until the music became

normal and the high frequencies could be heard again. Again, like for the case of the individual

tests, the thresholds were set slightly underneath their average values acquired during the test

part. The tilt angle of the runners was set at 15º, an intermediate angle that made it easy for

them to make the music fade if they run quite straight and when they tilted and ran correctly the

music started to sound correctly really fast. For the vertical displacement, the values changed a

little bit more. For two runners, the vertical displacement was set at 9 cm. If they intended to run

with a notorious bouncing, the boingy sound appeared easily, but if they focused on maintaining

a low vertical displacement without becoming awkward they managed really easily to make the

boingy sound disappear. A recorded example of this type of test is shown in the demonstration

video too.


Table 5 - VD + tilt test parameters


10 km/h 9 cm - 15º


Jordi Bolíbar

46

Combined perception: Vertical displacement + tilt + step distance

After successfully testing the combination of the vertical displacement and the tilt, all the three

parameters where analysed at the same time. For that, the third runner was asked to wilfully

produce any combination of parameters (being over or under the threshold) in order to check if

the audio feedback responded correctly and adjusted to the runner’s pose. Once more, it proved

easier for him to deal with them one at a time.

In the end, he managed to control the audio as he wanted, despite that sometimes the audio

feedback reacted with a delay of around 3-4 seconds. Some combinations proved to be quite

difficult and awkward to achieve, since they were not natural positions that a runner would

attempt, but still he managed to produce them. The results can be seen in the video too, where

the whole experiment was recorded.

The parameters used for this experiment can be seen in table 47.

Table 6 - Complete test parameters


9 km/h 10 cm 5 cm 15º

Video

As it has been previously mentioned, a video which shows the basic functionalities of the

program, as well as the different steps to follow and the perception tests has been developed.

Since the project contains a high amount of audio-visual content, it is much more intuitive and

logical to show the results through a video rather than just describing how the music or the

training sound or look like. The video has been uploaded to Vimeo for everybody to check. The

url to the video is the following one:

https://vimeo.com/42772997

https://vimeo.com/42772997


Jordi Bolíbar

47

Conclusions and further work

Conclusions

The tests with real runners confirmed that the initial goals of the project have been achieved. A

program has been developed which allows runners to analyse their current technique and change

certain elements (the vertical displacement of the CG, the tilt of the torso and the distance

between the projection of the CG to the floor and the landing foot). The feedback to the runner

is provided with audio consisting in both real-time filtered music and auditory icons. The runner

is able to identify the mistakes on his/her pose by just listening to the audio feedback. The

response from the runners has been good and they have stated that they enjoyed the training as

well as the whole concept of the project.

The response of the audio feedback has proved to be robust in most cases. The code is able to

deal with different situations and noisy data from the Kinect tracking. However, still some

problems may occur from time to time due to the difficulties for Kinect to track the runner from

behind. Kinect was initially designed to track a person from the front, and the user is supposed

to perform clear slow movements towards the camera. On the other hand, for this project, the

runner had his arms and legs very close to each other and to the body. The legs move quite fast

so sometimes it becomes quite hard for the Kinect to correctly track them. The best solution to

this problem is to reboot the tracking software (OSCeleton) and start tracking the user again.

Another “trick” to make the tracking more robust is to ask the runner to initially run with the

legs slightly more separated than normal and after a few seconds, when the legs have been

tracked for a while, start running normally again. Nonetheless, it is just a matter of time that a

better version of Kinect is released, which would perfectly work with the developed software

from this project.

Further work

There is an aspect of this project that would be really interesting to work on in the future, which

has been left out due to time reasons. The program should be tested with a subject who had a

real running problem: analyse the technique, understand which elements of the running gait

could be improved, and set a target position as a goal to achieve. Then, plan a set of trainings

using this software and carefully analyse the results and changes in his technique. Finally, check

again in an outdoor court in order to see if the treadmill training has stuck in the runner’s mind

and he is capable of applying the acquired knowledge without receiving the audio feedback.

Moreover, it would be interesting to analyse more elements in the training. For instance, try to

find a proper way to track the arms and analyse their swing, and how it synchronises with the

movement of the legs. Actually, many other elements could be added to this program in order to

make the technique analysis even more complete.


Jordi Bolíbar

48

It would be a matter of finding a way to correctly analyse them by using a robust tracking, and

being able to give them an audio feedback that does not interfere with the already developed

ones. Notwithstanding, a too high amount of information could become too vexing to handle for

the runner; especially if he is asked to interpret many different audio feedbacks at the same

time. On the other hand, it would be interesting to carry out a broader study on how audio

feedback is perceived from a large number of runners, and see if it could be improved in some

aspects in order to adapt it to the taste of different people.

Another remarkable concept that could be easily developed from this project would be to use

Kinect Audio-Runner with a different aim, such as rehabilitation or aiding old people to walk.

This could be easily implemented by investigating which are the interesting parameters for the

user; and use the current ones plus any new one which might be needed. For instance, for old

people who need to train to walk with longer steps, the step distance sonification could be used,

as it has already been shown in the results and the video.


Jordi Bolíbar

49

Bibliography

[1] Thomas Hermann, Andy Hunt& John G. Neuhoff. 2011. The Sonification Handbook.

Logos Verlag. ISBN 978-3-8325-2819-5.

[2] Ross Tucker. 2007. Running Technique part I: The philosophy of how we run.

http://www.sportsscientists.com/2007/09/running-technique-part-i-philosophy-of.html.

Latest visit May 2012.

[3] Ross Tucker. 2007. Running Technique part II: Biomechanics of running and Pose.

http://www.sportsscientists.com/2007/09/running-technique-part-ii-biomechanics.html.

Latest visit May 2012.

[4] Eriksson, M., & Bresin, R. (2010). Improving running mechanics by use of interactive

sonification. In Bresin, R., Hermann, T., & Hunt, A. (Eds.), Proceedings of the Interaction

Sonification workshop (ISon) 2010 (pp. 95-98). Stockholm, Sweden: KTH Royal Institute

of Technology.

[5] GaëlDubus. 2011. Evaluation of four models for the sonification of elite rowing. J

Multimodal User Interfaces (2012) 5:143–156.

[6] Stephen Barrass, Nina Schaffert & Tim Barrass. Probing Preferences between Six Designs

of Interactive Sonifications for Recreational Sports, Health and Fitness.In Bresin, R.,

Hermann, T. & Hunt, A. (Eds.), Proceedings of the Interaction Sonification workshop

(ISon) 2010 (pp. 23-29). Stockholm, Sweden: KTH Royal Institute of Technology.

[7] Varni, G., Dubus, G., Oksanen, S., Volpe, G., Fabiani, M., Bresin, R., Välimäki, V.,

&Kleimola, J. (2012).Interactive sonification of synchronisation of motoric behaviour in

social active listening of music with mobile devices. Journal on Multimodal User

Interfaces, 5(3), 157-173.

[8] Branko Skof & Stanko Stuhec. Kinematic analysis of Jolanda Ceplak’s running technique.

http://www.coachr.org/kinematic_analysis_of_jolanda_ce.htm. Last visit April 2012.

[9] Jessica Gonowon. 2007. The Physics of Efficient Running. http://ffden-

2.phys.uaf.edu/212_spring2007.web.dir/jessica_gonowon/gonowon_page1.html. Last

visited April 2012.

[10] Tom F. Novacheck. 1997. The biomechanics of running. Gait and Posture 7 (1998) 77–

95.

[11] Steve Magness. 2010. How to run: running with proper biomechanics.

http://www.scienceofrunning.com/2010/08/how-to-run-running-with-proper.html. Last

visited March 2012.

[12] Okemos Playmakers running store. 2009. Good Form Running.

http://www.youtube.com/watch?v=Tx6x2cD6Y8Q&list=PLAF9B61E2EB3A9230&index

=1&feature=plpp_video. Last visited May 2012.

[13] Evelyne Kohler et al. 2002.Hearing sounds, understanding actions: Action representation

in mirror neurons. Science 297, 846 (2002). DOI: 10.1126/science.1070311.

http://www.sportsscientists.com/2007/09/running-technique-part-i-philosophy-of.html

http://www.sportsscientists.com/2007/09/running-technique-part-ii-biomechanics.html

http://www.coachr.org/kinematic_analysis_of_jolanda_ce.htm

http://ffden-2.phys.uaf.edu/212_spring2007.web.dir/jessica_gonowon/gonowon_page1.html

http://ffden-2.phys.uaf.edu/212_spring2007.web.dir/jessica_gonowon/gonowon_page1.html

http://www.scienceofrunning.com/2010/08/how-to-run-running-with-proper.html

http://www.youtube.com/watch?v=Tx6x2cD6Y8Q&list=PLAF9B61E2EB3A9230&index=1&feature=plpp_video

http://www.youtube.com/watch?v=Tx6x2cD6Y8Q&list=PLAF9B61E2EB3A9230&index=1&feature=plpp_video


Jordi Bolíbar

50

[14] Sensebloom. 2011. OSCeleton. https://github.com/Sensebloom/OSCeleton#readme. Last

visited May 2012.

[15] Pure data community website.http://puredata.info/. Last visited May 2012.

[16] Wikipedia. 2012. Kinect.http://en.wikipedia.org/wiki/Kinect. Last visited April 2012.

[17] OSCeleton Google Group. 2011. Kinect distances in mm.

http://groups.google.com/group/osceleton/browse_thread/thread/9cdcd25584a4be54/f8027f

65e08199cb?lnk=gst&q=distance+mm#f8027f65e08199cb. Last visited March 2012.

[18] Andy Farnell. Code examples for “Designing sound” textbook. 2010. MIT Press.

http://mitpress.mit.edu/designingsound/boing.asp. Last visited May 20120.

[19] Kinect support.Lighting. http://support.xbox.com/en-GB/kinect/setup-and-

playspace/lighting. Last visited May 2012.

[20] Brekel.Brekel Kinect. http://www.brekel.com/?page_id=160. Last visited June 2012.

[21] Nike. Introducing Nike+ Kinect Training. http://nikeinc.com/news/introducing-nike-

kinect-training. Last visited June 2012.

https://github.com/Sensebloom/OSCeleton#readme

http://puredata.info/

http://en.wikipedia.org/wiki/Kinect

http://groups.google.com/group/osceleton/browse_thread/thread/9cdcd25584a4be54/f8027f65e08199cb?lnk=gst&q=distance+mm#f8027f65e08199cb

http://groups.google.com/group/osceleton/browse_thread/thread/9cdcd25584a4be54/f8027f65e08199cb?lnk=gst&q=distance+mm#f8027f65e08199cb

http://mitpress.mit.edu/designingsound/boing.asp

http://support.xbox.com/en-GB/kinect/setup-and-playspace/lighting

http://support.xbox.com/en-GB/kinect/setup-and-playspace/lighting

http://www.brekel.com/?page_id=160

http://nikeinc.com/news/introducing-nike-kinect-training

http://nikeinc.com/news/introducing-nike-kinect-training


Jordi Bolíbar

51

Appendix

List of used terms

CG: Centre of gravity

EMG: Electromyography

OSC: Open Sound Control

Pd: Pure data

SDK: Software Development Kit

VD: Vertical displacement

WAV: Waveform Audio File Format

TRITA-CSC-E 2012:062 ISRN-KTH/CSC/E--12/062-SE

ISSN-1653-5715

www.kth.se

kinect audio-runner: audio feedback for improving...

Documents