Grasping Graphs by Ear: Grasping Graphs by Ear:

Sonification of Interaction Sonification of Interaction

with Hidden Graphswith Hidden Graphs

Leena Vesterinen

Department of Computer Sciences

University of Tampere Finland

Leena.Vesterinen@uta.fi

March, 2005

AAFG 2005

The goal of the game “Hidden Graphs” is a blind inspection of the

hidden graphs. Player is to capture as many features as possible in

the graph following the guiding sound signals.

3 different sounds are used to guide the player actions in detection

of the hidden graphs.

3 major concepts are employed: Capture radius, Directional-

predictive sound signals (DPS) and Basic behavioral patterns (BBP).

Player task: Player is to choose the right capturing strategy and

recognize the hidden graph.

Researcher task: Researcher is to optimize the “dialogue” with player

through basic behavioral patterns coordinated to directional-

predictive sound signals, and to facilitate shaping the personal

behavioral strategy.

L. Vesterinen p 02_17 21.03.2005

Grasping Graphs by Ear

L. Vesterinen p 03_17 21.03.2005

Grasping Graphs by Ear

For instance, by grasping virtual graphs, children would develop skills

in cross-modal coordination: The use of special basic behavioral

patterns for efficient inspection primarily within the game field, and

learning how and when is suitable to apply one or another gesturing

in dependence on discovered features.

Basic cognitive processes: Sound feedbacks learning and experience

should progress through the game from the concrete level to more

abstract level. Gradual improvement on hearing of feedbacks (and/or

haptics) should finally form the personal Behavioral strategy.

L. Vesterinen p 04_17 21.03.2005

Sonification: Use of non-speech audio to convey information.

The sonification field is composed of the following three components:

(1) psychological research in perception and cognition; (2) sonification

tools for research and application; (3) sonification design and

application.

In particular, sonification is a potential solution for communication and

interpretation of data.

Plenty of research is done on sonification and applied to numerous

application domains: to provide navigation cues, information

visualization (charts and graphs) and non-visual drawing.

Regarding to our study - remarkable work is done by B.N. Walker and

J. Lindsay, 2004 based on the work of Tran et al., 2000.

[B.N. Walker and J. Lindsay, 2004] [Tran et al. 2000] [Walker et al., 2003] [Franklin et al.,

2004] [Holland et al., 2002]

Grasping Graphs by Ear

L. Vesterinen p 05_17 21.03.2005

The capture radius of an auditory *beacon as the range at which the

system considers a user to have research the *waypoint where beacon

in positioned. In practise, as a participant moves close enough to

waypoint, sound signal is given as an indication for leading the player

towards the goal, capturing the graph.

Directional-predictive sounds (DPS) are used in relation to capture

radius, for graph inspection. 3 unique DPS sounds are combinations of

pure sine wave signals with variable tone,

pitch and volume.

Grasping Graphs by Ear

80

20

* Waypoint: The coordinates of a specific location.

* Beacon: The object at the specific location in the

coordinate system.

Pilot Game Testing – 4 subjects: intermediate/experienced, with a

normal vision, normal hearing

Software: ‘Hidden Graphs’ game, PC version

Hardware: AceCad AceCat Flair USB graphics tablet on a standard

laptop with two speakers

Conditions: silent, closed room

Testing procedure: during the test, subjects were blindfolded, hearing

the sounds from two speakers. 3 separate test sessions, due to high

concentration level and duration of time during the test (>60min< ). 30

games in each session. Each game (1/30) involved playing at the

preliminary inspection phase and confirmation phase. In the preliminary

inspection phase the player captured the graph trying to memorise the

graph. Later player entered the confirmation mode and captured the

same graph again as accurately as possible.

L. Vesterinen p 06_17 21.03.2005

Grasping Graphs by Ear

Behavioral strategies:

BBP1: ‘Spiral’ and straight line gestures were applied as the basic behavioral patterns for the graph capturing. Player is to scale, change direction or speed of the gesture during the inspection, in relation to the DPS- signals. CS – crossing sound: Player is capturing the graph inside the capture radius. Apply straight line gesture.BS- backward sound: Player is getting out of the capture radius. Apply spiral gesture (scale to big).TS – towards sound: Player is returning towards the capture radius. Apply spiral gesture (scale to small).

L. Vesterinen p 07_17 21.03.2005

Grasping Graphs by Ear

BBP2: ‘S’ –shape and straight line gestures were applied as the basic behavioral patterns for the graph capturing. Player is to scale, change direction or speed of the gesture during the inspection, in relation to the DPS- signals. CS – crossing sound: Player is capturing the graph inside the capture radius. Apply straight line gesture.BS- backward sound: Player is getting out of the capture radius. Apply ‘S’-shape gesture (scale to big).TS – towards sound: Player is returning towards the capture radius. Apply ‘S’-shape gesture (scale to small).

BBP3: Combination of the BBP1 and BBP2 behavioral patterns following the same rule format.

L. Vesterinen p 08_17 21.03.2005

Grasping Graphs by Ear

Movement vectors

L. Vesterinen p 09_17 21.03.2005

Directional-predictive signals

Grasping Graphs by Ear

TS BS CS

L. Vesterinen p 10_17 21.03.2005

Graphs used in testing

Grasping Graphs by Ear

5 different shape graphs were used in the game testing.

L. Vesterinen p 11_17 21.03.2005

level: Rc = 20 pxls – 1st level; Rc = 15 pxls – 2nd level; Rc = 10 pxls – 3rd level

Grasping Graphs by Ear

The average distance to graph inspected and the time spent in the training phase in dependence on the game level

4

8

12

16

20

10 15 20

Capture radius pxls.

Dis

tans

e to

gra

ph p

xls.

010203040506070

Ave. distance Training phase s

The relative frequency of DPS-sounds used (the average number is in a log. scale) for graph inspection in the training phase in dependence on the game level

1

10

100

1000

10 15 20

Capture radius pxls.

Num

ber o

f DP

S-s

ound

s

CS tra. BS tra. TS tra.

L. Vesterinen p 12_17 21.03.2005

level: Rc = 20 pxls – 1st level; Rc = 15 pxls – 2nd level; Rc = 10 pxls – 3rd level

Grasping Graphs by Ear

The relative frequency (the average number) of DPS-sounds used for graph inspection at the confirmation phase in dependence on the game level. Percentage of inspected points.

1

10

100

1000

10 15 20Capture radius, pxls

Num

ber

of D

PS

-sou

nds

0%

20%

40%

60%

80%

CS conf. BS conf.

TS conf. Inspect. points

The average distance to graph inspected and the time spent at the confirmation phase in dependence on the game level. Std.dev. of the ave.distance in each level.

0

4

8

12

16

20

10 15 20

Capture radius, pxls

Ave.

dis

tance t

o t

he g

raph,

pxls

0

5

10

15

20

25

Ave. distance Confirm. phase s

L. Vesterinen p 13_17 21.03.2005

Grasping Graphs by Ear10 107.5 14 1215 117 12.33333 920 153 3 2

ratioCS/BS CS/TS

10 7.678571 8.95833315 9.486489 13 120 51 76.5

corr. 0.883879 0.891465

Comparison ratios for the BBP1, 2 & 3 correlation (prel.insp.) >

corr. Bp1 & Bp2 0.661423corr. Bp2 & Bp3 -0.036429corr. Bp1 &Bp3 0.357832

0.00

1.50

3.00

4.50

6.00

7.50

9.00

1 2 3 4 5

Graphs

CS

to B

S r

atio

BBP1 BBP2 BBP1+BBP2

1.50

2.50

3.50

4.50

5.50

1 2 3 4 5

Graphs

CS

to B

S r

atio

BBP1 BBP2 BBP1+BBP2

corr. Bp1 & Bp2 -0.16422corr. Bp2& Bp3 0.687668corr. Bp1 & Bp3 0.588959

< Typical correlation coefficients

Level 1, capture radius 20 pixels – starting level.

The frequency of the levels within the 30 games set in testing: over 90% of the games played in level 2 or 3, capture radius 10 or 15 pixels. 4% of the games played in level 4.

Starting position of the capture was free and did not make difference on performance. Separating mouse from the tablet would cause confusion for the player about the position.

Smaller the capture radius, longer the confirmation phase and more pixels captured in the target graph.

The average distance to the graph within the different levels, did not vary much. Although, std. deviation shows that the ave. distances in different levels are variable. Std. dev. on ave. distance within levels in BBP2, is the largest.

Different sounds (CS-BS, BS-TS, CS-TS) in each behavioral pattern associated strongly and positively, resulting in high correlation.

L. Vesterinen p 14_17 21.03.2005

Grasping Graphs by Ear

L. Vesterinen p 15_17 21.03.2005

Grasping Graphs by EarOverall, there was a little correlation between the 3 different behavioral strategies, for CS/BS sounds. The most, positive correlation was found in capture radius 15 and 20, between BBP1 and BBP3.

Statistics indicated that the graph 2 got the least points inspected in each level, in all 3 behavioral strategies.

We can conclude from the calculated statistics that the BBP3 appears to be the most efficient behavioral strategy to be used for the smaller capture radius (Rc = 10).

Overall, seems that the BBP1, 2 and 3 are very close to each other in performance measures, therefore no absolute and clear best BB strategy could be stated.

For future development, more testing could be done using limited test time, with additional behavioral strategies, increased number of test players and larger data set to make further conclusions.

References:

Brewster S. A, Using Non-Speech Sounds to Provide Navigation Cues to, ACM

transactions on computer human interaction, 5, 3, 1998

Brown L.M and Brewster S.A, Drawing by ear: interpreting sonified line graphs,

Proceedings of the 2003 international Conference on auditory displays, 6-9 July, 2003.

Brown L. M. et al., Design guidelines for audio presentation of Graphs and Tables,

Proceedings of the 2003 International Conference on Auditory Displays, 6-9 July,

2003.

Evreinov G, ‘Hidden Graphs’ game, Tampere university

Franklin K. F and Roberts J.C, Pie Chart Sonification, Proceedings Information

Visualization (IV03), pages 4-9. IEEE Computer Society, July 2003

Franklin K.M and Roberts J.C, A Path Based Model for Sonification, Information

Visualization, pages 865-870. IEEE Computer Society, July 2004.

Holland S, Morse D.R and Gedenryd H, AudioGPS: Spatial Audio Navigation with a

Minimal Attention Interface, ACM Personal and Ubiguitous computing, 6, 4, pages 253

– 259, 2002.

Jacobson R, Representing Spatial Information Through Multimodal Interfaces, IEEE 6 th

International conference on Information Visualization (IV’02), July 10 - 12, 2002

L. Vesterinen p 16_17 21.03.2005

Kramer G, ed. Auditory Display: Sonification, Audification, and Auditory Interfaces,

Proc. Volume XVIII, Reading MA: Addison Wesley, 1994.

Walker B.N and Lindsay J, “Auditory Navigation Performance is Affected by

Waypoint Capture Radius,” in Proc. of ICAD 04, Sydney, Australia, July 6-9, 2004

Walker B. N. and Lindsay J, Effect of Beacon Sounds on Navigation Performance

in a Virtual Reality Environment, Proceedings of the 2003 International

Conference on Auditory Display, Boston, MA, USA, 6-9 July 2003.

Tran T. V.; Letowski T.; Abouchacra K. S. Tran, Evaluation of acoustic beacon

characteristics for navigation tasks. Ergonomics, 1 June 2000, 43, 6, pp. 807-

827(21).

Web site (2005) www.audiogames.net

Web site (2004) www.gamesfortheblind.com

Web site (2003) http://www.nkl.fi/julkaisu/nvrkirja/luku1.htm#1.1

Click here to go to first slide

L. Vesterinen p 17_17 21.03.2005