escop 2007 - marseille1 interactive sonification of geographical maps: a behavioural study with...

ESCOP 2007 - Marseille 1

Interactive sonification of geographical maps: a behavioural study with blind subjects

Marta Olivetti Belardinelli 1,2, Franco Delogu 1,2, Massimiliano Palmiero 1, Stefano Federici 1,3, H. Zhao4,

Catherine Plaisant 4

1 Department of Psychology, University of Rome “La Sapienza”, Italy2 E.CO.N.A. Interuniversity Center for Research on Cognitive Processing in Natural and Artificial Systems3 Department of Human and Educational Science, University of Perugia, Italy4 Human Computer Interaction Laboratory, University of Maryland, USA


Blindness and spatial cognition:an open debate

DOES EARLY VISUAL EXPERIENCE DEFINETELY AFFECT SPATIAL PROCESSING?

Controversial Results:

Visual experience is necessary to the acquisition of spatial information

(Lahav & Mioduser, 2003)

Blind people have difficulties in forming mental representations of space

(Vecchi, 1998)

Blind people acquire spatial informationusing tactile, auditory, kinaesthetic information

(Alfonso et al. , 2005; Zhao et al., 2004)

Spatial representations generated by different sensory modalities

are equivalent to visuo-spatial representation

(Bryant, 1992)


Non- visual display solutions

Tactile graphics

Screen Readers

- Cumbersome to produce - Limited in the amount of information - Psychophysics limitations

- Strictly sequential - Reading single values can overcome the WM capacity- lack of spatial reference

Audio-Based Systems: can these difficulties be bypassed bySONIFICATION?

http://www.tinyapps.org/screenshots/remotepad.png


Sonification of spatial information

Sonification is the use of nonspeech audio to convey information (Kramer et al. 1997)

• Displays several parameters simultaneously (loudness, pitch, timbre, rhythm)

• Easy use of analogies: rising volume easy interpreted as something increasing

• Sonificated information may result in augmented tactile maps (Parente, P., and Bishop, G., 2003)

• Easy to transmit geographical and environmental data (Zhao et al., 2004)

Is it possible to effectively transmit the features of large-scale geopolitical maps

using sonification?

AIM


iSONIC (developed by the Human Computer Interaction Laboratory of the University of Maryland

for facilitating blind users to acquire geographical information)

APPARATUS

E.G. Unemployment rates (here in three different colors:

High Medinum Low


VIOLIN PITCH : LEVEL OF EMPLOYMENT

HIGH MEDIUM LOW

NON VERBAL SOUNDS

VERBAL SOUNDS

STATE NAMES

STATE BORDER: CLICK

END OF HORIZONTAL SWEEP = BELL

BACKGROUND / SEA = GUITAR FRET

HORIZONTAL POSITION = STEREO PANNING

value

APPARATUS - Sounds

ON SUBJECTS DEMANDS BY PRESSING THE SPACE-BAR

DURING EXPLORATION


APPARATUS - Navigation Methods

KEYBOARD TOUCH-PAD

STEREOPHONIC HEADPHONE Stereo-panning effect


APPARATUS - Keyboard Interface

1 2

4 5

3

7

6

8 9UP

DOWN

LEFT RIGHT

GIST: 0 KEY Automatic spatial sweep

ARROW KEYS Relative navigation

NUMBER-PAD FROM 1 TO 9 Absolute navigation

SPACE BAR: DETAILS ON DEMAND (STATE NAMES)

DISCRETE STEPS


APPARATUS - Touchpad Interface

FINGER

SPACE BAR: DETAILS ON DEMAND (STATE NAMES)

GIST: 0 KEY Automatic spatial sweep

CONTINUOUS EXPLORATION Integrated audio-tactile exploration


TOUCH-PAD

THE EXPERIMENT - HypothesisiSONIC effectively allows users

to correcly recognize the geographical maps

BLIND FOLDED

SUBJECTS

CONGENITALLY BLIND SUBJECTS

ACQUIRED BLIND SUBJECTS

As spatial representations are not necessarily linked to the visual modality, the three groups do not differ on the tactile recognition of the

maps

Blind subjects use different strategies with respect to sighted people,

using more specifically sound information

KEYBOARD

The two interfaces do not show any differences on discriminating the target from

the distractors

The two methods of exploration allow users to acquire the same information by means of

different strategies


THE EXPERIMENT - DesignBETWEEN SUBJECTS DESIGN 2x3

THE INFLUENCE OF: TOUCH-PAD KEYBOARD

BLIND FOLDED SUBJECTS

CONGENITALLY BLIND SUBJECTS

ACQUIRED BLIND SUBJECTS

VISUAL IMPAIRMENT

TYPE OF INTERFACE

ON:

MAP RECOGNITION

SPATIAL REPRESENTATION (Global shape of the map, Shape and Size of the states, Estimated number of states)

MAP NAVIGATION STRATEGIES (Gist, Step by Step, 3x3 Quadrant)

SELF REPORTS, QUESTIONNAIRE ANSWER (Self-Confidence, Understanding, Satisfaction)

X


THE EXPERIMENT - Subjects

35 subjects 16 females and 19 males, participated in the

study [mean age = 32,46; SD= 6,73] with no auditory sensory deficits.

15 blindfolded sighted

(8 femals and 7 males)

10 acquired blind

(4 females and 6 males)

10 congenitally blind(4 females and 6 males)


THE EXPERIMENT – Materials/1 MAPS

4 iSONIC Software Maps (TARGETS)

Violin Pitch Level of Unemployment

HIGH MEDIUM LOW

16 Tactile Paper Maps (TARGETS PLUS DISTRACTORS)

Textures type of Unemployment

DOTTED HERRINGBONE STRIPED

PLUS BOUNDARIES (BORDER LINE WITHOUT TEXTURES) OF IDAHO MAP

a

b

c

d


THE EXPERIMENT – Materials/2 4 QUESTIONNAIRES

SELF REPORT LEVEL OF ACCURACY OF USA 9 QUADRANTS

EXAMPLE: FIRST QUADRANT a) high; b) Medium;

c) Low; d) Background 1 2

4 5

3

76

8 9

X

SELF-REPORT LEVEL OF ACCURACY OF:

SPACE ORIENTATION, GEOMETRICAL SHAPE OF IDAHO MAP, NUMBER OF IDAHO REGIONS (The regions are the counties of the Idaho state)

SELF EFFICACY/CONFIDENCE SCALE IN RECOGNITION OF TACTILE MAPS

1 2 3 4 5 6 7 8 9 10X

SELF-REPORT EVALUATION OF SUBJECTS UNDERSTANDING AND SATISFACTION

1 2 3 4 5 6STRONGLY DISAGREE

STRONGLY AGREE

EXAMPLE: INTERFACE WAS EASY TO USE

X

1

2

3

4

For the first three tasks


THE EXPERIMENT – ProcedurePsychoacustics Trials:

Training

Sound Localization and Pitch Discrimination

Exploration Four Tasks

Tactile Recognition TasksFour Tasks X

TSKB

(3 minutes)

First task

Questionnaires SELF REPORT LEVEL OF ACCURACY OF: Usa 9 Quadrants for Task 1 – 2 - 3 and space orientation, geometrical shape and number of regions forTask 4

Final Questionnaire SELF-REPORT EVALUATION OF SUBJECTS UNDERSTANDING AND SATISFACTION

Pre

-Exp

erim

enta

l ph

ase

Exp

erim

enta

l ph

ase

Post-Experimental Phase


THE EXPERIMENT – Results/1Tactile Map Recognition

Post-hoc analysis shows that Mean recognition confidence was significantly higher for target tactile maps than for distractors in all the four tasks (Post-Hoc LSD: p<0.05)MAP - TASK 1

Current Effec t: F(3, 102)=19,338, p= ,00000

task1a task1b task1c task1d

MAP

1

2

3

4

5

6

7

8

9

ME

AN

RE

CO

GN

ITIO

N C

ON

FIN

DE

NC

E

MAP - TASK 2Current Effec t: F(3, 102)=14,088, p= ,00000


MAP

1

2

3

4

5

6

7

8

9

ME

AN

RE

CO

GN

ITIO

N C

ON

FID

EN

CE



MAP

2

3

4

5

6

7

8

9

ME

AN

RE

CO

GN

ITIO

N C

ON

FID

EN

CE



MAP

1

2

3

4

5

6

7

8

9

10

ME

AN

RE

CO

GN

ITIO

N C

ON

FID

EN

CE

TA

RG

ET

TA

RG

ET

TA

RG

ET

TA

RG

ET


THE EXPERIMENT – Results/2Tactile Map Recognition – EFFECT OF BLINDNESS

In all of four tasks, congenitally blind, acquired blind and sighted subjects did not differ in discriminating the target from distractors:

Task 1 [F(2, 32)=0,36, p > .05]

Task 2 [F(2, 32)=1,11, p > .05]

Task 3 [F(2, 32)=1,77, p > .05]

Task 4 [F(2, 32)=2,70, p > .05]

GROUP - TASK 1Current Effec t: F(2, 32)= ,36125, p=,69961

SIGHTED ACQUIRED BLIND CONGENITALLY BLIND

GROUP

4,5

5,5

6,5

7,5

8,5

9,5

10,5

ME

AN

RE

CO

GN

ITIO

N C

ON

FID

EN

CE


THE EXPERIMENT – Results/3Tactile Map Recognition – EFFECT OF INTERFACE

In all of the four tasks, the use of Key-Board and Touch-Pad during the exploration did not lead to significant differences in discriminating the target from distractors :

Task 1 [F(1, 29)=,46, p > .05]

Task 2 [F(1, 29)=,21, p > .05]

Task 3 [F(1, 29)=,24, p > .05]

Task 4 [F(1, 29)=,30, p > .05]

INTERFACE - TASK 1Current Effec t: F(1, 29)= ,46883, p= ,49896

TOUCH-PAD KEYBOARD

INTERFACE

5

6

7

8

9

10

ME

AN

RE

CO

GN

ITIO

N C

ON

FID

EN

CE


THE EXPERIMENT – Results/4Questionnaire - SELF REPORT LEVEL OF ACCURACY OF USA 9 QUADRANTS

Which level of the value (low, medium, high unemployment, or sea) is more represented into each quadrant?

Participants showed a better accuracy in the first task, the easiest, than in the second and third task, more complex (Post-Hoc LSD: p<.05).

0,3

0,4

0,5

0,6

0,7

0,8

0,9

Task 1 Task 2 Task 3

Task

Acc

ura

cy (

per

cen

tage

)

[F(2, 16)=8,3451, p=,00329].


0

100

200

300

400

500

600

Touch-Pad Keyboard

Nu

mb

er o

f A

ctio

ns

du

rin

g th

e ex

plo

rati

on

THE EXPERIMENT – Results/5Map Navigation Strategies

Touch-Pad users moved more rapidly from region to region than Keyboard users.

545

96


THE EXPERIMENT – Results/7Final Questionnaire – STEREO PANNING EFFECT

0,00,51,01,52,02,53,03,54,04,55,05,56,0

Sighted Acquired Blind Congenitally Blind

Group

Ho

w u

sefu

l fo

r th

e o

rien

tati

on

w

as t

he

ster

eo p

ann

ing

?Both groups of blind subjects more than sighted subjects reported to be helped by the

stereo panning (Post-Hoc LSD: p<0.05)


THE EXPERIMENT – Results/8Final Questionnaire – SUBJECTIVE EVALUATION OF INTERFACE

2,5

3,0

3,5

4,0

4,5

5,0

5,5

6,0

6,5

sighted blindfolded acquired blind congenitally blind

Group

How

eas

y to

use

was

the

inte

rfac

e?Both groups of blind subjects more than sighted ones rated the interface

to be easy to use.


GENERAL DISCUSSION1. The software allows correct recognition of geographical maps.

2. Sonification allows to transmit complex spatial information like large-scale geopolitical maps, even though difficulties in recognizing specific details of the maps were found.

3. No substantial differences between blind and sighted users, nor between early and late blind subjects were found, plausibly indicating that multi-sensory spatial representations are possible. The a-modal theory of spatial representation should be considered (Avraamides, Loomis, Klatzky, & Golledge 2004; Bryant, 1992).

4. Blind subjects, more than sighted subjects, reported to be helped by the stereo-panning effect.

5. With respect to sighted subjects blind subjects use different strategies based on a greater attention and experience to sound information.

6. No significant differences between interfaces were found.


CONCLUSION

1. Considering the limitations of tactile maps sonification can be effectively used to present geo-political information.

2. These findings have relevant empirical implications in order to allow the blind community an easy access to geopolitical data and more in general to complex spatial information.

3. Our data support the convincement that visual experience is not necessary for an efficient spatial cognition.

4. Further studies are necessary to examine the effectiveness of sonification in other application contexts, and its integration in multi-modal contexts. A dynamic combination of non-speech sounds and haptics could represent a new frontier in the implementation of effective non-visual displays. A multimodal approach allows users to access spatial information in more flexible ways.


THANKS FOR YOUR ATTENTION

REFERENCES:

•Avraamides, M., Loomis, J. Klatzky, R. L., & Golledge, R. G. (2004). Functional equivalence of spatial representations derived from vision and language: Evidence from allocentric judgments. Journal of Experimental Psychology: Human Learning,Memory & Cognition, 30, 801-814•Bryant, D. J. (1992) A Spatial Representation System in Humans. Psycholoquy 3(16) •Jacobson, R.D. (1998) Navigating maps with little or no sight: A novel audio-tactile approach. Proceedings of Content Visualization and Intermedia Representations. August 15, University of Montreal, Montreal.•Parente P. and Bishop G. (2003). “BATS: the blind audio tactile mapping system”, in Proc. ACM Southeast Regional Conf. 2003•Tinti C., Adenzato M. & Tamietto M., Cornoldi C. (2006). Visual experience is not necessary for efficient survey spatial cognition: Evidence from blindness. The quarterly journal of experimental psychology, 59 (7), 1306–1328 •Zhao H., Plaisant C., Shneiderman B., and Duraiswami R. (2004). Sonification of geo-referenced data for auditory information seeking: design principle and pilot study, Proc. ICAD 2004.

escop 2007 - marseille1 interactive sonification of geographical maps: a behavioural study with...

Documents

spatial processing

spatial representations

usa slide

exploration slide

aim slide

spatial cognition

information kramer

high medinumlow slide