Relative influence of sensory cues in a multi-modal virtual environmentClaudia HendrixPi-Ming ChengWilliam Durfee

Department of Mechanical Engineering

SIGHT(visual)

TOUCH(haptic)

SOUND(aural)

Take-home messages

Virtual environments can do a good job emulating soft materials

Haptic cues are important for identifying materials

Adding visual or sound cues improves ability to discriminate among materials

Of haptic, visual and sound, sound least important

Virtual prototyping using sight, sound and touch is feasible, but effects of sensory interaction are complex and vary from person to person

Some prior multi-modal VR work

Marks [1978]: Reviews psychophysics of sensory interaction

Richard and Coiffet [1995]: Adding substitute sensory haptic feedback improves performance on grasping and place tasks

Hendrix and Barfield [1995]: Adding synthesized sound sources to a virtual world increases sense of “presence”…..but not sense of “realism”

Srinivasan et al [1996] and Durfee et al [1997]: Visual displays influence perception of haptic stiffness

DiFranco et al [1997]: Audio cues influence perception of haptic stiffness

Virtual product prototyping Apply virtual reality technology to create product prototypes Move beyond CAD-based visual rendering

See Hear Touch

? ?

? ?

Panel controls: simplified paradigm for research

•Fixed in space, single d-o-f, low-force, simple graphics•Sufficiently complex to enable exploration of research questions

With panel controls paradigm...

Visual Aural Haptic

“Head-related transfer function”

system

Experiment system

visual display

Ethernet

SGI

PC

servoamp

motion, force, torque motor

haptic display

speaker

aural display

green screen

camera

to SGI

Haptic display

Haptic display control

Pentium 90MHz host computer

DSP board

shared memory

encoder interface

ADC

ADC

servoamp

accelerometer

encoder

torque transducer

conditioning electronics

com

man

d o

ut

motor

DAC

Materials modeled as spring-damper, PD impedance controller, 60 u-sec update rate

Aural display Record sound clips for

different strike velocities Synchronized playback (Synthesized sound based

on physical models too complex…for now)

Match virtual to one of 3 real EXPERIMENT #1

“BEST” MATCH– VVR = V1

– HVR = H1

– AVR = A1

EXPERIMENT #2SENSORY CONFLICT– VVR = V2

– HVR = H1

– AVR = A3

VVR

HVR

AVR

V1

H1

A1

V2

H2

A2

V3

H3

A3

VIRTUAL

REAL

metal hard foam soft foam

Material properties:V = visualH = hapticA = aural

subject

motorgreen screen

probereference materials

speakers

camera

to SGI

test material

Experiment setup

Sti

ffn

ess

(Nm

/ra

d)

35

7.2

3.2

REAL VIRTUAL

Soft foam

Hard foam

Metal

Soft foam

Hard foam

Metal

Ability of haptic interface to mimic material stiffness

Protocol, Expt #1 (“best” VR)

Test conditions MATERIAL TYPE

Metal Hard Softfoam foam

TEST Virtual X X X MATERIAL

Real X X X

PROCEDURE:Probe test and reference materials, then answer:

1. Which material best matches test?

2. What is the quality of the match? (1-5)

6 test conditions, 4 replications = 24 trials/session

12 subjects

Matching results

Material, matches (%), quality rating (1-5)

METAL

HARDFOAM

SOFTFOAM

M 100 (4.6)HF 0SF 0

M 0HF 85 (4.3)SF 15 (3.7)

M 0HF 2 (3.0)SF 98 (4.5)

When test was...

Subjects matched to...

Test material was real

METAL

HARDFOAM

SOFTFOAM

When test was...

Subjects matched to...

Test material was virtual

M 60 (3.5)HF 40 (3.0)SF 0

M 0HF 83 (3.9)SF 17 (2.7)

M 0HF 2 (2.0)SF 98 (4.2)

Quality ratings

0

1

2

3

4

5

Metal Hard foam Soft foam

Qu

ali

ty r

ati

ng

Real correct

Virtual correct

Real incorrect

Virtual incorrect

Protocol, Expt #2 (sensory conflict)

Vsoft foam

Hmetal

Asoft foam

Looks and sounds like soft foam, feels like metalmetal

hardfoam

softfoam

metal

hardfoam

softfoam

metal

hardfoam

softfoam

Visual

Aural

Haptic

27 test conditions (always virtual), 3 replications = 81 trials per session

TEST MATERIAL

Matching with sensory conflict

M

HF

SF

TESTMATCH

M 11HF 79SF 10

M 14HF 58SF 28

M 13HF 11SF 76

HAPTIC

M

HF

SF

TESTMATCH

M 8HF 26SF 65

M 14HF 50SF 36

M 15HF 72SF 13

VISUAL

M

HF

SF

TESTMATCH

M 18HF 43SF 39

M 10HF 40SF 50

M 10HF 65SF 25

AURAL

M

HF

SF

TESTMATCH

M 31HF 62SF 7

M 4HF 47SF 49

M 4HF 13SF 83

HAPTIC + AURAL

M

HF

SF

TESTMATCH

M 38HF 61SF 1

M 8HF 76SF 15

M 7HF 7SF 86

HAPTIC + VISUAL

M

HF

SF

TESTMATCH

M 22HF 32SF 46

M 11HF 50SF 39

M 8HF 68SF 24

AURAL + VISUAL

1-WAY

2-WAY

3-way match

M

HF

SF

TESTMATCH

M 47HF 44SF 8

M 0HF 78SF 22

M 0HF 3SF 97

HAPTIC + VISUAL + AURAL

Quality when matched

0

1

2

3

4

5

one two

Modalities matched

Qu

alit

y ra

tin

g

H

AV H

+A

H+V A

+V

Cues used by the subjects

Number ofSubjects

Most importantcharacteristics

8 Haptic then visual1 Visual then sound1 Visual then haptic1 Haptic alone1 Haptic then sound

“What strategy did you use to make your matches?”

What we learned

Future work:1. Use 2 out of 3 cues (V,H,A) to determine relative influence2. Mechanical design task using virtual and real prototypes

National Science Foundation (NSF/MIP-9420394, and a graduate training grant, Cognitive Sciences Center, University of Minnesota, DMC8857851)

Work supported by:

Virtual environment does well when emulating soft materials

Haptic cues are important for matching Adding visual or sound cues to haptics improves

perceived quality of match Subjects tended not to use sound cues for matching