THE UNIVERSITY THE UNIVERSITY OF BRITISH COLUMBIAOF BRITISH COLUMBIA

Multimodal Design & Technologies

Sidney Fels

Fels: Multimodal Design and Technologies 2

Overview

• Introduction

• Human I/O: Interface Examples

• Bringing Modalities Together

• Summary

Fels: Multimodal Design and Technologies 3

Introduction

• Two messages1. knowledge of modalities motivates

design

2. modalities can complement each other– challenges and pitfalls

• Applications in virtual environments– communication of experience– intimacy and embodiment

Fels: Multimodal Design and Technologies 4

Human Information Processing• Input

• 5 usual senses– vision, hearing, touch, taste, smell

• position and motion sensing systems

• Output• intentional

– neuromuscular, movable, verbal

• non-intentional / biopotentials– galvanic skin response (GSR), heart rate, brain,

muscle activation

• Cognitive– memory, decision making, tracking, learning

Fels: Multimodal Design and Technologies 5

Visual Display Technologies

• examples of designs exploiting human capabilities– Virtual Retinal Display (VRD)– Cubby– CAVE Automatic Virtual Environment

Fels: Multimodal Design and Technologies 6

Virtual Retinal Delay (VRD)

ArgonLaser

Delivery Optics

C&D Electronics

Scanners

AO Modulators

Red LaserDiode

From HIT lab,U. Of Washington(Furness et al., 1991)Microvision continuing work

Fels: Multimodal Design and Technologies 7

Cubby (Djajadiningrat and Gribnau, 2000)

QuickTime™ and aCinepak decompressorare needed to see this picture.

Fels: Multimodal Design and Technologies 8

CAVE Illustration (U. of Illinois, 1992)

Fels: Multimodal Design and Technologies 9

Haptics Display Technologies

• examples of touch and force feedback– Pantograph– Phantom– CyberForce

Fels: Multimodal Design and Technologies 10

Pantograph (Hayward, McGill)

• 2 dof translation - 10cm X 10cm• FF/B mouse• acceleration best for shock and hard

contacts

Fels: Multimodal Design and Technologies 11

Phantom (Massey, SensAble Devices)

• thimble on finger– single point force feedback– 3 degrees of freedom

Fels: Multimodal Design and Technologies 12

CyberForce (Kramer, Immersion Corp.)

• Exoskelton– arm forces

• Tendons– grasp forces

Fels: Multimodal Design and Technologies 13

tricks: visual / aural illusionsexamples of metacognitive gap

• visual: never show the point penetrating the surface, even if it is

• aural: play a crisp contact sound on contact

this makes the surfaceappear stiffer/harder(very robust effect)

actual:

displayed:

BUT: if time offset too great, opposite effect

Fels: Multimodal Design and Technologies 14

Other Haptic Devices

• Tactile feedback

• Temperature

• Sheer forces

Fels: Multimodal Design and Technologies 15

challenges for haptic interaction design

• continuous vs. discrete manual control

• displaying interaction potential

• embedding haptic interfaces

• ensuring tight sensory coupling

• interacting with other modalities

Fels: Multimodal Design and Technologies 16

Olfactory Devices

• How do we smell?– different theories:

• i.e. chemical, infrared absorption

– different perceptual mappings:• smell prism• four odors: fragrant, acrid, burnt and caprylic• domain specific: wine, beer, etc.• many, many others...

– Acuity is great - 10,000 times more sensitive than taste

– negative adaptation occurs• you get used to the smell

Fels: Multimodal Design and Technologies 17

Olfactory Interfaces• Smell camera

– Susnick & Raknow, Nature, 2000, 406, 710-714.

• Smell display– liquids, gels, microencapsulation

• Application Research:– Joe Kaye:

• inStink, Dollars & Scents, Scent Reminder, and more...

– Fels, Gauthier, Smith: Interactive Yoga system

• Smell Display Research and Products:– DigiScents (bankrupt), TriSenx (bankrupt)– DIVEpak (Southwest Research Institute, 1993)– see notes

Fels: Multimodal Design and Technologies 18

Olfaction Display Challenges

• smell synthesis– many smells vs. mixing base smells

• control breathing space– sealed room with air filtration– air control in front of and behind user– sealed pod– tethered mask– tubes into an HMD from pack– built into HMD

Fels: Multimodal Design and Technologies 19

Taste

• taste buds for– sensations of sour, salty, bitter, sweet and

umami– receptors not completely resolved

• umami receptor (Zucker et al., 2002)

– extremely complex and poorly understood

• interacts with olfaction

• display = food?

Fels: Multimodal Design and Technologies 20

Biopotentials: Examples• Galvanic Skin Response (GSR)

– Affective Computing, i.e. Galvactinator, Scheirer and Picard

• Heart Rate (HR)– 2 Hearts Musical System, (McCaig and Fels,

2002)

• Brain activity (EEG)– Brain Computer Interface (BCI)

• Muscle activity (EMG)

Fels: Multimodal Design and Technologies 21

• Multitude of input/output systems– all active at once

• I/O mechanisms usually depend upon – cognitive context– emotional contexts

• All these systems available for applications– complement each other

• Multimodal design looks at:– integration– substitution– complement

Summary of Human I/O

Fels: Multimodal Design and Technologies 22

Multimodal Design: Bringing it Together

• User centred and non-user centred

• Intimacy and Embodiment– automatic behaviour– sources of aesthetics

Fels: Multimodal Design and Technologies 23

Intimacy and Embodiment

• Want interfaces that feel “good” to use

• Humans and machines intimately linked– degree of intimacy supported may

determine success

• Types of relationships:– human to human– human to machine

Fels: Multimodal Design and Technologies 24

Intimacy• Intimacy is a measure of subjective match

between the behaviour of an object and the control of that object.– extension of “control intimacy” from electronic

musical instruments analysis (Moore, 1997)

• High intimacy implies:– object feels like an extension of self– satisfaction derives from interacting with object– emotional expression flows

• requires cognitive effort to prevent

Fels: Multimodal Design and Technologies 25

Intimacy, Embodiment and Aesthetics

Case 1: Object disembodied from Self

Case 2: Self embodies Object

Case 3: Self disembodied from Object

Case 4: Object embodies Self

self object

objectself

objectself

object self

Aesthetic

Control

Result

Reflection

Belonging

Fels: Multimodal Design and Technologies 26

Intimacy and Embodiment Design Examples

• Many excellent examples out there of interesting designs– Many at Siggraph:

• Jam-o-Drum (Blaine et al.), Wooden Mirror (Rozin), etc.

– Tangible Bits work (Ishii et al.)– Ubicomp (Weiser and more)– Wearable computing (Mann and others)– Art, Entertainment– Virtual reality

Fels: Multimodal Design and Technologies 27

Application: Iamascope (Fels and Mase, 1997)

Fels: Multimodal Design and Technologies 28

Iamascope Overview Video

QuickTime™ and aCinepak decompressorare needed to see this picture.

Fels: Multimodal Design and Technologies 29

FlowField: Semantics of Caress (Chen, Fels and Min, 2003)

• Whole hand interaction in VR– use multi-touch sensitive pad – immersive display

• Idea:– allow users direct manipulation of fluid

• use particle simulation for fluid– aesthetics was important

• hand manipulation on hard surface mapped to obstructions in the flow field

Fels: Multimodal Design and Technologies 30

FlowField: Semantics of Caress

QuickTime™ and aYUV420 codec decompressorare needed to see this picture.

Fels: Multimodal Design and Technologies 31

• Multimodal interfaces need to consider– human information processing– matching interface to task

• use complementary modes where appropriate

– intimacy and embodiment

• Plenty of research opportunities

Summary

Fels: Multimodal Design and Technologies 32

THE UNIVERSITY THE UNIVERSITY OF BRITISH COLUMBIAOF BRITISH COLUMBIA

Thank-you!

Please fill out your comments forms.

www.icics.ubc.ca/hci/multimodal/www.icics.ubc.ca/hci/multimodal/