wpi hive: introduction to virtual reality interesting issues, open problems prof. robert w. lindeman...

WPI HIVE:Introduction to Virtual Reality

Interesting Issues, Open Problems

Prof. Robert W. LindemanProf. Robert W. Lindeman

Worcester Polytechnic InstituteWorcester Polytechnic Institute

Department of Computer ScienceDepartment of Computer [email protected]@wpi.edu

Robert W. Lindeman - WPI Department of Computer Science

2

Overview• Administrative StuffAdministrative Stuff• What is Virtual Reality?What is Virtual Reality?• Why study Interaction in VR?Why study Interaction in VR?• Open ProblemsOpen Problems


3

Motivation• Much excitement (and hype) about how Much excitement (and hype) about how

VR was going to change thingsVR was going to change things• VR has not made inroads into everyday lifeVR has not made inroads into everyday life

• Lagging technologyLagging technology• Lack of understanding of usability issuesLack of understanding of usability issues• Lack of “killer app”Lack of “killer app”

• Still remains mainly in research labsStill remains mainly in research labs• Video games show great promiseVideo games show great promise• Training scenarios - surgery, military, therapyTraining scenarios - surgery, military, therapy

• Long-Term GoalLong-Term Goal• Make VR more usableMake VR more usable


4

Nothing New?SENSORAMA by Morton Heilig (1960)


5

Background• VR defined:VR defined:

• Fooling the senses into believing they are experiencing Fooling the senses into believing they are experiencing something they are not actually experiencingsomething they are not actually experiencing

• Virtual reality systems consist of:Virtual reality systems consist of:• Graphical/audio/haptic/... renderingGraphical/audio/haptic/... rendering• ContentContent• Tracking of people and objectsTracking of people and objects• Collision detectionCollision detection• Interaction techniquesInteraction techniques• Optional, but common:Optional, but common:

• NetworkingNetworking• Autonomous agentsAutonomous agents


6

Keys to Success• High fidelity (or realism)High fidelity (or realism)

• Graphics, audio, haptics, behaviors, etc.Graphics, audio, haptics, behaviors, etc.

• Low latencyLow latency• TrackingTracking• Collision detectionCollision detection• RenderingRendering• NetworkingNetworking

• Ease of useEase of use• Low cumber for usersLow cumber for users• Easy integration for programmersEasy integration for programmers

• Compelling ContentCompelling Content


7

The Senses• See (Visual Sense):See (Visual Sense):

• Visuals are excellent!Visuals are excellent!

• Hear (Aural Sense):Hear (Aural Sense):• Spatialized audio is very good!Spatialized audio is very good!

• Smell (Olfactory Sense):Smell (Olfactory Sense):• Very hard! Too many types of receptors.Very hard! Too many types of receptors.

• Touch (Haptic Sense):Touch (Haptic Sense):• Application specific and cumbersome Application specific and cumbersome

• Taste (Gustatory Sense):Taste (Gustatory Sense):• We know the base tastes, but that is it!We know the base tastes, but that is it!


8

See: Head-Mounted Displays


9

See: Projection-Based Environments


10

See: Projection-Based Environments (cont.)


11

Hear: Sound in VR• Display techniquesDisplay techniques

• Multi-speaker output (sound cube)Multi-speaker output (sound cube)• HeadphonesHeadphones

• Waveform filteringWaveform filtering• Simple balance & volume controlSimple balance & volume control• Head-Related Transfer FunctionsHead-Related Transfer Functions

• Software "Standards"Software "Standards"• OpenALOpenAL• A3D from Aureal (RIP!)A3D from Aureal (RIP!)• VRSonic.comVRSonic.com


12

Smell: Olfactory Sense• Two main problemsTwo main problems

• Scent generationScent generation• Tens of thousands of receptor typesTens of thousands of receptor types

• Scent deliveryScent delivery• Easier problemEasier problem


13

Smell: Air Cannon (Yanagida, 2004)


14

Touch: Haptic Feedback in VR• Tactile: Surface Tactile: Surface

propertiesproperties• Most densely populated Most densely populated

area is the fingertip (okay, area is the fingertip (okay, it's the tongue)it's the tongue)

• Kinesthetic: Muscles, Kinesthetic: Muscles, Tendons, etc.Tendons, etc.• Also known as Also known as

proprioceptionproprioception


15

Vibrotactile Feedback Projects

Navy TSAS Project

GW TactaBoard andTactaVest


16

The TactaBoard 2.1


17

Empirical Studies• TactaChair experimentsTactaChair experiments

• Vibration location identificationVibration location identification• Priming for a visual search taskPriming for a visual search task• Intensity matchingIntensity matching

• Current experimentsCurrent experiments• Exposure during room clearing tasksExposure during room clearing tasks• Spatial awarenessSpatial awareness• Team member location for team trainingTeam member location for team training• Robot tele-operationRobot tele-operation


18

TactaChair Experiments


19

Exposure Experiments• Looking at the use of Looking at the use of

spatialized vibrotactile spatialized vibrotactile feedback as a training aid feedback as a training aid on "victim" search taskson "victim" search tasks


20

Exposure Experiments (cont.)• Use vibration to convey exposureUse vibration to convey exposure• Results to appear in ACM CHI 2005Results to appear in ACM CHI 2005


21

Issues to be Addressed• Transfer effects from virtual to real Transfer effects from virtual to real

environmentsenvironments• How do subjects perform after training in VR?How do subjects perform after training in VR?

• Psychophysical issuesPsychophysical issues• Sensory substitutionSensory substitution

• Cognitive IssuesCognitive Issues• Does the addition of haptic cues increase Does the addition of haptic cues increase

cognitive load? cognitive load?

• Multi-modal integrationMulti-modal integration


22

Interaction in VR• Use of a keyboard and mouse is not Use of a keyboard and mouse is not

tractabletractable• Can't see themCan't see them• Want to move aroundWant to move around• No good 3D mappingsNo good 3D mappings

• How can we allow easy interaction that How can we allow easy interaction that takes advantage of real-world takes advantage of real-world experience?experience?• This is the problem that we need to solve!This is the problem that we need to solve!


23

Primary Interface Tasks• Navigation and WayfindingNavigation and Wayfinding• Object SelectionObject Selection• Object ManipulationObject Manipulation

• Direct ManipulationDirect Manipulation• Indirect ManipulationIndirect Manipulation

• Can compare different techniques Can compare different techniques empiricallyempirically


24

Desktop Interaction:SensAble PHANToM

http://www.sensable.com/


25

Direct Manipulation

Courtesy: Virginia Tech


26

Can We Do WIMP in VR?



27

Wearable Interaction with Haptics:Immersion CyberGrasp

http://www.immersion.com/


28

Wearable Interaction:Rob's Hand-Held Windows

http://www.cs.wpi.edu/~gogo/


29

How Do We Do Menus?

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.



30

Interface Devices



31

Augmented Reality (AR)

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.



32

Applications• Data perceptualizationData perceptualization

• Map variables to tactorsMap variables to tactors

• Spatial awarenessSpatial awareness• Driver warning system (vibrotactile Bott's dots)Driver warning system (vibrotactile Bott's dots)

• Navigational aidNavigational aid• Firefighter guidanceFirefighter guidance

• Non-verbal communicationNon-verbal communication• Map hand signals to vibrotactile patternsMap hand signals to vibrotactile patterns


33

The Unified Scene Graph• A way of specifying visual, audio, and A way of specifying visual, audio, and

tactile object propertiestactile object properties• Design goalsDesign goals

• Provide programmer with a single scene graphProvide programmer with a single scene graph• Need to retain speedNeed to retain speed• Need to support different "renderers"Need to support different "renderers"

• Main conceptsMain concepts• Material node on steroidsMaterial node on steroids• Compile/flatten routinesCompile/flatten routines


34

The Unified Scene Graph (cont.)• Associate all properties in Associate all properties in MaterialMaterial

NodeNode Universe

TransformationTransformation

TransformationMaterial

Table Material Material

Mug Coffee

Viewer


35

Summary of Research Activities• Human-Computer InteractionHuman-Computer Interaction

• Usability studiesUsability studies

• Computer GraphicsComputer Graphics• Virtual Reality and GamingVirtual Reality and Gaming

• Interesting input devicesInteresting input devices• Haptic/tactile feedbackHaptic/tactile feedback• Multi-modal interaction and integrationMulti-modal interaction and integration

• Pervasive and Wearable ComputingPervasive and Wearable Computing• Robot Tele-OperationRobot Tele-Operation


36

Open Problems in VR• Tracking sucks!Tracking sucks!

• Optical, magnetic, ultrasonic, inertial, hybridOptical, magnetic, ultrasonic, inertial, hybrid

• Integration of sensory stimuliIntegration of sensory stimuli• Good combination of direct and Good combination of direct and

indirect user interactionindirect user interaction• User studies!User studies!

• Tough to designTough to design• Easy to get subjectsEasy to get subjects


37

More Info

• [email protected]@wpi.edu• http://www.http://www.cscs..wpiwpi..eduedu/~/~gogogogo//• http://www.http://www.cscs..wpiwpi..eduedu//~gogo/hive/~gogo/hive/

wpi hive: introduction to virtual reality interesting issues, open problems prof. robert w. lindeman...

Documents