1

Multi-modal Exploration of

Large Scientific Data Using

Virtual Reality

Dr. Pierre Boulanger

Department of Computing Science

University of Alberta

Department of Computing Science

University of Alberta

Shuttle Radar Topography Mission

(SRTM)

February 11, 2000, the Shuttle Radar

Topography Mission (SRTM) was

launched into space as part of one of

the payload of the Shuttle Endeavor.

Using a new radar sweeping technique

most of the Earth's surfaces was

digitized in 3D in approximately 10

days.

SRTM acquired enough data during its

mission to obtain a near-global high-

resolution database of the Earth's

topography.

Terrain Model of Mount St. Helens

Terrain Model Rendering

Terrain Model After Compression

and Hole Filling

Low Altitude Airflow Over Mount St.

Helens

Aburra Valley Colombia

CFD Model

Convective Winds Simulations

Based on Landsat IR Data

2

Virtual Analysis of a Francis Turbine

at “La Herradura” in Colombia• The main objective of the DIFRANCI Project is to apply a condition

assessment methodology following holistic approach to the

maintenance of the Francis turbines of "La Herradura" hydropower

plant in Colombia.

• Project in collaboration between:• Empresas Públicas de Medellín

• Colombian Agency for Science and Technology

• EAFIT University, Medellin, Colombia

• EPFL, Lausanne, Swizerland

• UofA, Edmonton, Canada

Digitizing the Turbine Using a Hand Held

Scanner

Scanning Using Handy Scan from

Creaform 3D

Final Scanned Reversed Engineered

Model

Scanned Model of the Turbine

Extracted FEM Mesh

Rapid Virtual Prototyping

• Once a 3D model is created, virtual

prototyping allows product testing without

the need to build a real prototype

• Allows for shape and functional

optimization

• Allows to tract the complete life cycle of a

product

• Rapid Virtual Prototyping requires powerful

computing infrastructure especially if it is

interactive

CFD Simulation of Francis Turbine

ProjectParticle Flow

3

Pressure Variations vs Time Results of CFD Analysis

Wall Pressure Comparisons of the computed pressure

recovery coefficient with the experimental

values, medium size mesh, 1.15ψ= in the

case of the FLINDT draft tube.

The UofA/EAFIT Virtual Wind Tunnel Definition of Interactive CFD

Need: A set of tools that allow the

designer to test interactively the

behavior of a design under various flow

conditions.

Task: The main task performed by an

interactive CFD system can be defined

as solving simulations of fluid flow for an

object, given the possible scenarios

defined by the user and to display the

results interactively.

User/Consumer: The target market is

none other than designers interested in

testing their design’s behavior under

fluid flow conditions to optimize and test

their design.

Interactive CFD User needsUofA/EAFIT Virtual Wind Tunnel

Architecture

4

Standard Scientific

Visualization/Simulation Pipeline

Data ImageFilter Map Render

Simulation

Parameters

Data

Repository

High Speed Network

Simulator

Remote

Desktop

Problems With Current Practice

• Hard and expensive process to determine

dominant parameters in a CFD simulation

model.

• Simulation runs cannot be steered

resulting in useless computations.

• It is like working blind.

• Current pipeline do not allow

collaboration. It is always after the fact

that the simulation data is analyzed and

shared by a group of engineer.

Even Better: Collaborative

Visualization/Simulation Steering Environment

Simulation

Parameters

Filter Map Render

User1

ImageData

Client 1

Simulation

Server

Filter Map Render

User n

ImageData

Client n

UofA Advanced Collaborative

Immersive Environments

3D Sound

Rendering

Input

Sensors

3D

Graphic

Rendering

Massive

Storage

Haptic

Rendering

New VizRoom

AMMI Lab Local

High Speed Network

Definition of Virtual Reality

• A virtual reality system is an interface

between a human and a machine capable

of creating a real-time sensory experience

of real and artificial worlds through the

various human sensory channels.

• These sensory channels for man are:

Vision, Audition, Touch, Smell, and Taste.

Burdea, 1993

Definition of Real-Time

Real-time, in virtual reality, means

that the computer system can detect

the input of the user and react to it

fast enough so that it appears to be

instantaneous.

5

How is VR Different than CG?

• Objects in the environment have a

strong sense of spatial presence,

creating the effect that the objects

exist independently of the user.

• Control of interaction within the

environment is often through direct

manipulation of objects as in the real

world.

How is VR Different than CG?

• The computer interface is “hidden” in

the sense that the user interacts with

objects in the environment rather

than a computer which controls

objects in the environment.

• The user is immersed in the

environment, i.e., the user

experiences the environment from

within.

Immersion=“Presence”

• Presence is a state of

consciousness where the

human actor has a sense of

being in the location

specified by the displays.

• The unique feature of

"virtual reality" systems is

that they are general

purpose presence

transforming machines.

Multi-modal systems• use more than one sense or mode of interaction

• e.g. visual and aural senses: a text processor may speak

the words as well as echoing them to the screen

Multi-media systems• use more than one media to communicate information

• e.g. a computer-based teaching system

- may use video, animation, text and still images

- different media all using visual mode of interaction

- may also use sounds, both speech and non-speech

Multimodal vs. Multimedia

Multimodal Human–Computer

Interaction

Information processing

Information processing

Human

Computer

Internal perception / action feedback loop

AI agents internal decision loop

Multimedia output

Multimodal inputInterface

Level of Interactivity

• Reactive: where little user control

takes place over the content's

structure with program directed

options and feedback

• Co-active: providing user control for

sequence, timings and style.

• Pro-active: where the user controls

both structure and content at most of

the levels.

6

First Level of Function Segregation Virtual Wind Tunnel Interaction Map

n-DData

VirtualWorld

Mapping

Data Exploration and the Mapping

Problem

Visual

Haptic

Sound

Visualization

Westgrid 4K x 2K Display

Visualization Toolkits

• Computation/Analysis + Visualization

• NIH Image and its PC version Scion

• Matlab

• Programming Toolkits

• The Visualization ToolKit (VTK)

• Insight ToolKit (ITK)

• VisAD and Vis5D (also “visualization spreadsheet”)

• SCIRun

• Graphical Programming Toolkits

• Open Data Explorer (OpenDX)

• Paraview

• Advanced Visual Systems (AVS/Express)

• Amira

• Slicer

Multivariate Display Techniques

• Glyphs: 2D Scalar

• Heterogeneous Techniques: 2D and 3D

• Texture

• Spot Noise (van Wijk), Healey, Ware

• Layering

• 2D Scalar: Slivers, DDS (3D?)

• 2D Scalar, Vector, Tensor: Laidlaw, Crawfis

• Problem Reduction

• Dimensional reduction, Cluster analysis

• Smart Particles

• Time and space multiplexing

• Mapping different fields over time

• Magic

7

Glyph: Flow Probe Multiple Views in Space

Definition of Haptic Gibson (1966)

• A haptic system is defined as "The

sensibility of the individual to the world

adjacent to his body by use of his

body".

• The haptic perceptual system is

unusual in that it can include the

sensory receptors from the whole body

and is closely linked to the movement

of the body so can have a direct effect

on the world being perceived.

Human Haptics

• Two complementary channels:

~ Tactile

• Strictly responsible for the variation

of the cutaneous stimuli

• Presents spatial distribution of

forces

~ Kinesthetic (Proprioception)

• Refers to the human perception of

one’s own body position and motion

• Presents only the net force

information

Tactile Display

• Skin sensation is essential for many manipulation and exploration tasks, for example, medical palpation. Tactile display devices stimulate the skin to generate these sensations of contact.

• The skin responds to several distributed physical quantities:

1. High-frequency vibrations: Surface texture, slip, impact, and puncture.

2. Small-scale shape or pressure distribution

3. Thermal properties

CyberTouch of Immersion

• CyberTouch™ is a tactile feedback option for Immersion's CyberGlove.

• It features small vibrotactile stimulators on each finger and the palm of the CyberGlove.

• Each stimulator can be individually programmed to vary the strength of touch sensation.

• The array of stimulators can generate simple sensations such as pulses or sustained vibration, and they can be used in combination to produce complex tactile feedback patterns.

8

Other Tactile Displays

• Thermal properties: We infer

material composition and

temperature difference. Thermal

display devices usually based

on Peltier thermoelectric

coolers.

• Many other tactile display

modalities: electro-rheological

devices for conveying

compliance, electro-cutaneous

stimulators, ultrasonic friction

displays, and rotating disks for

creating slip sensations.

Haptic Rendering of Surfaces

The Models of the Probe

a line segment

a point

a 3D object

State of the Art

• Difficult to simulate proprioception on the

entire body

Fingertip

Arm +

2 fingers

Wrist

Foot

VR Architecture of CoRSAIRe/CFD

Environment at LIMSI

Menelas 2010

CFD Exploration Using LIMSI

CoRSAIRe System

9

CFD Exploration Using CoRSAIRe

System Haptic Data Rendering-I

Tourque Nulling Transverse Damping

Pao and Lawrence, 1998

Haptic Data Rendering-II

Relative Drag Feature Shift

Pao and Lawrence, 1998 van Reimersdahl et al., 2003

AMMI Lab Multi-Modal Interface

for CFD (Visual and Sound)

Type of Modalities

Modalities used for the interface

• Visual Mono/Stereo/CAVE

• Haptic

• Perception of fluids flow

• Objects manipulations

• Setting of boundary conditions

• Sonification of fluids

Multi-Modal Exploration of CFD

Flow

10

Project Background

Input:

• Fluid field with velocity vector, pressure, and other data

• Changes with time

Output:

• Sound characterizing the given fluid field

• Ambient: global to the whole field

• Local: at the point or area of interaction

• Local region: particles of the specific subset area around the pointer contribute to the sound

Multi-Modal Rendering System

Structure

Solution Data Server

Max/MSP Program

Main Program as

Max/MSP object

Visualization

Program

Haptic Program

Haptic DeviceImage

Sound

Haptic Rendering

Read from the haptic device and sends pointer info to the sound and visual programs

• Pointer position and orientation (converted to the data field dimensions)

• Buttons: interaction sphere diameter- local region

Render a force feedback:

• Virtual walls: provides a force disallowing movement of the device outside of the data field boundary

• Other feedback possible: produce a force that is proportional to the flow density and its direction

Visual Rendering

• Displays vector field, virtual pointer

(microphone) and interaction sphere

• SGI OpenGL Performer Library for graphical

representation

Sound Rendering I

• Calculates velocity vector at the position of the virtual

microphone depending on interaction sphere radius

(using Schaeffer’s interpolation scheme):

• Small : from vertices of the grid cell

• Large: from all the vertices inside the influence sphere

• Velocity value & angle at that position

sForallNode

nm

sForallNode

nmn

m

rr

rrt

t2

2

/1

/)(

)(

p

p

Sound Rendering II

• Two output values for both angle and

velocity:

• Output = value / max value

• Output = (value / max value) 5/3

• Relationship between loudness level and

intensity: S ~ a3/5 [B.Gold]

• Thus, a function between values and

amplitude should be:

a = const * data value5/3

to imply: S ~ data value

)(tv

Virtual

Microphone

Direction

11

• Amplitude ~ * 5/3

• were v5/3-> [0,1]

• and a5/3-> [0.5, 1]

Sound Rendering III

Frequency ~

were v -> [0,1] -> [500, 1500]

3/5v

)(tv

)(tv

White band noise is modified in amplitude

and frequency to simulate a wind effect

Sonification Types

Positive vs. Negative Amplitude Modulation• velocity value is mapped to either increase or

decrease in amplitude of the sound

Amplitude vs. Frequency Modulation• highest velocity value is mapped to either

loudest noise or highest pitch noise

Before vs. after interpolation• many separate sounds for each vertex in the

local area vs. one sound of the interpolated value at the position of virtual pointer

Hypothesis

• According to multimodal theory adding sound rendering to visual rendering should improve exploration of CFD flow fields

• Testability: In our experiments, we will determine if our hypothesis is correct for an eddy localization task

• Simplicity: This hypothesis is simple and can easily be tested

• Null hypothesis: The combination of visual and auditory do not improve eddy localization efficiency.

Variables

Independent Variables

• Eddy localization in the flow field

• Starting point in the volume

• Sound mapping: frequency or amplitude modulation

• Sound rendering: positive or negative amplitude

modulation

• Interpolation type: Schaeffer’s or multi-sound

sources

• Virtual microphone radius R

Dependent Variables

• Eddy localization error

• Time and length of trajectory to get to the eddy

Interface Evaluation Procedure

1. Design the experiment.

2. Conduct the experiment.

3. Collect the data.

4. Analyze the data.

5. Draw your conclusions & establish

hypotheses

6. Redesign and do it again.

Usability Experimental Setup

Visual and/or Audio cues, haptic - navigation

12

Usability Study Experimental

Setup

Participants asked to locate vortex centers

40 fields: 25x25x25

• Random vortex locations

• Red arrow / specific sound

• 15 warm-up trials

• 36 experimental trials

• Random start point

• Random setup

Usability Study Results

Worst results for the audio-alone system:• participants are slower in locating the goal

• participants are less efficient in exploring the

volume

• Participants are less precise in locating the goal

Multimodal vs. Visual Only

Interface

Equal or better results for the multi-modal system

• Participants explore less space

• Participants are much faster in locating the goal position

Importance of Sound Rendering

Parameters

Specific system setup helps to improve performance

The Best Configuration:

Positive amplitude with

large radius and before

interpolation

Usability Study Conclusion

• The multimodal system is more

efficient to localize eddies than either

pure visual or pure audio systems

• Specific mapping parameters influence

system performance

• Different audio parameters are better

for audio-only than for multi-modal

interface

• Different audio parameters might be

better for different conditions

Multi-modal Exploration of Electric

Filds

Melenas 2010

13

Molecule Docking

Melenas 2010

Solar TErrestrial RElations

Observatory (STEREO)

NASA Picture

May, 2006

Pictures by Johns-Hopkins Applied Physics Laboratory

STEREO Data

Usable Data from STEREO

• Stereo Pairs of the sun in visible and ultra-violet bands

• 3-D Reconstructions of the Corona and the sun main body

• 3-D distribution of plasma characteristics of solar energetic particles

• Local vector magnetic field

• Plasma characteristics of protons, alpha particles and heavy ions

• Trace the generation and evolution of traveling radio disturbances

STEREO Multi-Modal Interface

System for Data Driven Simulations

NASA

STEREODatabase

Sun Corona

Simulation

Sun Particle

Emission

Simulation

Sun

Electro-Magnetic

Emission

Simulation

Multi-Modal Data

Exploration

Interface

Earth

Electro-Magnetic

Disturbances

Simulation

14

Next!• The GPU Revolution

• SGI Petaflops in a rack

• True real-time simulation

and visualization

• Next Seminar Dec 1.Recent Developments of Closed-loop Simulation and

Visualization Interfaces Using GPU

SGI Prism XL

Tesla M2050 / M2070 GPU

Computing Module