1

Beyond BallBeyond Ball --andand --StickStickPart 1: Using vision to thinkPart 1: Using vision to think

Mario ValleMario ValleSwiss National Supercomputing Centre (CSCS)Swiss National Supercomputing Centre (CSCS)

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Kekulé dreamKekulé dream

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Everyday mental modelsEveryday mental models

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Mental modelsMental models

“All our ideas and concepts are only internal pictures”

Ludwig Boltzmann (1899)

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Models for mental simulationModels for mental simulation

For example we use mental simulation (manipulation ofa mental model) to solve the problem:

“Which rotated image corresponds to the first one?”

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Models guide our perceptionModels guide our perception

The “cocktail” effect

Who called me?

2

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Models simplify and abstractModels simplify and abstract

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Where mental models liveWhere mental models live

Then the new concepts discovered crystallize as new knowledge building links to existing information

(Wickens memory and perception model)

Mental models are built in our working memory

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

ImaginationImagination

“Imagination is

more important

than knowledge”

Albert Einstein

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Imagination from where?Imagination from where?

“Imagination is vision running backwards”

S. Greenfield

Leo Leoni, Fish is Fish, Pantheon, 1970

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Working memory limitsWorking memory limits

Then the new concepts discovered crystallize as new knowledge building links to existing information

Mental models are built in our working memory

But working memory:

� Has limited capacity(7 ± 2 items)

� Disappears in ~2 min

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

External cognitionExternal cognition

We have always created extensions to our mind and body to overcome our limitations

3

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Example from mental mathExample from mental math

34 x72

------68

238-------2448

Time needed

0

20

40

60

80

100

120

Mental On paper

sec

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Vision and cognitionVision and cognition

At high level vision and cognition are linked.

We say “I see!” to mean “I understand!”

The visual system is an extension of our brain: 1/3 – 1/4 is dedicated to visual perception.

Therefore why don’t we use our visual system to comprehend numerical data?

Old IBM advertising

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Visual patterns discoveryVisual patterns discovery

Trends, Clusters, Gaps, Outliers, Correlations, ...

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Vision enables direct perceptionVision enables direct perception

The Mandelbrot set has a symmetrical structure that looks like an insect. Around the central body are placed various smaller scale replicas of the same set. The biggest replica is located on the left side of the main body. All around there are detail rich threadlike structures…

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Vision provides an holistic viewVision provides an holistic view

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

How many ‘3’ are there?How many ‘3’ are there?

89739057092794057962976509829408028085080830802809850-802808567847298872ty458202094757720021789843890r455790456099272188897594797902855892594573979209

4

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Immediately seen if color addedImmediately seen if color added

89739057092794057962976509829408028085080830802809850-802808567847298872ty458202094757720021789843890r455790456099272188897594797902855892594573979209

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Preattentive perceptionPreattentive perception

"Civilization advances by extending the number of important operations which we can perform without thinking about them."

Alfred North Whitehead

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Structure formationStructure formation

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

See nonexistent structuresSee nonexistent structures

After Garcia-M

ata & Shaffner (1934)

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Change blindness Change blindness

Our eye is not a movie camera!

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

““ Using vision to think”Using vision to think”

S. K. Card,J. D. Mackinlay,B. Shneiderman

5

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Few directly perceived phenomena Few directly perceived phenomena

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Most data remains not accessibleMost data remains not accessible

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Scientific visualization birthScientific visualization birth

“Visualization offers a method for seeing the unseen . It enriches the process of scientific discovery and fosters profound and unexpected insights. In many fields it is already revolutionizing the way scientists do science”

Visualization in Scientific Computing,McCormick et al.ACM SIGGRAPH, 1987

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Visualization definedVisualization defined

“[Visualization is] the use of computer-supported, interactive, visual representations of data to amplify cognition...”

Card, Mackinlay, and Shneiderman

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Uses color, shape, interactionUses color, shape, interaction

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Uses spatial metaphorsUses spatial metaphors

6

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

(too much) visual metaphors(too much) visual metaphors

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Natural hierarchy of newsNatural hierarchy of news

www.marumushi.com/apps/newsmap

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

The scientific discovery processThe scientific discovery process

EXPERIMENT ORDATA COLLECTION

DATA

Computation orTransformation

Rendered imageInsight

Hypothesis

User

interaction

guesswork

By J. Watson

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Visualization processVisualization process

Conceptualmodel

Conceptualmodel

DataStudy object

Datamodel

Datamodel

acquisition

Preconceptions &interpretation

Influen

ce

Interaction

Algorithms

RenderPerceptio

n

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Three visualization rolesThree visualization roles

1. Confirmatory visualization� Verify that some hypothesis holds

2. Exploratory visualization� Exploration-driven research

3. Presentation and communication� Present what has been discovered

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Classical data analysisClassical data analysis

problem

data

model

analysis

conclusions

• Focus on the model

• Hypothesis-driven research

• Quantitative methods

7

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Exploratory data analysisExploratory data analysis

problem

data

analysis

model

conclusions

• Focus on the data

• Exploration-driven research

• Graphical methods

• Evolutionary

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Present and communicate resultsPresent and communicate results

Francesco Gervasio –

ETH Zürich

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

The visualization mottoThe visualization motto

“Discover the unexpected, describe and explain the expected”

National Visualization and Analytics Center™Pacific Northwest National Laboratory

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Confused with Computer GraphicConfused with Computer Graphic

“But I was convinced that visualization is about creating nice images only!”

Goal of visualization is to improve cognition using visual methods, not to create illusion

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Limit visualization usefulnessLimit visualization usefulness

ModelingComputation

Visualization

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

An active role for visualizationAn active role for visualization

ModelingComputation

Visualization

8

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Visualization is a toolVisualization is a tool

EXPERIMENT ORDATA COLLECTION

DATA

Computation orTransformation

Rendered imageInsight

Hypothesis

User

interaction

guesswork

VISUALIZATION

VISUALIZATION

By J. Watson

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Ultimate goal: comprehensionUltimate goal: comprehension

Purpose of computing is insight, not numbers

Richard HammingNumerical Methods for Scientists and Engineers 1962

Purpose of visualization is insight, not pretty pictures

Visualization community

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

RecapRecap

� Visualization is a tool, an internal interface in the scientific discovery loop

� Visualization helps exploration besides presentation and hypothesis-driven research

� Visualization goal is not (only) to produce nice images, but to gain insight

Chemistry VisualizationChemistry Visualization

Now apply visualization principles to chemistry dat aNow apply visualization principles to chemistry dat a

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Chemistry segmentationChemistry segmentation

Materialscience

Crystallo-graphy

Genomic

Teachingsupport

Nanostructures

…

Moleculardynamics

ChemistryChemistry

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Chemists and modelsChemists and models

9

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Dogma: visualization helps insight Dogma: visualization helps insight

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Different clarity, why?Different clarity, why?

HEMOGLOBIN (VAL BETA1 MET, TRP BETA37 TYR) MUTANT

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Chemistry dataChemistry data

Conceptualmodel

Conceptualmodel

DataStudy object

Datamodel

Datamodel

acquisition

Preconceptions &interpretation

Influen

ce

Interaction

Algorithms

RenderPerceptio

n

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Do you remember GIGO?Do you remember GIGO?

GIGO /gi:'goh/ [acronym]1. Garbage In, Garbage Out : usually said in response to users who complain that a program didn't "do the right thing" when given imperfect input or otherwise mistreated in some way.

2. Garbage In, Gospel Out : this more recent expansion is a sardonic comment on the tendency human beings have to put excessive trust in “computerized” data.

Source: Jargon File 4.2.0

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Data influences visualizationData influences visualization

If you don’t know what the data represent this visualization is as good as any other.

Instead if you know the dataset content the visualization can foster insight because it is tuned to what the data represent.

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Chemistry data kindsChemistry data kinds

O

O

O

OH

Data from prof. A. Oganov – ETH Zürich

Structures

Scalar volumes

Spectra

Tables

10

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Time dependent dataTime dependent data

Images from AmiraMol

Sergey Churakov – PSI Villigen

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Death Receptor Signaling pathway

Gene expression

Not only numerical dataNot only numerical data

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Information VisualizationInformation Visualization

Scientific VisualizationPrimarily focused on physical dataData normally geometry basedScalar, vector and tensor data types

Information visualizationFocused on abstract and non physical dataNormally abstract, non geometric dataMultidimensional and non numeric data types

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Scientific visualization (streamlines)Scientific visualization (streamlines)

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Non spatial information Non spatial information

Not all data has an associated geometry (web site accesses, marketing data, etc.). Other have only an associated topology (trees, graphs).

A geometry should be assigned to visualize them.

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Think multidimensionalThink multidimensional

7 atoms x 3 coordinates each = trajectory in a 21-dimensional space

x1

x2

x3

x4

x21

11

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Represent dataRepresent data

Conceptualmodel

Conceptualmodel

DataStudy object

Datamodel

Datamodel

acquisition

Preconceptions &interpretation

Influen

ce

Interaction

Algorithms

RenderPerceptio

n

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Traditional representationsTraditional representations

� Ball & Stick (and its derivatives)

� Surfaces

� Spectra and line charts

� 2D charts & contours

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Simplified representationsSimplified representations

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Representation constraintsRepresentation constraints

“A stagnant set of representations limits the

way scientists think about their models and

thereby limits potential insights”

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Example: join the nine dots gameExample: join the nine dots game

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Wrong solution (5 lines)Wrong solution (5 lines)

12

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Right solution (4 lines)Right solution (4 lines)

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Unstated constraintUnstated constraint

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Why limit yourself?Why limit yourself?

http://www.gihanperera.com/mindgames/dots9ans.html

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Example: ice meltingExample: ice melting

Davide Donadio – ETH Zürich

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Example: mission impossibleExample: mission impossible

Unusual carbocations

More examples at: http://chemgroups.ucdavis.edu/~tantillo

Unbuildable molecules

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Visualization algorithmsVisualization algorithms

Conceptualmodel

Conceptualmodel

DataStudy object

Datamodel

Datamodel

acquisition

Preconceptions &interpretation

Influen

ce

Interaction

Algorithms

RenderPerceptio

n

13

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Visualization algorithmsVisualization algorithms

Data from prof. A. Oganov – ETH Zürich

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

3D is not everything3D is not everything

3D structure displaysare valuable tools, but ...

� limited to viewing part of structure

� unsuited for quick comparisons

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

2D plots are still important2D plots are still important

2D structure plots are still the coreof chemical information:

� show complete structure

� easy recognition of patterns

Chemists know how good structures looks like.

N

Ni

N

NN

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Try to reconstruct 3D from 2DTry to reconstruct 3D from 2D

Top view

Front view

Side view

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

3D to understand shape3D to understand shape

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

How to convey depth perceptionHow to convey depth perception

Without With depth cueing

14

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Shadows and modelsShadows and models

No shadow Hard shadow Soft shadowBeyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Using artificial depth cuesUsing artificial depth cues

Drop lines / projections� Can clutter the image

Occlusion� More natural� But hides part of the data

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Human perceptionHuman perception

Conceptualmodel

Conceptualmodel

DataStudy object

Datamodel

Datamodel

acquisition

Preconceptions &interpretation

Influen

ce

Interaction

Algorithms

RenderPerceptio

n

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Influence of color perception Influence of color perception Gaussian cube

with default colorm

ap

Adjusted data range

Better color mapping

Perceptually tuned

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Eye attractorsEye attractors

Smooth bonds Two colors bonds

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Interact with the representationInteract with the representation

Conceptualmodel

Conceptualmodel

DataStudy object

Datamodel

Datamodel

acquisition

Preconceptions &interpretation

Influen

ce

Interaction

Algorithms

RenderPerceptio

n

15

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Reduces 3D ambiguitiesReduces 3D ambiguities

Illusion due to thefixed viewpoint

The famous Ames Room optical illusion

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Lets interact and exploreLets interact and explore

Interactive element tableChanges to the atomic properties with the cursors are immediately reflected as highlighted elements

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Building a volume mental modelBuilding a volume mental model

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Use touch (and maybe smell…)Use touch (and maybe smell…)

Tangible model

The system adds to the real image other molecules or show the electrical field around the solid model http://www.scripps.edu/mb/olson/pyartk/pyartk.html

Breaking barriersBreaking barriers

Where the true power of visualization liesWhere the true power of visualization lies

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Data fusionData fusion

Data from different sources and of different kind can be fused together with great advantage:

� Create an interpretative context for the data

� Suggest correlations

� Highlight cause-effect relationships

16

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Charts

Data are not ‘single’Data are not ‘single’

Loosely correlated charts

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

2D GIS view

Charts

Data are not ‘single’Data are not ‘single’

Now fused in a 2D view plus GIS context map

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

2D GIS view

3D Data FusionCharts

Data are not ‘single’Data are not ‘single’

Complete (and useful) data

fused together

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Mix standard techniquesMix standard techniques

Electronic ring currents in benzene

Data simulation by Daniel Sebastiani, Max Planck Institute

Winds over Europe

Daily weather forecast model computed at CSCS for MeteoSwiss

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Mix standard techniquesMix standard techniques

Volume rendering

Vector glyphs, Line Integral Convolution (LIC) and alpha blending to cut the hole

Sergey Churakov – PSI Villigen

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Enlarge visualization spaceEnlarge visualization space

Continuum step

Atomistic step

17

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Example: fluid dynamicsExample: fluid dynamics

Vladimir Slavin – Brown University Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Consider other data typesConsider other data types

Brain diffusion tensors

Isotropic Component of NMR 31P Shielding Tensor

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Multidimensional visualizationMultidimensional visualization

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Multiple linked viewsMultiple linked views

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Data MiningData Mining

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

Crystallographic data miningCrystallographic data mining

Usage of computed simplified X-Ray structure factors to detect structural changes

Data from prof. A. Oganov – ETH Zürich

18

Beyond Ball-and-Stick Tutorial – Mario Valle – CSCS User Day 27/09/2005

RecapRecap

� The importance of choosing the right representation

� There are (virtual) limits from the usual chemistry visualization methods

� Big advantages from going beyond the usual horizons

Beyond BallBeyond Ball --andand --StickStick

Thanks for your attention!Thanks for your attention!

Mario ValleMario Valle

mvalle@cscs.chmvalle@cscs.chhttp://www.cscs.ch/~mvalle/ http://www.cscs.ch/~mvalle/