Visualization and Persuasion

Amitabh VarshneyDepartment of Computer Science

University of Maryland at College Parkhttp://www.cs.umd.edu/~varshney

Slide 2

The Biggest Challenge of Our Era Data and Information Overload

Data produced during 2003 – 2005 exceeds all of thepreviously created data by mankind

… and 90% of the new data is digital

Acquisition:• Audioscapes, Image and Video Archives• GIS (satellites to low-flying planes), Laser Scanners• Medical Imaging (CT, MRI, fMRI, …)• Molecular Imaging (NMR, Crystallography, Microarrays, …)

Generation:• Meteorology: Atmospheric simulation and prediction• Biology: Molecular dynamics, protein docking• Space Sciences: Solar wind turbulence, micro-gravity fluid mechs.• High-Energy Physics: Multi-scale Plasma Dynamics

Slide 3

UMD CPU-GPU Cluster• 15 Compute Nodes and 13 Render Nodes• 4 Nodes for Scheduling and File Server• 1 User node (keyboard, terminal)• Each node has:

– Dual 3GHz Intel Xeon CPUs– NVIDIA GeForce 7800 GTX GPU– 8 GB RAM– 100 GB Disk

• Parallel fileserver has 10 TB disk storage• Network interconnect: 10 Gbps Topspin

Infiniband

Slide 4

UMD Active Tiled Display Wall

Total resolution: 50 mega pixels(5 x 5 array of 1920 x 1200 Dell 24” LCD screens)Ultrasonic air-mouse interface

Slide 5

Rhetorics

Rhetoric is the faculty of discovering in any particularcase all of the available means of persuasion

– Aristotle (Book I)

• Aristotle assumed that the agent of persuasion is the oratorand the medium of persuasion is the language

• Vast number of elements in any discussion

Rhetor’s ultimate goal, whenever possible, is to makethe most relevant object, concept, or value fill theaudience’s entire “field of consciousness”.Perelman & Olbrechts-Tyteca, The New Rhetoric: A Treatise on Argumentation, 1971

Slide 6

Communication – Language vs Visual

• Language: linear, apprehended slowly, systematicprocessing

• Images: Comprehended wholistically, instantaneously,emotional processing

• Images (or image-evoking language) are powerful,precisely because they provide superior opportunities todominate the field of consciousness

Slide 7

Visual Persuasion

• Advertising

• Criminal-Justice System

• Political Campaigns

• State Propaganda

• Religion, and many others …

… touches every part of our lives.

Most rely on Aristotle’s pathos, sometimes on ethos,seldom on logos

Slide 8

Visual Persuasion in Geology

from The Map That Changed the World by Simon Winchester

William Smith (1769 – 1839)

• Observed stratification of rocks during his work on coal mines and later canals

• The order of rocks were the same

• The depths at which they occurred differed

• Hypothesized a 3D structure of layers at various slopes beneath the Earth’s surface

• Used measurements along canals, mine shafts, and extrapolated…

Slide 9

Visual Persuasion in Geology

from The Map That Changed the World by Simon Winchester

• The first systematic spatialstudy of various rock strata

• Used fossil types in variouslayers of rocks to fine-tune

• Created the first geologicalmap in 1815•Visually persuaded the world ofthe innate structure of theEarth; immensely influential in:

•Geology

•Natural History and Biology

•Mining coal, oil, iron, diamond,platinum, silver, …

Slide 10

Visual Persuasion in GeoPhysics

From Pitman and Heirtzler, Science, Dec 1966

Magnetic Profile and Bathymetry data from USS Eltanin, track 19

Slide 11

Visual Persuasion in GeoPhysics• The first visual proof of symmetry of magnetic anomalies• Visually persuasive evidence leading to the acceptance ofthe theory of sea-floor spreading

Poster source: NOAAImage from: The Earth by Tarbuck and Lutgens

Slide 12

Visual Persuasion in Music

Beethoven’s Für Elise

Mary had a Little Lamb

Arc Diagrams by Martin Wattenberg, IEEE InfoVis 2002

Slide 13

General Observations• Common amongst examples

– Carefully hand-crafted– Took immense effort– One of a kind

• General principles:– Results convey important elements in a clean and

visually consistent way– They emphasize salient features and suppress others– They are parsimonious visualizations and they make

the few resources work extra hard (space, color, …)• Building Blocks

– Visual attention– Data salience

Slide 14

Visual Attention A way of seeing is also a way of not seeing – a focus on

object A involves a neglect of object BKenneth Burke

• Where we look has significant implications for:– what we perceive– how we interpret– how we act

• Visual attention is the primary filter by which we can copewith our immense sensory bandwidth– Retinal information is too vast and most of it has no survival value

• Eye-tracking can quantify overt visual attention

Slide 15

What Controls Visual AttentionBottom-up Image Properties (task independent)

Top-down Semantics and Task-driven Properties

[Yarbus 1967]

[Yarbus 1967]

Slide 16

Related Work• Image saliency maps

– Tsotsos et al. 95, Milanese et al. 94, Itti et al. 98, Rosenholtz 99

• Applications: compression and cropping– Privitera and Stark 99, Chen et al. 03, Suh et al. 03

Itti et al. PAMI 98

Suh et al. UIST 03

Saliency-basedcropping

Without cropping

Slide 17

Distinctive 3D structure pops out pre-attentively

Saliency has a 3D Component

Based on Enns and Rensink 90

3D featurespop out quickly

2D featuresnot pre-attentive

Slide 18

Mesh SaliencySaliency should find regions that aredifferent from their surrounding context

Curvature SaliencyArmadillo Leg

Lee, Varshney, Jacobs, SIGGRAPH 2005

Slide 19

Mesh SaliencySaliency should capture interestingfeatures at all meaningful scales

Saliency at a large object scale Saliency at a small object scale

Slide 20

Mesh Saliency

Saliency Computation Overview

Curvature Saliency Maps at multiple scales

NonlinearNormalization

Center-Surround

Slide 21

!

!

"

"

##

##

=

),v(Nx

),v(Nx

)]/(vxexp[

)]/(vxexp[)x(

)),v((G

$

$

$

$

$

2

22

2

22

2

2C

C

C ( x ): Mean curvature at vertex v

Gaussian-weighted average is:

Gaussian Weights

Center-Surround Operator

Slide 22

{ } object theof diagonal theof 0.3% 65432 =! """"""# ,,,,,i

)),v((G)),v((G)v( iii !! 2CCS "=

Center-Surround OperatorSaliency map at each scale i is:

Slide 23

{ }43210

SSSSSS ,,,,i!{ }

object the of diagonal the of %.å

,,,,i

30

65432

=

! """""#

)),v((G)),v((G)v( iii !! 2CCS "=

Center-Surround Operator

Saliency map at each scale i is:

Slide 24

Mesh Saliency Results

Stanford Armadillo Cyberware Isis

Slide 25

Visual Computing• The process of transforming data to images involves

– Dimension explosion – add normals, reflectance, camera,lighting, transfer functions, …

– Dimension reduction – incorporate everything to give us a grid ofcolors and their locations

• How is data salience propagated through such apipeline?– Are we certain it survives?

• Every communication has a message– And every point and every voxel a default importance

• How can we preserve data salience in rendered images?

Slide 26

Persuasive Visualization• Define Salient

– Human– AI assistant

• Identify Salience– Features– Patterns– Coherence– Interactions

• Preserve it through the Visual Computing pipeline:– Camera– Geometry / Data Representation– Transfer Functions– Lighting– Texture, Color, Glyphs

• Validate in final rendering

Slide 27

Simplification: Scale the quadric error by the saliency topreserve more triangles for salient regions

Viewpoint Selection: Find the viewpoint that maximizes thesum of the visible saliency• Gradient-descent-based optimization

Cyberware Male Mesh Saliency

Saliency Applications

Slide 28

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

Viewpoint Selection

Curvature

Saliency

Sum ofvisible

curvature

Sum ofvisible

saliency

Slide 29

Preserving Salience• Transfer Functions: Modulate opacity/transfer

functions [Groeller et al. TVCG 04 –VisSym 07]

• Rendering Stylization: [Kim and Varshney]

Original Suggestive Contours SC weighted by Saliency

Slide 30

Lighting in Photography

from Envisioning Science by Felice Frankel

Slide 31

Visual Continuity & Consistency• Shots often composited from multiple takes

• Lighting is painstakingly made consistent

• Consistent lighting considered important for visualcontinuity and storytelling

Slide 32

Is Consistent Lighting Necessary?• Nature has one dominant light source

• Evolution might have endowed us with anability to discern inconsistency in illumination

• Just as it has inconsistency in perspective

Analysis of Perspective by Chris Tyler, Smith-Kettlewell Institute

Slide 33

Illumination Inconsistencies

Illumination inconsistencies are not perceptually salientOstrovsky, Sinha, Cavanagh, Perception 2006

Recent research suggests that illuminationconsistency is not resolved at the low-levelhuman vision

Find the cube lit inconsistentlywith respect to others:

On average, users take 8 seconds to answer and are then wrong 30% of the time

Slide 34

George Washington Crossing the Delawareby Emanuel Gottlieb Leutze

Discrepant Lighting in Art

Slide 35

Discrepant Lighting in Art

The Music Lesson by Jan Vermeer

Slide 36

Light Collages: Basic Idea

Allow local lighting parameters to be definedindependently at local regions

Lee, Hao, Varshney, Visualization 2004, TVCG 2005

Slide 37

Light Collages Overview• Segmentation [Mangan & Whitaker 99]

• Light Placement andAssignment to patches

• Proximity Shadows• Silhouette Enhancement

3D Mesh Segmentation Light Placement& Assignment

SilhouetteEnhancement

ProximityShadows

Slide 38

Results – Pelvis & Skull

(a) Uniform 4 lights (b) Light Collages:with 4 lights

(c) Light Collages:Silhouettes+ Shadows

Slide 39

Manuscript courtesy of Paul Debevec, USC and XYZ RGB Inc.

1 uniform light 4 uniform lights Light Collageswith 4 lights

Results - Manuscript

Slide 40

Visual Persuasion• We can draw viewer attention in several ways• Obtrusive methods like arrows or flashing pixels

– Distracts the viewer from exploring other regions• Principles of visual perception used by artists

and illustrators– Gently guide to regions that they wished to

emphasize

Kim and Varshney, IEEE Visualization 2006

Slide 41

Visual Persuasion (Art)

The Feast of Belshazzar, Rembrandt, 1635

Slide 42

Saliency-Guided Visual Persuasion• Saliency Field• Enhancement Operators• Emphasis Field• Saliency Enhancement• Saliency-enhanced Volume Rendering• Validation by eye-tracking based user

study

Transfer Functions

Saliency Fieldby User Input

Emphasis FieldComputed

EnhancementOperators

Saliency-enhancedVolume Rendering

Validation by eye-tracking device

SaliencyEnhancement

Slide 43

Emphasis Field Computation

• Mesh Saliency: S (v) = G(C, v, σ) – G(C, v, 2σ)• We introduce the concept of an Emphasis Field E

to define a Saliency Field S in a volumeS (v) = G(E, v, σ) – G(E, v, 2σ)

Given a saliency field, can we design somescalar field that will generate it?

KnownUnknown

Known Unknown

Slide 44

Emphasis Field Computation• Expressible as simultaneous linear equations

• Saliency Enhancement Operator (C-1)– CE =S , which implies E = C-1S– Given a saliency field S , the enhancement operator C-1

will generate the emphasis field E

where cij is the difference between two Gaussianweights at scale σ and at scale 2σ for a voxel vjfrom the center voxel vi

=

Slide 45

Emphasis Field Computation• We like to use enhancement operators at multiple

scales σi– Let E i be the emphasis field at scale σi

– Compute this by applying the enhancement operator Ci-1 on

the saliency field S– Final emphasis field is computed as the summation of E i

Slide 46

Emphasis Field in Practice• A system of simultaneous linear equations in n

variables– Generally, can handle arbitrary saliency regions and values– Computationally expensive: O(kn2) or O(n3)

• Alleviate this by solving a 1D system of equations

– Given a saliency field– Solve 1D system of equations at

multiple scales and sum them up– Approximate results using

piecewise polynomial radialfunctions [Wendland 1995]

• Interpret results to be along the radial dimension– Assume spherical regions of interest (ROI)

Slide 47

Saliency-guided Persuasion• Previous Gaussian-based Enhancement of a Volume

– Volume Illustration [Rheingans and Ebert TVCG 01]– Importance-based regional enhancement

• Various rendering stylizations and effects possible– Brightness– Saturation

Slide 48

Visual Persuasion - Brightness

Traditional VolumeRendering

Gaussian-basedEnhancement

Saliency-guidedEnhancement

Traditional VolumeRendering

Gaussian-basedEnhancement

Saliency-guidedEnhancement

Slide 49

User Study• Validated results by an eye-tracking-based

user study• Eye-tracker ISCAN ETL-500

• Records eye movements at 60Hz

• Hypotheses: The eye fixations increase overthe region of interest (ROI) in a volume by thesaliency-guided enhancement compared to– the traditional volume visualization

(Hypothesis H1)– the Gaussian-based enhancement

(Hypothesis H2)

Slide 50

User Study – Results

Traditional Volume RenderingTraditional Volume RenderingWith Fixation Points

Saliency Field Gaussian-based EnhancementGaussian-based EnhancementWith Fixation Points

Saliency-guided EnhancementWith Fixation Points

Saliency-guided Enhancement

Slide 51

Data Analysis

The percentage of fixations on the ROI for the original,Gaussian-enhanced, and Saliency-enhanced

visualizations

Kim and Varshney, IEEE Visualization 2006

Slide 52

Concluding Remarks

• Computational models that mimic perceptualsalience for 3D objects

• Methods to persuasively guide and hold visualattention

• Data is always sacrosanct but visualization isnever neutral– View location, view orientation, lighting (number, and

types), reflectance, transfer functions, …• Every communication has a message

– Whether intentional or not

Slide 53

Concluding Remarks

• Every communication genre employs its distinctrhetorics– Movies vs documentaries– Posters vs newspapers– Photography vs art

• Towards a Language of 3D Visual Communication– Data: nouns– Visual Salience: adjectives– Visual Persuasion: verbs

Slide 54

Acknowledgements

Students: Youngmin Kim, Chang Ha Lee, David Dao, Rob Patro,Derek Juba, Cheuk Ip, Roman Stanchak

Colleagues: David Jacobs, Hamish Carr, Helwig Hauser, FrancoisGuimbretiere, Joseph Ja’Ja’, Chris Johnson, Arie Kaufman,

Raghu Machiraju, Torsten Moeller, Dianne O’Leary

Sponsors: National Science Foundation, DARPA, DTO-IARPA,Army Research Labs, Kitware