1

Visualizing Big Data

(Many Cases & Dimensions)

CS 7450 - Information Visualization

April 12, 2011

John Stasko

Topic Notes

Spring 2011 CS 7450 2

Previously

• We looked at a number of techniques for projecting >2 variables down onto the 2D plane

Parallel coordinates

Scatterplot matrix

Table lens

etc.

2

Spring 2011 CS 7450 3

Varieties of Techniques

Spring 2011 CS 7450 4

Potential Limitations

• What happens when you have lots and lots of data cases?

3

Spring 2011 CS 7450 5

Parallel Coordinates

Out5d dataset (5 dimensions, 16384 data items)

(courtesy of J. Yang)

Spring 2011 CS 7450 6

Potential Limitations

• Or, you may have many, many variables

Hundreds or even thousands

4

Strategies

• How are we going to deal with such big datasets with so many variables per case?

• Ideas?

Spring 2011 CS 7450 7

General Notion

• Data that is similar in most dimensions ought to be drawn together

Cluster at high dimensions

• Need to project the data down into the plane and give it some ultra-simplified representation

• Or perhaps only look at certain aspects of the data at any one time

Spring 2011 CS 7450 8

5

Spring 2011 CS 7450 9

Mathematical Assistance 1

• There exist many techniques for clustering high-dimensional data with respect to all those dimensions

Affinity propagation

k-means

Expectation maximization

Hierarchical clustering

Spring 2011 CS 7450 10

Mathematical Assistance 2

• There exist many techniques for projecting n-dimensions down to 2-D(dimensionality reduction)

Multi-dimensional scaling (MDS)

Principal component analysis

Linear discriminant analysis

Factor analysis

Data miningKnowledge discovery

Comput Sci & Eng coursesVisual Analytics, Prof. Lebanon

6

Spring 2011 CS 7450 11

Other Techniques

• Other techniques exist to reduce data

Sampling – We only include every so many data cases or variables

Aggregation – We combine many data cases or variables

Spring 2011 CS 7450 12

Our Focus

• Visual techniques

• Many are simply graphic transformations from N-D down to 2-D

7

Example

• Big document collection

• Accumulate all different words used throughout

• Each word becomes a dimension

• Value of that data case (document) in a dimension is the number of times the word appears in that document

• (May be thousands of dimensions)

Spring 2011 CS 7450 13

PNNL’s SPIRE

Spring 2011 CS 7450 14

Each dot is adocument

Similarity provokesnearby positioning

Will see more laterin term on Text day

Wise et alInfoVis „95

8

Pluses & Minuses

• Can have as many cases as there are pixels and unlimited number of dimensions

• Shows similarity of data cases

• Only a dot for each case

• Doesn‟t say much about dimensions or cases

Spring 2011 CS 7450 15

Spring 2011 CS 7450 16

Use?

• What kinds of questions/tasks would you want such a technique to address? Clusters of similar data cases

Useless dimensions

Dimensions similar to each other

Outlier data cases

…

• Think back to our “cognitive tasks” discussion

9

Spring 2011 CS 7450 17

Today

• We‟ll examine a number of other visual techniques intended for larger, high-dimensional data sets

Spring 2011 CS 7450 18

Can We Make a Taxonomy?

• D. Keim proposes a taxonomy of techniques

Standard 2D/3D display

Bar charts, scatterplots

Geometrically transformed display

Parallel coordinates

Iconic display

Needle icons, Chernoff faces

Dense pixel display

What we‟re about to see…

Stacked display

Treemaps, dimensional stacking

TVCG „02

10

Spring 2011 CS 7450 19

Dense Pixel Display

• Represent data case or a variable as a pixel

• Million or more per display

• Seems to rely on use of color

• Can pack lots in

• Challenge: What‟s the layout?

Spring 2011 CS 7450 20

One Representation

Each variable is in a windowData cases in grid in each window

Similarity ofwindow viewstells you aboutsimilarity ofdimensions

Uses color scale

11

Spring 2011 CS 7450 21

Alternative

• Grouping arrangement

• Doesn‟t use multiple windows

• Each data case has its own small rectangular icon

• Plot out variables for data point in that icon using a grid layout

Spring 2011 CS 7450 22

Another View

LevkowitzVis „91

12

Spring 2011 CS 7450 23

Example Large View

Spring 2011 CS 7450 24

DB Applications

• Database of data items, each of n dimensions

• Issue a query that specifies a target value of the dimensions

• Often get back no exact matches

• Want to find near matches

Keim & KriegelIEEE CG&A „94

13

Spring 2011 CS 7450 25

Relevance Factor

• How close an item is to the query

• Data items have some value that can benumerically quantified

• Each dimension is some distance awayfrom query item

• Sum these up for total distance• Relevance is inverse of distance

Spring 2011 CS 7450 26

Example

• 5 dimensions, integers 0->255

• Query: 6, 210, 73, 45, 92

• Data item: 8, 200, 73, 50, 91

• Distance: 2 + 10 + 0 + 5 + 1 = 18

• Relevance: 1275 - 18 = 1267

14

Spring 2011 CS 7450 27

Issues

• What if dimensions are real numbers or text strings?

• What if they‟re the same type, but of different orders of magnitude?

• Have to define some kind of distance, then a weight function to multiply by

Spring 2011 CS 7450 28

Technique

• Calculate relevance of all data points

• Sort items based on relevance

• Use spiral technique to order the values –Emanate out from center

• Color items based on relevance

15

Spring 2011 CS 7450 29

Relevance Colors

High Low

Empirically established

Spring 2011 CS 7450 30

Technique

0

21

3

456

109

8

7

16

Spring 2011 CS 7450 31

Spiral Method

Highest relevancevalue in center,decreasing valuesgrow outward

Spring 2011 CS 7450 32

Display Methodology

Example: five-dimensional data

Totalrelevance

Dim 1 Dim 2

Dim 3Dim 4Dim 5

Spiralin eachwindow

Items ordered by total relevance

Same itemappears insame placein each window

17

Spring 2011 CS 7450 33

Example Display

Spring 2011 CS 7450 34

Alternative

• Grouping arrangement

• Doesn‟t use multiple windows

• Create all relevance dimensional depictions for an item and group them

• Spiral out the different data items‟ depictions

18

Spring 2011 CS 7450 35

Grouping Arrangement

Spring 2011 CS 7450 36

Example Display

Multi-window Grouping

8 dimensions

1000 items

19

Related Idea

• Pixel Bar Chart

• Overload typical bar chart with more information about individual elements

Spring 2011 CS 7450 37

Keim et alInformation Visualization „02

Spring 2011 CS 7450 38

Idea 1

Height encodes quantity Width encodes quantity

20

Idea 2

• Make each pixel within a bar correspond to a data point in that group represented by the bar

Can do millions that way

• Color the pixel to represent the value of one of the data point‟s variables

Spring 2011 CS 7450 39

Spring 2011 CS 7450 40

Idea 3

Each pixel is a customerColor encodes amount spent by that person

High-bright, Low-darkOrdered by that color attribute tooRight one shows more customers

21

Idea 4

Spring 2011 CS 7450 41

Product type is x-axis dividerCustomers ordered by

y-axis: dollar amountx-axis: number of visits

Color is (a) dollar amount spent, (b) number of visits, (c) sales quantity

Idea 5

Spring 2011 CS 7450 42

Can divide on two differentattributes on x and y

Order items on both x and y

Color maps to someattribute(Same item always atsame x,y position)

22

Idea 6

Spring 2011 CS 7450 43

Mapping specified by 5 tuple <Dx, Dy, Ox, Oy, C>

Dx – Attribute partitions x axisDy – Attribute partitions y axisOx – Attribute specifies x orderingOy – Attribute specifies y orderingC – Attribute specifies color mapping

Example Application

Spring 2011 CS 7450 44

1. Product type 7 and product type 10 have the top dollar amount customers (dark colors of bar 7 and 10 in Figure 13a)2. The dollar amount spent and the number of visits are clearly correlated, especially for product type 4 (linear increase of dark colors at the top of bar 4 in Figure 13b)3. Product types 4 and 11 have the highest quantities sold (dark colors of bar 4 and 11 in Figure 13c)4. Clicking on pixel A shows details for that customer

23

Thoughts?

• Do you think that would be a helpful exploratory tool?

Spring 2011 CS 7450 45

High Dimensions

• Those techniques could show lots of data, but not so many dimensions at once

Have to pick and choose

Spring 2011 CS 7450 46

24

Another Idea

• Use the dense pixel display for showing data and dimensions, but then project into 2D plane to encode more information

• VaR – Value and relation display

Spring 2011 CS 7450 47

Yang et alInfoVis „04

Algorithm

• Find a correlation function for comparing dimensions

• Calculate distances between dimensions (similarities)

• Make each dimension into a dense pixel glyph

• Assign position for each glyph in 2D plane using multi-dimensional scaling

Spring 2011 CS 7450 48

25

Spring 2011 CS 7450 49

Slide 1

Spring 2011 CS 7450 50

IndividualDimensions

26

Spring 2011 CS 7450 51

IndividualDimensions

Let’s take a

closer look...

Spring 2011 CS 7450 52

The values of the data points mapped to colors

of pixels in a given glyph...

27

Spring 2011 CS 7450 53

The values of the data points mapped to colors

of pixels in a given glyph...

Spring 2011 CS 7450 54

The values of the data points mapped to colors

of pixels in a given glyph...

1st

2nd

3rdetc...

28

Questions

• What order are the data cases in each dimension-glyph?

Maybe there is a predefined order

Choose one dimension as “important” then order data cases by their values in that dimension

“Important” one may be the one in which many cases are similar

Spring 2011 CS 7450 55

Spring 2011 CS 7450 56

Reordering Data

• Two different orderings of cases shown below (a-dimension near top is prototype, b- dimension near bottom is prototype)

29

Comparing dimensions

Spring 2011 CS 7450 57

One dimension chosenas focus

Others shaded for howsimilar they are

dark – big differencelight – small difference

Interaction

Spring 2011 CS 7450 58

a – lots overlapsb – shrink sizec – jitter positiond – enlarge some

30

Spring 2011 CS 7450 59

Contributions

• Highly scalable way to view dimensional relationships

• Computationally efficient

• Uses MDS for dimensions, not just data cases

Limitations

• Those glyph overlaps are a problem

• Similar dimensions are positioned near each other with lots of overlap

Spring 2011 CS 7450 60

31

Follow-on Work

• Use alternate positioning strategies other than MDS

• Use Jigsaw map idea (Wattenberg, InfoVis „05) to lay out the dimensions into a grid

Removes overlap

Limits number that can be plotted

Spring 2011 CS 7450 61

Yang et alTVCG „07

New Layout

Spring 2011 CS 7450 62

Plot the glyphsinto the grid positions

32

HCE

• Hierarchical Clustering Explorer

• Implements “rank by feature” framework

• Help guide user to choose 1D distributions and 2D scatterplots from various dimensions of a data set

• Combine statistical analysis with user-directed exploration

Spring 2011 CS 7450 63

Seo & ShneidermanInformation Visualization „05

Idea

• Choose a feature detection criterion to rank 1D and 2D projections of a data set

• Use person‟s perceptual abilities to pick out interesting items from view

Spring 2011 CS 7450 64

33

HCE UI

Spring 2011 CS 7450 65

Cases incolumns,variablesin rows

Group similarcases

Seven tabs at bottom to choose from

Some chosen distributions and scatterplots

Operation

• When you choose the histogram ordering or scatterplot ordering tabs at the bottom left, these give results based on various statistical measures

• You can then choose some of them to visualize

Spring 2011 CS 7450 66

34

Spring 2011 CS 7450 67

Demo

HW 8 Return

• Solution

Spring 2011 CS 7450 68

35

Spring 2011 CS 7450 69

Upcoming

• Evaluation Reading:

PLaisant

• Casual InfoVis Reading:

Pousman et al