exploratory data analysis

Exploratory Data Analysis

John Tukey• Developed these procedures to help one

get a first look at distributions of scores.• What is the shape of the distribution?• Are there any suspicious scores.• Stem and Leaf Display• Box and Whiskers Plot

Stem and Leaf Display

• See the pulse rate data at Exploratory Data Analysis (EDA).

• The scores range from 48 to 104.• We probably want to group them into 5 to

15 intervals.• I’ll use two intervals for the 40’s, two for

the 50’s, etc.

The StemConsists of a column of leading (aka “most significant” digits, the leftmost digits in the scores. I’ll add to the stem the leaves, the trailing (rightmost, least significant) digits of each score

The Stem With Leaves

Next, I’ll arrange the leaves (within each row) from lowest to highest and add a “depth” column.

Leaves Arranged in Order

The Depth Column

• This column tells you how many scores there are in that row and all rows between it and the closer tail of the distribution.

• The row that contains the median has the row frequency in parentheses.

Rotated Display

It looks like a histogram, but the bars made up of the scores.

From this display, can you identify any scores that are odd, compared to most of the other scores?

Box and Whisker Plot

• Median Location = (N + 1)/2 = 97/2 =48.5.• The median will be located between the

48th and the 49th scores from either tail.

Are 40 scores from 68 to 48. Count up 8 more scores, starting with the first 70. The 48th score is a 70, the 49th score is a 70, the median is 70.

The Hinge Location

• = (Median Location + 1)/2. • Drop any decimal on the median location• For our data, hinge location = (48 + 1)/2 =

24.5. • Now, the upper hinge is the 24.5th score

from the upper end of the distribution.

There are 24 scores from 80 up to 104. Go in toward the median one more score. The 25th score from the highest is a 78. The upper hinge is (78 + 79)/2 = 79.

The 26th score from the lowest score is a 64. Move towards the lower tail by one score and you see the 25th score is also a 64. One more, the 24th score is also a 64. The lower hinge is 64.

The H-Spread

• = the difference between the upper hinge and the lower hinge. For our data, 79 - 64 = 15.

• This is the range of the middle 50% of the scores.

• You also know this as the interquartile range.

The Inner Fences

• The upper inner fence = the upper hinge plus 1.5 H‑spreads. For our data, 79 + 1.5(15) = 101.5.

• The lower inner fence is the lower hinge minus 1.5 H‑spreads, 64 - 1.5(15) = 41.5.

• These are invisible fences, they are not plotted.

Adjacent Values

• These are scores that are outside of the middle 50% of the scores but within the inner fences.

• For our data, these will be scores that fall– between 79 and 101.5 or– between 41.5 and 64

Outliers

• These are scores that are beyond the inner fences.

• For our data, these are scores that are– Less than 41.5 or– Greater than 101.5

Outer Fences

• These invisible fences are 3 H-spreads beyond the hinges.

• For our data the lower outer fence is at 79 - 3(15) = 34

• and the upper outer fence is at 79 + 3(15) = 124

• Scores that are beyond the outer fences are called way-outliers.

Drawing the Plot

• Prepare a numerical scale.• Draw a box that extends from the lower

hinge to the upper hinge.• Draw a line through the box at the median.• May also insert a symbol at the mean.• Draw whiskers out to the most extreme

adjacent values on each side

Whiskers

• For our data, the lowest adjacent value is the 48, so we draw the whiskers on the lower end out to 48.

• We do not go all the way out the inner fence unless there is a score there.

• The highest adjacent value is a 99, so we draw whiskers on the upper end out to 99.

Outliers

• Every outlier is plotted with a special symbol, often a O for a regular outlier and an * for a way-outliers.

• Some programs will also print the identification number next to every outlier

• These days, we use statistical software to make these displays and plots rather than doing them by hand.

Plots Produced by SAS

How tall, in inches, is your ideal mate?

Eight Foot Tall Mate !

• That is a WAY-OUTLIER for sure !• Investigation of the original data sheets

revealed that the actual response was 69 inches, not 96 inches.

Exploratory Data Analysis (EDA)

• It is highly recommended that you read the document linked above.

• It includes additional examples and a bit of silliness that might help you remember key concepts.

• Do watch the video clip of the Id attempting to cross an outer fence on the Forbidden Planet.