What is Your Data Telling You?

Clara Ballantine, SIA OntarioFalls VLC Team CallJanuary 11, 2011

ObjectivesIn the next 30 Minutes we will: Differentiate among data for accountability,

for research and for improvement Review how to effectively display Falls data

using a run chart Analyze your run chart data “by the rules” Consider strategies to share your data

effectively with others

Aspect Improvement Comparison or Accountability

Clinical Research

Aim:

Improvement of care Comparison, choice, reassurance, spur for

change

New knowledge

Methods: Test

observability Test observable No test, evaluate

current performance Test blinded

Bias

Accept consistent bias Measure and adjust to reduce bias

Design to eliminate bias

Sample size

“Just enough” data, small sequential samples

Obtain 100% of available, relevant, data

“Just in case” data

Flexibility of hypothesis

Hypothesis flexible, changes as learning

takes place

No hypothesis Fixed hypothesis

Testing strategy

Sequential tests No tests One large test

Determining if change is

improvement

Run charts or Shewhart charts

No change focus Hypothesis tests (T-tests, F-tests, Chi-square), p-value

Confidentiality

of data

Data used only by those involved in the improvement

Data available for public consumption

Research subjects’ identities protected

Table 2.1: Data for Improvement, Accountability, Research

Data for Improvement, Accountability and Research

Source: The Data Guide: Learning from Data to Improve Healthcare. Developed from Solberg, Leif I., Mosser, Gordon and McDonald, Susan. “The Three Faces of Performance Measurement: Improvement, Accountability and Research.” Journal on Quality Improvement. March 1997, Vol.23, No. 3.

What Do We Need to Know? How much variation do we have? Is this process changing significantly over time? Have our changes resulted in improvement? Are we holding the

improvement?

Run Chart

• A line graph of data plotted over time• Data is kept in time order• Can see flow of data• Helps answer our improvement questions

Before and After Test(Change Made Between Week 7 & Week 8)

8

3

0123456789

10

Before Chance (measure point on week 4) After Change (measure point on Week 11)

Del

ay T

imes

(H

rs)

Case 1

0

2

4

6

8

10

12

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Weeks

De

lay

Tim

e (

Hrs

)

Made Change

Why Use a Run Chart?

I @:@ -1 : - f, I I .

'And this is the period when the cat was away. '

8

Run Chart – Falls 4.0

Baseline

1MD, 1RN trained

Staff Training complete

Team Leader

resigned

New staffstarted

Thanks to: Virginia FlintoftSHN Central Measurement Team

Source: National Nursing Home Improvement Collaborative: Pressure Ulcer Prevention and Treatment Handbook, Qualis Health

Key Point #1

• Annotate your run chart – it tells your story• Annotation helps you see the impact of your

test of change and can help you decide to “adopt, adapt or abandon” a change.

How to Annotate a Run Chart

11

1. Select “Insert” tab

2. Select “Text Box”

How to Annotate a Run Chart

12

3. Enter text in Text Box

4. Right click on Text Box and select “Format Shape “ from menu.

How to Annotate a Run Chart

13

5. Format text box – use your imagination!• fill colour• line colour and style • etc.

% Timely Reperfusion

Months

pe

rce

nt

1/07 2 3 4 5 6 7 8 9 10 11 12 1/08 2 3 4 5 65 7 8 9 10 11 1232 23 32 38 35 35 40 21 38 26 22 27 23 32 36 29 38 42 39 36 50 48 39 44

DateData

Run Chart

1/072 3 4 5 6 7 8 9 10 11 12 1/08

2 3 4 5 65 7 8 9 10 11 12

15

20

25

30

35

40

45

50

55

60

Change 1

Chg 2,3

Chg 4,5,6

Chg 7

Chg 8,9Median 35

MEDIAN

In a series of numbers, the median is physically the middle number .

It has the same number of points equal to it or above it as it has equal to it or below it.

Finding the Median: Reordering the Data504844424039393838383635353532323229272623232221

• To find the median reorder the numbers from high to low and find the number physically in the middle. If you have two numbers left in the middle, add them together and divide by two.

• Excel: place cursor in blank cell and type=MEDIAN(A2:A21) where A2 is the first cell you want to include and A21 the last)

Key Point # 2

• Calculate and show a Median line to see trends and a Goal line

• Analyze your data “by the rules”

Heart Failure Readmission Rates*Source: Residents Satisfaction on Discharge Hand-Off, Peg Bradke, St. Luke’s Hospital, Cedar

Rapids, Iowa, Hospital to Home: Optimizing the Transition January 2009

*Percent of heart failure Residents readmitted for exacerbation of their heart failure.

0

5

10

15

20

25

30

May

-05

Jul-0

5

Sep-

05

Nov

-05

Jan-

06

Mar

-06

May

-06

Jul-0

6

Sep-

06

Nov

-06

Jan-

07

Mar

-07

May

-07

Jul-0

7

Sep-

07

Nov

-07

Jan-

08

Mar

-08

May

-08

Jul-0

8

Sep-

08

Nov

-08

Per

cen

tag

e

Rate (%) Median

GoodAug 06 = Implemented use of new Residents education materials

Jan 07 = Initiated complimentary visits

Rule 1 Six or more consecutive POINTS either all above or all below the

median. Skip values on the median and continue counting points. Values on the median DO NOT make or break a shift.

Median=10Median=11

Rule 1

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25Mea

sure

or C

hara

cter

istic

Median 10

Ott, Ellis, Process Quality Control, McGraw-Hill Book Company, NY, 1975

Source: National Nursing Home Improvement Collaborative: Pressure Ulcer Prevention and Treatment Handbook, Qualis Health

Rule 2Five points all going up or all going down. If the value of two or more successive points is the same count the first one then ignore the identical points when counting; like values do not make or break a trend.

Rule 2

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Mea

sure

or C

hara

cter

istic

Median 11

Olmstead, Pl, “Distribution of Sample Arrangements for Runs Up and Down, Annals of Mathematical Statistics, Vol 17, pp. 24- 33, March, 1945

Rule 3• To Determine The Number of Runs Above and Below the Median:

– A run is a series of points in a row on one side of the median. Some points fall right on the median, which makes it hard to decide which run these points belong to.

– So, an easy way to determine the number of runs is to count the number of times the data line crosses the median and add one.

– Statistically significant change signaled by too few or too many runs.

Rule 3

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10

Measure

or C

hara

ceristic

Median 11.4

Data line crosses onceToo few runs: total 2 runs

Rule 3: # of RunsTable for Checking for Too Many or Too Few Runs on a Run Chart

Total number of data

points on the run chart

that do not fall on the

median

Lower limit for the number of runs(< than this number of runs is “too few”)

Upper limit for the number of runs(> than this number of runs is “too many”)

10 3 9

11 3 10

12 3 11

13 4 11

14 4 12

15 5 12

16 5 13

17 5 13

18 6 14

19 6 15

20 6 16

21 7 16

22 7 17

23 7 17

24 8 18

25 8 18

Table is based on about a 5% risk of failing the run test for random patterns of data. Frieda S. Swed and Churchill Eisenhart, (1943). “Tables for Testing Randomness of Grouping in a Sequence of Alternatives. Annals of Mathematical Statistics. Vol. XIV, pp.66 and 87, Tables II and III

RULE 4For detecting unusually large or small numbers:

• Data that is Blatantly Obvious as a different value• Everyone studying the chart agrees that it is unusual• Remember:

– Every data set will have a high and a low - this does not mean the high or low are astronomical

Rule 4

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Measure

ment or C

hara

cte

ristic

What do “They” Want to Know?Senior leaders: How are we doing? How do we compare with others?Falls project team: What is the impact of our work? Where do we focus our efforts?Staff How is this helping our patients/residents/clients? What can we do?

Engage Senior Leaders With Data and Evidence

The "whiskers" depict the 95th% CI of the National Mean. Your mean is statisically higher if

it is above the "whiskers"; it is the same as the National Mean if it is within the "whiskers"; and it is statistically lower if it is below the "whiskers".

INTERVENTION - FALLS: 3.0 Percentage of Residents with Completed Fall Risk Assessment on Admission

0%20%40%60%80%

100%

Month

Percent

Local Team National Goal

Bring Data to Improvement Teams Frequently: Process

1MD, 1RN trained

Baseline

Staff Training complete

Team Leaderresigned

New staffstarted

Median

Bring Data to Improvement Teams Frequently: Outcome

1.0 Falls Rate per 1000 Resident Days

0

2

4

6

8

10

12

MonthFalls

/ 100

0 D

esid

ent D

ays

Actual Goal

Post Process Measure Data for All Staff to See Progress

5.0 Percentage of "At Risk" Residents with Falls Prevention/Protection Intervention Implemented

0%10%20%30%40%50%60%70%80%90%

100%

Month

Perc

enta

ge w

ith Im

plem

ente

d Fa

lls P

reve

ntio

n/Pr

otec

tion

Actual Goal

Median

Post Process Measure Data for All to See Opportunities for Improvement

3.0 Percentage of Residents with Completed Fall Risk Assessment

0%10%20%30%40%50%60%70%80%90%

100%

Month

Perce

ntage

with

Fall R

isk

Asse

ssme

nt

Actual Goal

Key Point #3

• Share your data effectively with others at all levels of your organization

• Consider your audience when choosing which data and how to display

For Help With Your Data and Measurement Questions

Virginia Flintoft Clara Ballantine

416.946.8350 613.736.9142

virginia.flintoft@utoronto.ca clara.ballantine@qhn.ca

Dannie Currie Alexandru Titeu

902.842.0716 416.946.3103

curried@cbdha.nshealth.ca shn.ea@utoronto.ca

Chantal Bellerose

514.340.8222  ext 4901

cbellerose@jgh.mcgill.ca