Measuring Changes in Service in an Established

Telemedicine ProgramElizabeth A. Krupinski, PhD

Arizona Telemedicine Program

University of Arizona

RationaleWhen a telemedicine program first begins, it

is often sufficient to simply report frequencies

of numbers of patients, types of consults, sub-

specialties offered etc. Frequencies are very

useful and can be used as a straight-forward

measure of program success - the higher

the numbers, the greater the success.

For Example From its inception in the 2nd Quarter (Qtr) of 1997

through the 1st Qtr of 2000 the Arizona Telemedicine Program (ATP) conducted 1610 telemedicine consults

60% were Store-Forward (SF) and 40% were Real-Time (RT) interactions

75% were initial and 25% were follow-up consultations

These frequencies give a general idea of what the ATP has done over the past 3 years

RationaleHowever, as a program becomes more

established it is possible to examine these

simple frequencies in more sophisticated ways.

For example, looking at changes over time can

reveal whether consult volume has stabilized or

has it continued to grow, whether certain sub-

specialties are being used more

consistently than others, etc.?

Graphing Graphing is an easy way to illustrate data as a

function of time It is necessary to choose a meaningful unit of

time We typically use yearly quarters

• Provides enough data for statistical analyses• Parallels the seasons, which might affect volume• Easily tracks the Fiscal Year schedule

For Example

0

40

80

120

160

200#

Cas

es

2nd

Qtr

97

3rd

Qtr

97

4th

Qtr

97

1st

Qtr

98

2nd

Qtr

98

3rd

Qtr

98

4th

Qtr

98

1st

Qtr

99

2nd

Qtr

99

3rd

Qtr

99

4th

Qtr

99

1st

Qtr

00

Color coding each year helps make trends stand out * represent points of significant change - see ANOVA panel

*

*

Basic Statistical Analyses There is a wide variety of statistical techniques

that can be used to analyze frequency data Basic Summary Statistics can describe

Central Tendency & Dispersion• These are useful because they give an overall

picture of the data• But, because they summarize the data the time

variable is lost

For Example An examination of the consults provided by the

ATP on a monthly basis reveals:– Mean # consults = 46.00 – Standard Deviation = 21.75– Median = 54.00 – Inter Quartile Range = 35.00– Minimum = 7 consults– Maximum = 87 consults– Skew = -0.342– Kurtosis = -1.023

Advanced Statistical Analyses To examine the data and include the time

variable two other statistical techniques are useful:• Correlation & Regression Techniques• Analysis of Variance (ANOVA) Techniques

More complicated techniques also exist, but the two above can reveal a significant amount of information about changes in data

Regression Plot

0

10

20

30

40

50

60

70

80

90

100

0 1 2 3 4 5 6 7 8 9 10 11 12 13

# C

ases

Month 1

Month 2

Month 3

2nd Qtr 3rd Qtr 1st Qtr 97 98 00

QuarterlyBreakdown

Regression Statistics Regression Equation: Y = 12.91 + 4.97 X

• The value of 4.97 indicates the type (+ or -) & degree of slope of the regression line

Correlation Coefficient: r = + 0.782• Values closer to + 1 indicate a strong linear

relationship Conclusion: The ATP has continued to see a

linear increase in the number of consults with each quarter

Data Dispersion & Regression sesty = 13.76 (average dispersion of data

around the regression line) Heteroscedasticity: dispersion around the

regression line is not constant• Dispersion in the middle of the plot is much

greater than at either end• New sites were being added to the network so

there was a period of significant change and adjustment in the number of consults

# Consults by Site

0

20

40

60

80

100

120

140

160

# C

onsu

lts

1997 1998 1999 2000*

Douglas 3-21-99

Ganado 6-5-98

Nogales 5-2-97

Patagonia 1-13-98

Payson 7-9-97

Springerville 12-8-97

Tuba City 5-27-97

White River 3-4-99

DOC 12-4-97

Site & Date ofFirst Consult

* 1st Qtr 2000 only

ANOVA Analysis Initial omnibus F-test compares between

and within variances to determine if there is an overall difference among the quarters

Post-hoc Fisher’s Protected Least Squares Difference Tests determine exactly which quarters differ

Allows for identification of specific points of difference or change

For Example* F = 7.926, df = 11,23 p < 0.0001 2nd Qtr 1998 represents the first significant

increase in ATP case volume 1st Qtr 1999 represents another significant point of

increase in ATP case volume Otherwise case volume has been fairly consistent

since 3rd Qtr 1998

* see quarterly volume bar graph for illustration

# Medical Sub-Specialties Analyzing the number of sub-specialty

consults provided is another valuable measure of program success

Does a program start out offering lots of services then narrow down to a few, does it continue to offer a variety of services over time or is there another pattern of services provided?

ATP Top 10 Sub-Specialties

0

100

200

300

400

500

600

700

800

Der

m

Psy

ch

Car

d

Ort

ho

Neu

ro

Rhe

um

Ob/

Gyn

End

oc

Oto

rhin

Hem

/Onc

# C

onsu

lts

# Sub-Specialties Analysis Basic Statistics

• Mean # sub-specialties / month = 11.86• Standard Deviation = 3.56• Median = 13.00• Inter Quartile Range = 3.75• Minimum = 5• Maximum = 20• Skew = -0.28• Kurtosis = -0.77

Regression Plot

0

5

10

15

20

25

0 1 2 3 4 5 6 7 8 9 10 11 12 13

# S

pec

ialt

ies

Month 1Month 2Month 3

QuarterlyBreakdown

2nd Qtr 3rd Qtr 1st Qtr 97 98 00

# Sub-Specialties / Quarter

02468

1012141618

# Su

b-Sp

ecia

ltie

s

2nd

Qtr

97

3rd

Qtr

97

4th

Qtr

97

1st Q

tr 9

8

2nd

Qtr

98

3rd

Qtr

98

4th

Qtr

98

1st Q

tr 9

9

2nd

Qtr

99

3rd

Qtr

99

4th

Qtr

99

1st Q

tr 0

0

*

*

* significant increase

Sub-Specialty Analyses Regression: Y = 8.021 + 0.576 X

r = 0.554

sesty = 3.05

The fairly flat slope and moderate r-value suggest a constant relation between quarters & # of sub-specialties

ANOVA: F = 3.758, df = 11,23 p = 0.0036• Post-hoc tests: Significant increase in # of sub-

specialties in 1st Qtr 1998 & 1st Qtr 1999

Sub-Specialty Conclusions The ATP has provided consults in 53 different

sub-specialty services Although dermatology and psychiatry are the

services provided most often, the ATP has consistently provided consults in about 13 different* sub-specialties each month since the program’s inception

*The individual sub-

specialties vary each month

Summary The ATP has had a significant and

consistent increase in teleconsult volume since the program began

Numbers from the 1st Qtr 2000 suggest the trend will continue

The ATP has maintained its goal of being a multi-specialty telemedicine provider

This work was supported by

1) US Dept. Agriculture, Rural Utilities Service Distance Learning and Telemedicine Grant2) US Dept. Commerce, National Telecommunications and Information Administration TIIAP Grant3) Office of Rural Health Policy, HRSA Dept. Health & Human Services Rural Telemedicine Grant Program4) The State of Arizona