Key Performance Indicators, Centre Reports, and more
Stephen McDonald
Barbecue talk
20
40
60
80
100
120
per
mill
ion
per
yea
r
1970 1980 1990 2000 2010Year
Rate 95% CIIncident rate
Incident RRT, Australia only
More “good” news
5
10
15
20
30
40
50
60
70
50
100
150
200
0
500
0
200
400
600
0
100
200
300
1980 1990 2000 2010 1980 1990 2000 2010 1980 1990 2000 2010
0-24 25-34 45-54
65-74 75-84 85+
Rate 95% CI
per
mill
ion
per
yea
r
Year
Graphs by age group
AustraliaAge-specific incident RRT rates
Indigenous incidence rates
0
100
200
300
400
500
Per
mill
ion
per y
ear
1985 1990 1995 2000 2005 2010Year
Rate 95% CI
Aboriginal & TSI, Australia
Background
• A number of ongoing work themes exist within ANZDATA for generating output– Stock and flow figures– Annual Report– Contributor requests
• Responses to information needed for various projects
– Research projects (internal and external analyses)
– Outcomes reporting
Outcomes reporting
• Recent years have seen a growth of interest in outcomes reporting
• Centre reports have been part of ANZDATA for many years, with increasing emphasis in recent years– At “parent hospital level”– Limited distribution historically
Why measure outcomes?
.2
.5
1
2
4
Obs
erv
ed /
Exp
ecte
d m
ort
alit
y
0 20 40 60 80units
O/E 98% CI
All Australia & NZ Dialysis Units, 98% confidence intervals
Dialysis outcome
.2
.5
1
2
4
Adj
ust
ed r
ela
tive
risk
0 20 40 60 80
Units, ranked by RR
RR 95% CI
Mortality rate during dialysis treatment in Australia 2006-10, adjustedfor demographics and comorbidities
Variation in transplant outcomes
.25
.5
1
2
5
10
20
50R
R g
raft
failu
re
0 5 10 15 20Units
RR 95% CI
Fully adjusted 1 year graft survival, by unitAll transplant units, Australia and New Zealand, patients transplanted2005-2019, followup to 2010
What is happening to centre reports?
• Greater reporting of demographics and comorbidities
• Adjusted analyses in transplanting centre and dialysis reports– Details of models supplied
• Graphs– Funnel plots– CUSUM plots (transplant)
Centre reports – graph 1
0.00
0.25
0.50
0.75
1.00
Pat
ient
Sur
viva
l
0 1 2 3 4 5Years
CNARAustralia
New Zealand
Survival from 90th Day of Treatment
Centre reports – graph 2
0.00
0.25
0.50
0.75
1.00
Tech
niqu
e S
urvi
val
0 1 2 3 4 5Years
CNARAustralia
New Zealand
Technique Survival - PD at 90 days
But....
0
20
40
60
80
100
Ag
e (o
f pre
vale
nt p
atie
nts)
Everywhere else CNARTS
Adjusted graphs
.7
.8
.9
1
1.1
1.2
1.3
Adj
uste
d S
MR
CNAR Australia New Zealand
Adjusted SMR (95% CI)
Adjusted graphs
CNAR
0
.5
1
1.5
2
2.5
SM
R
0 50 100 150 200Expected Number of Deaths
How are reports derived?
You need a model
• Logistic regression model (transplant), Poisson model (dialysis)
• Adjusted for demographics, comorbidities (donor and XM variables)
• With this model, derive a probability of “expected” failure for each person / graft based on covariate matrix
• Compare this with actual outcomes
www.anzdata.org.au
Which predictors are important?
Recip
ient
age
gen
der &
gra
ft num
ber
+com
orbi
ditie
s
+ HLA
mat
chin
g
+ isc
haem
ic tim
e
+ do
nor a
ge
+ ca
use
dono
r dea
th0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Harrell's CSomer's D
Predictive power of multivariate Cox model predicting graft survival, all DD transplants 2001-2009, with sequential addition of covariate groups
www.anzdata.org.au
• Factors within the control of centre– These may be why a particular centre gets
good or bad results• Factors that occur as a result of treatment
decisions• For example, don’t adjust for
– Choice of dialysis modality, HD access– Use of immunosuppressives, rejection, 1
month graft function…
Don’t adjust for…
Other graphical demonstrations of output
• Funnel plots are a static measure and summarise performance (relative to a comparator) over a fixed period of time.– Lack a dynamic element– Weight recent and distant results equally
Adding time – CUSUM
0
100
200
300
400
Num
ber
of tx
-4
-2
0
2
4
Cum
ulat
ive
sum
O-E
01jan2004
01jan2005
01jan2006
01jan2007
01jan2008
01jan2009
Tx date
Twoway CUSUM for a transplant centre
Removing credit for good deeds
0
1
2
3
4
5
Cu
mul
ativ
e s
um
O-E
0 100
200
300
400
Tx number
Oneway CUSUM for for a hospital
Do we need to do more?
Why KPIs?
• Mortality is an insensitive and late indicators of problems– Hopefully rare– Outcome of complex series of events
• Incompletely ascertained
– Important to monitor as best we can• Key Process indicators
– Simpler to understand, easier to address– Need to be valid and correctable (and related
to meaningful outcomes)
KPI Project
• Dialysis KPI project commenced 2011 – At instigation of DNT committee
• 2 markers chosen – Peritonitis and HD access at first treatment– Deliberately limited to existing data collection
• NO additional data collected
– Based on real time ANZDATA data collection
Variation in HD access
8
5 5
40
31
65
18
66
28
10
57
2522
113
17
3668
810
15
7
47
19
12
2952
75434452
29
11
34344144
12
4464
20
1017
3144331923
139
2815
54
6 8
0
.2
.4
.6
.8
1
Pro
por
tion
of li
nes
0 20 40 60Centres
Proportion 95% CI
ANZDATA, access at first HD where first dialysis
Variation in peritonitis rate
27
24
12
6
4
Pat
ient
-mon
ths
per
epi
sod
e
.5
1
1.5
2
2.5
3
Epi
sod
es p
er
pat
ient
-ye
ar
Confidence intervals not shown where upper limit >3Units with <5 person-years PD over 2009 not shown
2009 onlyPeritonitis rates by treating unit
KPI reporting -- access
• Quarterly identified feedback to units
Peritonitis reporting
Where to from here?
• COMMUNICATE• Improve data collection• Improve access to results• Enhance reporting
– Add peritonitis rates– Access subdivided by late referral– Graphs etc etc
• Or is it all just too hard?
How do we view quality?
Centre reports -- SMR