performance reports andy bindman md department of medicine, epidemiology and biostatistics ucsf
Post on 11-Jan-2016
212 Views
Preview:
TRANSCRIPT
Performance Reports
Andy Bindman MD
Department of Medicine, Epidemiology and Biostatistics
UCSF
Calculating an Individual’s Risk
Solve the multivariate model incorporating an individual’s specific characteristics
For continuous outcomes the predicted values are the expected values
For dichotomous outcomes the sum of the derived predictor variables produces a “logit” which can be algebraically converted to a probability (enat log odds /1 + enat log odds )
Aggregating to the group level Sum observed events (eg deaths) for sub-group
Sum expected probability of events for same
Probabilities used to calculate expected events derived from entire data set and applied to individuals in sub-group (eg defined by care site)
The overall expected number of events must equal the observed number of events but this need not be the case at the level of subgroups
Comparing observed and expected outcomes to assess quality
Observed events or rates of events Expected events or rates of events Better quality implied when observed is lower
than expected (worse quality when observed higher than expected)
Risk adjusted rates
Standardizes rates across sub-groups so that they can be directly compared with a single number
Observed rate/expected rate of subgroup
x overall observed rate
Observed CABG Mortality Rates, NY 1989-1992
1989 1990 1991 1992
Volume 12,269 13,946 14,944 16,028
Deaths 432 437 460 446
Observed Mortality,%
3.52 3.14 3.08 2.78
Calculating Expected CABG Mortality Rates in New York by Year
Pool all 4 years of CABG patients Develop risk adjustment model for CABG patients Apply risk adjustment model for CABG patients
sub-grouped by year to determine expected number of deaths for each year
Divide expected number of deaths by number of cases per year to get expected death rate
Observed and Expected CABG Mortality Rates, NY 1989-1992
1989 1990 1991 1992
Volume 12,269 13,946 14,944 16,028
Observed Mortality,%
3.52 3.14 3.08 2.78
Expected Mortality,%
2.62 2.97 3.16 3.54
Annual Risk Adjusted Mortality Rate for CABG in New York
Observed rate per year/expected rate per year
X average death rate over 4 year period (3.1)
Observed, Expected and Risk-Adjusted CABG Mortality Rates, NY 1989-1992
1989 1990 1991 1992
Volume 12,269 13,946 14,944 16,028
Observed Mortality,%
3.52 3.14 3.08 2.78
Expected Mortality,%
2.62 2.97 3.16 3.54
Risk adj Mortality,%
4.17 3.28 3.03 2.45
What Happened with CABG Surgery Over Time in New York?
Operated on sicker patients
Observed mortality rate declined over time
Risk adjusted mortality rate declined even more
Did quality of CABG care improve over time?
NY CABG Risk Adjustment Model
Well designed model
C index = .787; Hosmer-Lemeshow chi square p=.16
Mortality is not a subjective outcome- hard to fake
Gaming might be possible with coding some predictors
Interpreting Risk Adjusted CABG Outcomes
Public reporting on hospital CABG mortality began in 1989
Low volume hospitals had higher mortality rates and some stopped performing CABG over time
Process indicators of cardiac care (beta blocker post MI) also improved in NY hospitals over time
Hospitals documented more co-morbidities over time resulting in inflated expected death rates
Some sick NY cardiac patients operated on in NJ
Applying Results To Providers
Possible to aggregate observed and expected rates of events to hospital, physician, or some other provider level grouping
Statistical problems arise when total number of expected events are small
Minimum of five expected events per group as a rule of thumb
Naming Names
Assigning assessments of quality to specific providers increases the stakes
Need to demonstrate validity of analytic approach
Reporting Results
Public reporting vs internal quality improvement
Data users tend to want gradations of quality along a continuum (excellent to poor)
However denoting those within a 95% confidence interval of the expected as average is less sensitive to noise in data
Bootstrap Procedure:Deriving Confidence Intervals
Multiple (e.g. 1000) random samples of same size of original derived from original sample with replacement
Calculate expected rate for each “new” sample
Create frequency distribution of expected rates
Empirically derive 95% CI (950 of 1000 centered around mean)
Consistency in the evidence Differences between observed and expected may be due
to things other than ‘quality’
Are the results consistent over time
Are results consistent with prior expectations such as volume-outcome relationships
Confirmation through very different types of evidence is a major goal- “external validation”
Observed / Expected Mortality
Volume-Outcome
Relationship between high volume providers and better outcomes
Most often studied in relationship to procedures
Consistent with notion that practice makes perfect
Hospital Volume and CABG Mortality in California Hospitals Using Registry, 2000-02
Volume Hospitals
N (%)
Patients
N (%)
OR
(CI)
>=600 6 (7) 16,145 (28) 0.56
(0.40-0.79)
300-599 16 (19) 17,052 (30) 0.80
(0.63-1.02)
200-299 14 (17) 8,168 (14) 0.74
(0.57-0.97)
<200 47 (57) 16,022 (28) referent
External validation of data Link hospital discharge data with CABG registry data Looking for missing cases, deaths and highly predictive risk
factors– 221 cases in discharge data not reported to registry– 26 additional deaths (498 total)– 63 undercodes and 123 overcodes of cardiogenic shock – 29 overcodes of salvage (51 total)
Direct auditing – Deaths– Highly weighted predictors particularly if subjective
Growing Number of Quality Initiatives Provide Opportunity for Cross Comparisons
AHRQ Quality Indicators JCAHO ORYX Hospital Core Performance CMS Hospital Quality Alliance National Quality Forum (NQF) Leapfrog NCQA HEDIS
JCAHO - Hospital AMI indicators
Process elements:
Aspirin at arrival, Aspirin at discharge, ACEI for LVSD, Smoking cessation counseling, Beta-blocker at discharge, Beta-blocker at arrival, Thrombolytic within 30 minutes, PCI within 120 minutes
Who uses these reports and how Patients
– Slow to catch on
– More important for those without other ways to judge quality Managers
– Aim to improve quality to avoid “naming and shaming” Payers (health plans)
– Selective contracting
– Pay for performance
How much do these reports matter?
California has lowered isolated CABG mortality by ~1% (from 3% to 2 %) during public report period
Approximately 20,000 procedures per year Reduction from 600 to 400 deaths Average survival ~5 years Even if half the change is due to gaming, 500 life
years saved
My Reflections on Performance Reports
View the risk adjusted estimates as ‘yellow flags’, not ‘smoking guns’
Risk models will probably improve with ability to add more clinical data available through electronic records
Research may not need to be perfect to bring about public health benefits
Attempts to improve quality need to consider unintended consequences on access/efficiency
Outcomes Research Opportunities
Validate risk adjustment models for new conditions
Are health care outcomes changing over time and if so why?
How can performance reports on health outcomes be used to create better health care quality?
top related