term 4, 2006bio656--multilevel models 1 part 07 evaluating hospital performance
TRANSCRIPT
![Page 1: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/1.jpg)
BIO656--Multilevel Models 1Term 4, 2006
PART 07PART 07
Evaluating Hospital PerformanceEvaluating Hospital Performance
![Page 2: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/2.jpg)
BIO656--Multilevel Models 2Term 4, 2006
PERFORMANCE MEASURESPERFORMANCE MEASURES
Patient outcomes • Mortality, morbidity, satisfaction with care
– 30-day mortality among heart attack patients (Normand et al JAMA 1996, JASA 1997)
Process • Medication & test administration, costs
– Laboratory costs for diabetic patients– Number of physician visits
• Hofer et al JAMA, 1999– Palmer et al. (1996)
![Page 3: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/3.jpg)
BIO656--Multilevel Models 3Term 4, 2006
DATA STRUCTUREDATA STRUCTURE
Multi-level• Patients nested in physicians, hospitals, HMOs, ...
• Providers clustered by health care systems, market areas, geographic areas
• Covariates at different levels of aggregation: – patient, physician, hospital, ...
Variation in variability• Statistical stability varies over physicians, hospitals, ..
![Page 4: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/4.jpg)
BIO656--Multilevel Models 4Term 4, 2006
MLMs are EffectiveMLMs are Effective
Correlation at many levels• Hospital practices may induce a strong correlation
among patient outcomes within hospitals even after accounting for patient characteristics
Structuring estimation• Stabilizing noisy estimates • Balancing SEs• Estimating ranks and other non-standard summaries
![Page 5: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/5.jpg)
BIO656--Multilevel Models 5Term 4, 2006
The Cooperative The Cooperative Cardiovascular Project (CCP)Cardiovascular Project (CCP)
• Abstracted medical records for patients discharged from hospitals located in Alabama, Connecticut, Iowa, and Wisconsin (June 1992May 1993)
• 3,269 patients hospitalized in 122 hospitals in four US States for Acute Myocardial Infarction
![Page 6: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/6.jpg)
BIO656--Multilevel Models 6Term 4, 2006
GOALSGOALS
• Identify “aberrant” hospitals with respect to several performance measures
• Report the statistical uncertainty associated with ranking of the “worst hospitals”
• Investigate if hospital characteristics explain variation in hospital-specific mortality rates
![Page 7: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/7.jpg)
BIO656--Multilevel Models 7Term 4, 2006
DATADATA
Outcome• Mortality within 30-days of hospital admission
Patient characteristics• Admission severity index constructed on the basis of
34 patient attributes
Hospital characteristics• Urban/Rural• (Non academic)/(versus academic)• Number of beds
![Page 8: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/8.jpg)
BIO656--Multilevel Models 8Term 4, 2006
Why adjust for case mix?Why adjust for case mix?(patient characteristics)
• Irrespective of quality of care, older/sicker patients with multiple diseases have increased need of health care services and poorer health outcomes
• Without adjustment, physicians/hospitals who treat relatively more of these patients will appear to provide more expensive and lower quality care than those who see relatively younger/healthier patients
If there is inadequate case mix adjustment, evaluations will be unfair• But, need to avoid over adjusting
![Page 9: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/9.jpg)
BIO656--Multilevel Models 9Term 4, 2006
Case-mix AdjustmentCase-mix Adjustment
Compute hospital-specific, expected mortality by: 1. estimating a patient-level mortality model using all hospitals 2. averaging the model-produced probabilities for all
patients within a hospital
• Hospitals with “higher-than-expected” mortality rates can be flagged as institutions with potential quality problems, but need to account for uncertainty
• Need to be careful, if also adjusting for hospital characteristics– May adjust away the important signal
![Page 10: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/10.jpg)
BIO656--Multilevel Models 10Term 4, 2006
(as we know, very poor approach)
• Wrong SEs• Test-based
![Page 11: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/11.jpg)
BIO656--Multilevel Models 11Term 4, 2006
Hospital Profiling of Mortality Rates Hospital Profiling of Mortality Rates Acute Myocardial Infarction Patients Acute Myocardial Infarction Patients
(Normand et al. JAMA 1996, JASA 1997)
![Page 12: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/12.jpg)
BIO656--Multilevel Models 12Term 4, 2006
Hierarchical logistic regressionHierarchical logistic regression
I: Patient within-provider• Patient-level logistic regression model with random
intercept & slope
II: Between-provider• Hospital-specific random effects are regressed on
hospital-specific characteristics– Explicit regression
![Page 13: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/13.jpg)
BIO656--Multilevel Models 13Term 4, 2006
Admission severity indexAdmission severity index(Normand et al. 1997 JASA)
![Page 14: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/14.jpg)
BIO656--Multilevel Models 14Term 4, 2006
0 + 1(sevij – sevbar)
0
1
sevbar
![Page 15: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/15.jpg)
BIO656--Multilevel Models 15Term 4, 2006
we use bwe use b0i0i + b + b1i1i(...)(...)
![Page 16: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/16.jpg)
BIO656--Multilevel Models 16Term 4, 2006
Interpretation of parameters is different for the two levels
bb0i0i = = **0000 + N(..), etc. + N(..), etc.
![Page 17: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/17.jpg)
BIO656--Multilevel Models 17Term 4, 2006
RESULTSRESULTS
• Estimates of regression coefficients under three models:– Random intercept only– Random intercept and random slope– Random intercept, random slope, and hospital
covariates• Hospital performance measures
![Page 18: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/18.jpg)
BIO656--Multilevel Models 18Term 4, 2006
Normand et al. JASA 1997Normand et al. JASA 1997
![Page 19: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/19.jpg)
BIO656--Multilevel Models 19Term 4, 2006
30-DAY MORTALITY30-DAY MORTALITY 2.5th and 97.5th percentiles for a
patient of average admission severity
Exchangeable model • Random intercept and slope, no hospital covariates
log(odds): (-1.87,-1.56) probability,scale: (0.13, 0.17)
Covariate (non-exchangeable) model • Random intercept and slope, with hospital covariates• Patient treated in a large, urban academic hospital
log(odds): (-2.15,-1.45)probability scale: (0.10,0.19)
![Page 20: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/20.jpg)
BIO656--Multilevel Models 20Term 4, 2006
Effect of hospital characteristics Effect of hospital characteristics on baseline log-odds of 30-day mortalityon baseline log-odds of 30-day mortality
• For an average patient, rural hospitals have a higher odds ratio than urban hospitals
– Indicates between-hospital differences in the baseline mortality rates
– Case-mix adjustment may be able to remove some of this difference
![Page 21: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/21.jpg)
BIO656--Multilevel Models 21Term 4, 2006
Estimates of Stage-II Estimates of Stage-II regression coefficientsregression coefficients
InterceptsIntercepts
![Page 22: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/22.jpg)
BIO656--Multilevel Models 22Term 4, 2006
Effect of hospital characteristics on Effect of hospital characteristics on association between severity and mortalityassociation between severity and mortality
(slopes)
• The association between severity and mortality is modified by hospital size
• Medium-sized hospitals have smaller severity/mortality associations than large hospitals
– Indicates that the effect of clinical burden (patient severity) on mortality differs across hospitals
![Page 23: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/23.jpg)
BIO656--Multilevel Models 23Term 4, 2006
Estimates of Stage IIEstimates of Stage IIregression coefficientsregression coefficients
SlopesSlopes
![Page 24: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/24.jpg)
BIO656--Multilevel Models 24Term 4, 2006
Homework is on front tableHomework is on front table
![Page 25: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/25.jpg)
BIO656--Multilevel Models 25Term 4, 2006
Observed and risk-adjusted hospital mortality ratesObserved and risk-adjusted hospital mortality ratesUrban Hospitals
Histogram displays (observed – adjusted)
![Page 26: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/26.jpg)
BIO656--Multilevel Models 26Term 4, 2006
Observed and risk-adjusted hospital mortality ratesObserved and risk-adjusted hospital mortality ratesRural Hospitals
Histogram displays (observed – adjusted)
Substantial adjustment for severitySubstantial adjustment for severity
![Page 27: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/27.jpg)
BIO656--Multilevel Models 27Term 4, 2006
FINDINGSFINDINGS
• There is substantial adjustment for admission severity
• Generally, urban hospitals are adjusted less than rural
• There is less variability in observed or adjusted estimated rates for urban hospitals than for rural hospitals
Can you explain why?
![Page 28: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/28.jpg)
BIO656--Multilevel Models 28Term 4, 2006
Normand et al. JASA 1997Normand et al. JASA 1997
![Page 29: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/29.jpg)
BIO656--Multilevel Models 29Term 4, 2006
Average the probabilitiesAverage the probabilitiesDon’t average the covariatesDon’t average the covariates
![Page 30: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/30.jpg)
BIO656--Multilevel Models 30Term 4, 2006
k denotes a draw from the posterior
![Page 31: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/31.jpg)
BIO656--Multilevel Models 31Term 4, 2006
Plug in the average covariatePlug in the average covariate
Keep the hospital variationKeep the hospital variation
![Page 32: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/32.jpg)
BIO656--Multilevel Models 32Term 4, 2006
![Page 33: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/33.jpg)
BIO656--Multilevel Models 33Term 4, 2006
Comparing measures Comparing measures of hospital performanceof hospital performance
Three measures of hospital performance• Probability of a large difference between adjusted
and standardized mortality rates
• Probability of excess mortality for the average patient
• Z-score
![Page 34: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/34.jpg)
BIO656--Multilevel Models 34Term 4, 2006
Hospital Rankings: Normand et al 1997 JASA
![Page 35: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/35.jpg)
BIO656--Multilevel Models 35Term 4, 2006
Hospital RanksHospital Ranks
• There was moderate disagreement among the criteria for classifying hospitals as “aberrant”
• Nevertheless, hospital 1 is ranked worst
• It is rural, medium sized non-academic with an observed mortality rate of 35%, and adjusted rate of 28%
![Page 36: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/36.jpg)
BIO656--Multilevel Models 36Term 4, 2006
Adjusting for hospital-level charateristics Adjusting for hospital-level charateristics
Changes the comparison group in “as compared to what?”
• All hospitals (unadjusted at hospital level)
• Hospitals of a similar size, urbanicity, ...
• Percent of physicians who are board certified
• Hospitals with a similar death rate
Variance reduction and goodness of fit should not be the primary considerations
• “As compared to what?” must dominate
![Page 37: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/37.jpg)
BIO656--Multilevel Models 37Term 4, 2006
DiscussionDiscussion
• Profiling medical providers is multi-faced and data intensive process with substantial implications for health care practice, management, and policy
• Major issues include data quality and availability, choice of performance measures, formulation of statistical models (including adjustments), reporting results
• The ranking approaches and summaries used by Normand and colleagues are very good, but some improvement is possible
![Page 38: Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697bfb81a28abf838c9f5c3/html5/thumbnails/38.jpg)
BIO656--Multilevel Models 38Term 4, 2006
Multi-level models address key technical & Multi-level models address key technical & conceptual profiling issues, includingconceptual profiling issues, including
• Adjusting for patient severity
• Accounting for within-provider correlations
• Accounting for differential sample sizes at all levels
• Stabilize estimates
• Structure ranking and other, derived comparisons