ahrq iqi clinical validation panels sheryl davies, ma stanford university
TRANSCRIPT
AHRQ IQI Clinical Validation AHRQ IQI Clinical Validation PanelsPanels
Sheryl Davies, MASheryl Davies, MA
Stanford UniversityStanford University
OutlineOutline
PurposePurpose Composition and RecruitmentComposition and Recruitment Rating ProcessRating Process Overall themesOverall themes Indicator resultsIndicator results Take home lessonsTake home lessons
Purpose of the Clinical PanelsPurpose of the Clinical Panels
IQIs and PQIsIQIs and PQIs– Developed 1999-2001Developed 1999-2001– Based on established indicatorsBased on established indicators– Did not undergo panel reviewDid not undergo panel review
Panel review establishes face validity of the Panel review establishes face validity of the indicatorsindicators
Standardize available evidence for all AHRQ Standardize available evidence for all AHRQ QIsQIs– Establish face validity for one stakeholder groupEstablish face validity for one stakeholder group– Update evidenceUpdate evidence
Panel review for IQIs considered for NQF Panel review for IQIs considered for NQF endorsementendorsement
MethodsMethods
Modified RAND/UCLA Appropriateness Method Modified RAND/UCLA Appropriateness Method (Nominal Group Technique)(Nominal Group Technique)
Physicians of various specialties/subspecialties Physicians of various specialties/subspecialties and other health professionals were recruited and other health professionals were recruited with the assistance of relevant organizationswith the assistance of relevant organizations
22 contacted organizations nominated 22 contacted organizations nominated – 103 clinicians nominated 103 clinicians nominated 60 accepted 60 accepted 45 45
eligible eligible
Panelists selected in order to form diverse panelsPanelists selected in order to form diverse panels
PanelistsPanelists
Female: 18%Female: 18% Academic: 71%Academic: 71% Geographic Geographic
– East: 44%East: 44%– West: 29%West: 29%– Other: 27%Other: 27%
Practice settingPractice setting– Urban: 67%Urban: 67%– Suburban: 42%Suburban: 42%– Rural: 18%Rural: 18%
Funding of primary hospital:Funding of primary hospital:– Private: 44%Private: 44%– Public: 27%Public: 27%
Underserved patient population: 56%Underserved patient population: 56%
Panel methods: RatingsPanel methods: Ratings
Initial ratingsInitial ratings– Packet of information summarizing evidencePacket of information summarizing evidence– Approx. 10 questionsApprox. 10 questions
Tailored to the indicator typeTailored to the indicator type 9 point scale9 point scale Overall usefulness for quality improvement, comparative Overall usefulness for quality improvement, comparative
reportingreporting– Compiled ratings provided to panelistsCompiled ratings provided to panelists
Conference callConference call– Discuss differencesDiscuss differences– Consensus on definition changesConsensus on definition changes
Final ratingsFinal ratings– Empirical analyses providedEmpirical analyses provided– Using same questionnaire as initial ratingsUsing same questionnaire as initial ratings
Results: Overarching themesResults: Overarching themes
Case mix variabilityCase mix variability ReliabilityReliability Volume measures as indirect measures Volume measures as indirect measures
of qualityof quality– Composite measuresComposite measures
Cardiac PanelCardiac Panel
Reviewed 5 indicators: Reviewed 5 indicators: – AAA Volume/MortalityAAA Volume/Mortality– Pediatric Heart Surgery Volume/MortalityPediatric Heart Surgery Volume/Mortality– Bilateral CatheterizationBilateral Catheterization
11 clinicians: vascular surgeons, 11 clinicians: vascular surgeons, pediatric cardiologists, pediatric pediatric cardiologists, pediatric cardiovascular surgeons, interventional cardiovascular surgeons, interventional cardiologists, pediatric ICU nurse, cardiologists, pediatric ICU nurse, surgical nursesurgical nurse
Cardiac PanelCardiac Panel
Indicator Overall – QI Overall - Comparative
AAA Mortality Acceptable (7) Unclear (6)
AAA Volume Acceptable (7) Acceptable (7)
Bilateral catheterization
Unclear (5) Unclear with disagreement (5)
Pediatric heart surgery volume
Acceptable (8) Acceptable (8)
Pediatric heart surgery mortality
Acceptable (8) Acceptable (8)
Cardiac PanelCardiac Panel
Indicator Overall – QI Overall - Comparative
AAA Mortality Acceptable (7) Unclear (6)
AAA Volume Acceptable (7) Acceptable (7)
• Case mix variability
• Ruptured vs. unruptured; endovascular vs. open
• Bias: Slight overadjustment for endovascular (12%) and underadjustment for ruptured (12%)
• Total volume (ruptured and unruptured) best predictor of outcomes
• Stratify by surgical approach (endovascular vs. open)
Cardiac PanelCardiac Panel
Indicator Overall – QI Overall - Comparative
AAA Mortality Acceptable (7) Unclear (6)
AAA Volume Acceptable (7) Acceptable (7)
• Case mix variability
• Ruptured vs. unruptured; endovascular vs. open
• Bias: Slight overadjustment for endovascular (12%) and underadjustment for ruptured (12%)
• Total volume best predictor of outcomes
• Stratify by surgical approach (endovascular vs. open)
Cardiac PanelCardiac Panel
Indicator Overall – QI Overall - Comparative
Pediatric heart surgery volume
Acceptable (8) Acceptable (8)
Pediatric heart surgery mortality
Acceptable (8) Acceptable (8)
• Case mix variability
• Supported use of RACHS
• Correlations of hospital volume for each RACHS complexity are robust (r = 0.74 – 0.95)
• Best predictor of outcome is total volume, rather than by complexity
Cardiac PanelCardiac Panel
Indicator Overall – QI Overall - Comparative
Bilateral catheterization
Unclear (5) Unclear w/ disagreement (5)
• Modification: Expand list of appropriate indications for bilateral catheterization
• Primarily a resource indicator
• Charting of indications may be poor
• May result in decrease of appropriate uses
Surgical Resection PanelSurgical Resection Panel
Reviewed 4 indicators:Reviewed 4 indicators:– Esophageal Resection Volume/MortalityEsophageal Resection Volume/Mortality– Pancreatic Resection Volume/MortalityPancreatic Resection Volume/Mortality
13 clinicians: thoracic surgeons, general 13 clinicians: thoracic surgeons, general surgeons (including GI and oncology), surgeons (including GI and oncology), oncologists, internist, oncologists, internist, gastroenterologists, surgical nursegastroenterologists, surgical nurse
Surgical Resection PanelSurgical Resection Panel
Indicator Overall – QI Overall - Comparative
Esophageal resection mortality Acceptable (7) Acceptable (7)
Esophageal resection volume Acceptable (7) Acceptable (7)
Pancreatic resection mortality Acceptable (7) Acceptable (7)
Pancreatic resection volume Acceptable (7) Acceptable (7)
Surgical Resection PanelSurgical Resection Panel
Esophageal resection • Case mix variability
• Risk adjustment performs well. Underestimates risk for patient with middle esophageal and unspecified site cancer (22%, 37%).
Pancreatic resection• Case mix variability
• Over-estimates risk for total pancreatectomy (68%), lesser extent underestimates risk for Whipple (15%). Issue raised with 3M.
• Low rates
Geriatric PanelGeriatric Panel
Reviewed 4 indicators:Reviewed 4 indicators:– Acute Stroke MortalityAcute Stroke Mortality– Hip Fracture MortalityHip Fracture Mortality– Hip Replacement MortalityHip Replacement Mortality– Incidental AppendectomyIncidental Appendectomy
14 clinicians: internists (including geriatrics 14 clinicians: internists (including geriatrics and hospital medicine), neurologists, general and hospital medicine), neurologists, general surgeon, interventional radiologist, orthopedic surgeon, interventional radiologist, orthopedic surgeons, neurosurgeon, diagnostic surgeons, neurosurgeon, diagnostic radiologist, nurse, physical therapistradiologist, nurse, physical therapist
Geriatric PanelGeriatric Panel
Indicator Overall – QI Overall - Comparative
Acute stroke mortality
Unclear (6.5) Unclear with disagreement (5)
Incidental appendectomy
Acceptable (7) Unclear (6)
Hip fracture mortality Acceptable (7) Acceptable (7)
Hip replacement mortality
Unclear due to disagreement (7)
Unclear (6)
Geriatric PanelGeriatric Panel
Indicator Overall – QI Overall - Comparative
Acute stroke mortality
Unclear (6.5) Unclear with disagreement (5)
Incidental appendectomy
Acceptable (7) Unclear (6)
Hip fracture mortality Acceptable (7) Acceptable (7)
Hip replacement mortality
Unclear due to disagreement (7)
Unclear (6)
• Case mix variability: Stroke type (hemorrhagic, ischemic, subarachnoid)
• Risk adjustment accounts for almost all difference in risk
• Patient factors such as delay in presenting for care
Geriatric PanelGeriatric Panel
Indicator Overall – QI Overall - Comparative
Acute stroke mortality
Unclear (6.5) Unclear with disagreement (5)
Incidental appendectomy
Acceptable (7) Unclear (6)
Hip fracture mortality Acceptable (7) Acceptable (7)
Hip replacement mortality
Unclear due to disagreement (7)
Unclear (6)
• Exclude patients with hip fracture
• Case mix variability
• Risk adjustment somewhat overestimates risk for revision
• Rates very low, reliability concerns
Geriatric PanelGeriatric Panel
Indicator Overall – QI Overall - Comparative
Acute stroke mortality
Unclear (6.5) Unclear with disagreement (5)
Incidental appendectomy
Acceptable (7) Unclear (6)
Hip fracture mortality Acceptable (7) Acceptable (7)
Hip replacement mortality
Unclear due to disagreement (7)
Unclear (6)
• “Is it still being done?”
• “If it is still being done, it shouldn’t be done. Then it is a good indicator”
• “I am having a hard time getting excited about this indicator”
Geriatric PanelGeriatric Panel
Indicator Overall – QI Overall - Comparative
Acute stroke mortality
Unclear (6.5) Unclear with disagreement (5)
Incidental appendectomy
Acceptable (7) Unclear (6)
Hip fracture mortality Acceptable (7) Acceptable (7)
Hip replacement mortality
Unclear due to disagreement (7)
Unclear (6)
• Limit to the elderly
• Case mix variability
• Risk adjustment accounts for both repair type and fracture location
ConclusionConclusion
Overall good reception of indicatorsOverall good reception of indicators Recommendations considered in context Recommendations considered in context
of other validation effortsof other validation efforts Indicator revisions implemented in Indicator revisions implemented in
February 2008February 2008 Further efforts to improve indicators or Further efforts to improve indicators or
develop additional evidence develop additional evidence
Questions?Questions?