Methodological
Issues
Presentation by Ian Brownwood for the
meeting of Health Promotion, Prevention
and Primary Care Subgroup 22 October,
2009, Paris
The Indicators
• Asthma*
• Chronic Obstructive Pulmonary Disease*
• Congestive Heart Failure*
• Angina
• Hypertension*
• Diabetes
– Short -term complications*
– Long -term complications
– Lower -extremity amputation*
– Uncontrolled
* To be published in the 2009 edition of OECD Health at a Glance
Their Nature
• Based on data from routine hospitaladministrative information systems.
• Conditions are considered amenable orsensitive to care provided in primary care.
• Lower values are intended to be reflectiveof effective primary care systems.
• Does not imply that indicator values ofzero can be achieved.
Their Definition
• AHRQ definitions except:
– Aged 15+ instead of 18+ years
– Exclusion of day cases.
– Mapping between ICD9 and ICD10.
– Procedure coding
• Based on PDX codes, except for:
– Asthma (i.e. cystic fibrosis and anomalies of the respiratory system)
– Lower-extremity amputation (i.e. trauma)
– COPD (i.e. qualification of bronchitis)
Validity and Comparability
Key issues considered:
1. Potential confounding factors
2. Data coverage of the hospital sector
3. Reporting of meaningful composites
4. Indicator instability in small populations
5. Indicator-specific issues:
• Technical specification of indicators
• Cross walk of ICD 9 and ICD 10 codes
• Country variations in coding practice/capacity
Confounding Factors
• Potential confounding factors
– Demand Prevalence
– Supply Hospital Expenditures, Staff, Beds, Bed days and Admissions
• Potential explanatory factors
– GP per 100,000
• Preliminary examination during HCQI data collection for 2008-09.
WE DID FIND…..
Demand
Demand
Supply
Supply
Explanatory
Explanatory
BUT THEN WE ALSO FOUND…
Demand
Supply
Explanatory
Conclusions
• Some evidence of relations but not strong or consistent.
• Data limitations:
– Invalidated country reported prevalence rates.
– Small sample size
– Temporal issues for diabetes long-term complications.
• Some of the key findings reflected in 2009 edition of Health at a Glance.
Key Issues
1. Potential confounding factors
2. Data coverage of the hospital sector
3. Reporting of meaningful composites
4. Indicator instability in small populations
5. Indicator-specific issues:
• Technical specification of indicators
• Cross walk of ICD 9 and ICD 10 codes
• Country variations in coding practice/capacity
Hospital Coverage
• Denominator =Population not Admissions (i.e. not same unit of measurement).
• Sensitive to coverage of hospital system.
• Not all countries provided the additional data (e.g. UK could be 10% understated).
• Adjustment of rates may result in slight overestimation in some cases (i.e. 40% of countries indicated they thought the public rate would be higher).
Adjusted Rates
Conclusions
• Without adjustment potential exists for underestimation of rates in some countries.
• Incomplete additional data for 2009-10.
• Unadjusted rates reported in 2009 edition of Health at a Glance.
• Reconsider application of adjusted rates based on enhanced data in 2010-11.
Key Issues
1. Potential confounding factors
2. Data coverage of the hospital sector
3. Reporting of meaningful composites
4. Indicator instability in small populations
5. Indicator-specific issues:
• Technical specification of indicators
• Cross walk of ICD 9 and ICD 10 codes
• Country variations in coding practice/capacity
Composite
What is the objective?
1. To explain the variation in the individual indicators through the use of one or more composite indicators?
2. To indicate the performance of the whole system through the aggregation of indicators that relate to parts of the system?
Some countries currently report composite indicators (e.g. Canada, Australia).
AHRQ Findings
• Indicators are positively correlated
• Positive factor loadings on the 1st factor
• 1st factor explains 94% of variation
• Indication for separate diabetes composite (high factor loading on the 2nd factor)
HCQI Findings
• Not all indicators are positively correlated (e.g. angina and diabetic amputations) ×
• Factor loading for amputation is negative ×
• 1st factor only explains 31% of variation ×
• No clear indication for a 2nd factor ×
Preliminary HCQI analysis does not provide clear support for use of composite.
Correlation
| Asthma COPD CHF Angina Hyper Diab ST Diab LT Diab UC DiabLEA
-------------+------------------------------------------------------------------------------------
Asthma | 1.0000
COPD | 0.1812 1.0000
CHF | 0.1559 0.0152 1.0000
Angina | -0.1658 0.4965 0.1733 1.0000
Hypertension | 0.0627 0.2765 0.4532 0.3830 1.0000
Diabetes ST | 0.5346 0.2123 0.2886 -0.2476 -0.0903 1.0000
Diabets LT | 0.1367 0.1675 0.0892 -0.0779 0.5676 0.1674 1.0000
Diabetes UC | 0.1403 0.2929 0.1552 0.2508 0.8065 0.0226 0.5757 1.0000
Diabetes LEA | 0.0692 -0.2891 0.3810 -0.3284 -0.1404 0.2650 0.2201 -0.3302 1.0000
Rotated factor loadings (pattern matrix) and unique variances
-------------------------------------------------
Variable | Factor1 Factor2 | Uniqueness
-------------+--------------------+--------------
Asthma | 0.1936 0.5950 | 0.6085
COPD | 0.5645 -0.1366 | 0.6627
CHF | 0.3596 0.4773 | 0.6429
Angina | 0.5136 -0.5182 | 0.4677
Hypertension | 0.9054 0.0088 | 0.1802
Diabetes ST | 0.0829 0.7547 | 0.4235
Diabetes LT | 0.6062 0.3689 | 0.4965
Diabetes UC | 0.8721 -0.0298 | 0.2385
Diabetes LEA | -0.2611 0.6924 | 0.4524
-------------------------------------------------
Proportion 0.3080 0.2284
Key Issues
1. Potential confounding factors
2. Data coverage of the hospital sector
3. Reporting of meaningful composites
4. Indicator instability in small populations
5. Indicator-specific issues:
• Technical specification of indicators
• Cross walk of ICD 9 and ICD 10 codes
• Country variations in coding practice/capacity
Indicator Stability
• Inherent instability in annual indicator values (high random error) associated with low numerator values indicates use of multi- year averaging or aggregation of numerator cases.
• Particular issue for smaller states (e.g. Iceland and Luxembourg).
• Use of the 3-year rolling average would facilitate comparison with other countries.
Key Issues
1. Potential confounding factors
2. Data coverage of the hospital sector
3. Reporting of meaningful composites
4. Indicator instability in small populations
5. Indicator-specific issues:
• Technical specification of indicators
• Cross walk of ICD 9 and ICD 10 codes
• Country variations in coding practice/capacity
Asthma
• Differential diagnosis of asthma/COPD.
• Code coverage issue between ICD 9 and ICD 10:
– Coding guidelines indicate that ICD 9 4932 equatesto ICD 10 J448.
– Supplementary data indicates that chronicobstructive asthma (4932) accounts for 10-50% ofasthma numerator cases (n=4).
– Plan to remove J448 from COPD and add to Asthmacodes to align with ICD 9 code coverage for futuredata collections.
Asthma and COPD
Asthma
COPD
(PulmonaryEmphysema)
COPD
(ChronicBronchitis)
Chronic Obstructive
Asthma (ICD 9 4932)Other Specified
COPD (ICD 10 J448)
COPD
• Alternative calculation suggested by Denmark:
– Aged 30+ years with J44 as PDX, and
– J44 as SDX and one of the following PDX:
• J96 (Respiratory Failure not elsewhere classified)
• J13-18 (Pneumonia)
– Reflect common diagnosis of patients with COPD (i.e. PDX = Pneumonia).
Potential Issues
• Use of SDX is potentially problematic.
• Create inconsistency in the numerator and denominator across indicators impact on ability to aggregate indicators.
Angina
• Calculation based on principal diagnosis.
• Supplementary data was collected during HCQI data collection for 2008-09 to explore variations in coding practices:
– Cases identified in SDX (any PDX).
– Cases identified in SDX (PDX of chest pain).
PDX and SDX
Explanation
• Secondary diagnosis coding
– Limited data availability for this analysis:
• Italy had the lowest number of SDX codes and highest proportion of PDX cases.
• However, this relationship did not hold for other countries.
• Other potential factors at play?
– E.g. Saver et al. (2009) attribute a sharp decline in angina admissions in the 1990’s to more aggressive diagnosis of coronary atherosclerosis.
Secondary Diagnosis
Saver et al. in Medical Care 2009:47, pp.1106-1110
Discharge Trends
Saver et al. in Medical Care 2009:47, pp.1106-1110
Angina DX on Admission and
Discharge DX
Hypertension
• Benign hypertension included in ICD 10 code list but not ICD 9 code list (i.e. 4011)
• Questions over validity clinical grounds for PDX (The Netherlands, NZ).
Lower Extremity Amputation
• Inclusion of toe amputation
– Interpretation in relation to quality of care
– Performed in hospital and ambulatory service settings and on a day case and inpatient basis.
• Use of secondary diagnoses
– Up to 50% cases identified in SDX codes
– Availability of secondary diagnosis codes?
• Trauma code exclusions
– Adequacy of code list
Toe Amputation
• Reflects tertiary prevention
– Can reflect good quality of care as it prevents further complications (OECD 2004)
– Failed primary and secondary prevention in primary care system retain in code set?
• Only multi-day stays reflected in data.
– If ambulatory care = day cases retain in code set?
Secondary Diagnosis
Secondary Diagnosis
Consider further data collection to locate where cases are identified in SDX codes, similar to PSI in 2008-09.
Options:
1. Limit calculation to:
– countries able to report above a lower threshold number of SDX codes , and/or
– an upper threshold number of SDX codes
2. Apply adjustment methodology, similar to PSI
3. Limit the calculation to PDX only.
Trauma Exclusions
• Currently includes traumatic amputation of toe, foot and leg (ICD 9 codes 895-897)
• There may be justification for extending the exclusion code list.
– The Netherlands suggest inclusion of:
• Fracture of specified bones of lower limbs (820 to 828)
• Late effect of muscle & skin injury (905,906)
• Crushing injury of lower limb & other sites (928, 929)
• Burns of lower limbs & other sites (945, 946)
Uncontrolled Diabetes
• Potentially problematic for ICD 10 countries.
• 60% (11) countries reporting data in 2008-09 could identify uncontrolled cases:
– Just<50% by using ICD 9
– Just >50% by using ICD 10 markers.
• Austria, Canada, Finland, Korea, UK, Norway, Poland and Sweden could not identify uncontrolled cases.
• Canada establishing a marker for 2009+
Suggested Approach to Key Issues
1. Continue to examine potential confounding and explanatory factors and report key findings with the indicators in OECD publications.
2. Collect further data on hospital coverage from all countries in 2010-11 and review approach.
3. Further analysis required before proceeding with the use of composite indicators.
4. Present 3-year averages for Iceland and Luxembourg in the future to address indicator instability.
5. Consider indicator-specific issues on a case by case basis next slide.
Indicator-Specific Issues
Indicator Suggested Approach
Asthma Revise ICD 10 code list (add J448)
Chronic Obstructive Pulmonary Disease For discussion
Congestive Heart Failure No action
Angina* For discussion
Hypertension Revise ICD 9 code list (add 4011)
Diabetes : Short –term complications No action
Diabetes: Long –term complications* For discussion
Diabetes: Lower -extremity amputation For discussion
Diabetes: Uncontrolled* For discussion
* Not currently considered for publication in 2009.
The Subgroup is invited to:
–comment on these methodological issues
–endorse plans for further analysis and refinement of the indicators in preparation for the next round of data collection in 2010-11.