methodological issues - oecd · respiratory system) –lower-extremity amputation (i.e. trauma)...

Methodological

Issues

Presentation by Ian Brownwood for the

meeting of Health Promotion, Prevention

and Primary Care Subgroup 22 October,

2009, Paris

The Indicators

• Asthma*

• Chronic Obstructive Pulmonary Disease*

• Congestive Heart Failure*

• Angina

• Hypertension*

• Diabetes

– Short -term complications*

– Long -term complications

– Lower -extremity amputation*

– Uncontrolled

* To be published in the 2009 edition of OECD Health at a Glance

Their Nature

• Based on data from routine hospitaladministrative information systems.

• Conditions are considered amenable orsensitive to care provided in primary care.

• Lower values are intended to be reflectiveof effective primary care systems.

• Does not imply that indicator values ofzero can be achieved.

Their Definition

• AHRQ definitions except:

– Aged 15+ instead of 18+ years

– Exclusion of day cases.

– Mapping between ICD9 and ICD10.

– Procedure coding

• Based on PDX codes, except for:

– Asthma (i.e. cystic fibrosis and anomalies of the respiratory system)

– Lower-extremity amputation (i.e. trauma)

– COPD (i.e. qualification of bronchitis)

Validity and Comparability

Key issues considered:

1. Potential confounding factors

2. Data coverage of the hospital sector

3. Reporting of meaningful composites

4. Indicator instability in small populations

5. Indicator-specific issues:

• Technical specification of indicators

• Cross walk of ICD 9 and ICD 10 codes

• Country variations in coding practice/capacity

Confounding Factors

• Potential confounding factors

– Demand Prevalence

– Supply Hospital Expenditures, Staff, Beds, Bed days and Admissions

• Potential explanatory factors

– GP per 100,000

• Preliminary examination during HCQI data collection for 2008-09.

WE DID FIND…..

Demand

Demand

Supply

Supply

Explanatory

Explanatory

BUT THEN WE ALSO FOUND…

Demand

Supply

Explanatory

Conclusions

• Some evidence of relations but not strong or consistent.

• Data limitations:

– Invalidated country reported prevalence rates.

– Small sample size

– Temporal issues for diabetes long-term complications.

• Some of the key findings reflected in 2009 edition of Health at a Glance.

Key Issues

1. Potential confounding factors

2. Data coverage of the hospital sector

3. Reporting of meaningful composites

4. Indicator instability in small populations

5. Indicator-specific issues:

• Technical specification of indicators

• Cross walk of ICD 9 and ICD 10 codes

• Country variations in coding practice/capacity

Hospital Coverage

• Denominator =Population not Admissions (i.e. not same unit of measurement).

• Sensitive to coverage of hospital system.

• Not all countries provided the additional data (e.g. UK could be 10% understated).

• Adjustment of rates may result in slight overestimation in some cases (i.e. 40% of countries indicated they thought the public rate would be higher).

Adjusted Rates

Conclusions

• Without adjustment potential exists for underestimation of rates in some countries.

• Incomplete additional data for 2009-10.

• Unadjusted rates reported in 2009 edition of Health at a Glance.

• Reconsider application of adjusted rates based on enhanced data in 2010-11.

Key Issues

1. Potential confounding factors

2. Data coverage of the hospital sector

3. Reporting of meaningful composites

4. Indicator instability in small populations

5. Indicator-specific issues:

• Technical specification of indicators

• Cross walk of ICD 9 and ICD 10 codes

• Country variations in coding practice/capacity

Composite

What is the objective?

1. To explain the variation in the individual indicators through the use of one or more composite indicators?

2. To indicate the performance of the whole system through the aggregation of indicators that relate to parts of the system?

Some countries currently report composite indicators (e.g. Canada, Australia).

AHRQ Findings

• Indicators are positively correlated

• Positive factor loadings on the 1st factor

• 1st factor explains 94% of variation

• Indication for separate diabetes composite (high factor loading on the 2nd factor)

HCQI Findings

• Not all indicators are positively correlated (e.g. angina and diabetic amputations) ×

• Factor loading for amputation is negative ×

• 1st factor only explains 31% of variation ×

• No clear indication for a 2nd factor ×

Preliminary HCQI analysis does not provide clear support for use of composite.

Correlation

| Asthma COPD CHF Angina Hyper Diab ST Diab LT Diab UC DiabLEA

-------------+------------------------------------------------------------------------------------

Asthma | 1.0000

COPD | 0.1812 1.0000

CHF | 0.1559 0.0152 1.0000

Angina | -0.1658 0.4965 0.1733 1.0000

Hypertension | 0.0627 0.2765 0.4532 0.3830 1.0000

Diabetes ST | 0.5346 0.2123 0.2886 -0.2476 -0.0903 1.0000

Diabets LT | 0.1367 0.1675 0.0892 -0.0779 0.5676 0.1674 1.0000

Diabetes UC | 0.1403 0.2929 0.1552 0.2508 0.8065 0.0226 0.5757 1.0000

Diabetes LEA | 0.0692 -0.2891 0.3810 -0.3284 -0.1404 0.2650 0.2201 -0.3302 1.0000

Rotated factor loadings (pattern matrix) and unique variances

-------------------------------------------------

Variable | Factor1 Factor2 | Uniqueness

-------------+--------------------+--------------

Asthma | 0.1936 0.5950 | 0.6085

COPD | 0.5645 -0.1366 | 0.6627

CHF | 0.3596 0.4773 | 0.6429

Angina | 0.5136 -0.5182 | 0.4677

Hypertension | 0.9054 0.0088 | 0.1802

Diabetes ST | 0.0829 0.7547 | 0.4235

Diabetes LT | 0.6062 0.3689 | 0.4965

Diabetes UC | 0.8721 -0.0298 | 0.2385

Diabetes LEA | -0.2611 0.6924 | 0.4524

-------------------------------------------------

Proportion 0.3080 0.2284

Key Issues

1. Potential confounding factors

2. Data coverage of the hospital sector

3. Reporting of meaningful composites

4. Indicator instability in small populations

5. Indicator-specific issues:

• Technical specification of indicators

• Cross walk of ICD 9 and ICD 10 codes

• Country variations in coding practice/capacity

Indicator Stability

• Inherent instability in annual indicator values (high random error) associated with low numerator values indicates use of multi- year averaging or aggregation of numerator cases.

• Particular issue for smaller states (e.g. Iceland and Luxembourg).

• Use of the 3-year rolling average would facilitate comparison with other countries.

Key Issues

1. Potential confounding factors

2. Data coverage of the hospital sector

3. Reporting of meaningful composites

4. Indicator instability in small populations

5. Indicator-specific issues:

• Technical specification of indicators

• Cross walk of ICD 9 and ICD 10 codes

• Country variations in coding practice/capacity

Asthma

• Differential diagnosis of asthma/COPD.

• Code coverage issue between ICD 9 and ICD 10:

– Coding guidelines indicate that ICD 9 4932 equatesto ICD 10 J448.

– Supplementary data indicates that chronicobstructive asthma (4932) accounts for 10-50% ofasthma numerator cases (n=4).

– Plan to remove J448 from COPD and add to Asthmacodes to align with ICD 9 code coverage for futuredata collections.

Asthma and COPD

Asthma

COPD

(PulmonaryEmphysema)

COPD

(ChronicBronchitis)

Chronic Obstructive

Asthma (ICD 9 4932)Other Specified

COPD (ICD 10 J448)

COPD

• Alternative calculation suggested by Denmark:

– Aged 30+ years with J44 as PDX, and

– J44 as SDX and one of the following PDX:

• J96 (Respiratory Failure not elsewhere classified)

• J13-18 (Pneumonia)

– Reflect common diagnosis of patients with COPD (i.e. PDX = Pneumonia).

Potential Issues

• Use of SDX is potentially problematic.

• Create inconsistency in the numerator and denominator across indicators impact on ability to aggregate indicators.

Angina

• Calculation based on principal diagnosis.

• Supplementary data was collected during HCQI data collection for 2008-09 to explore variations in coding practices:

– Cases identified in SDX (any PDX).

– Cases identified in SDX (PDX of chest pain).

PDX and SDX

Explanation

• Secondary diagnosis coding

– Limited data availability for this analysis:

• Italy had the lowest number of SDX codes and highest proportion of PDX cases.

• However, this relationship did not hold for other countries.

• Other potential factors at play?

– E.g. Saver et al. (2009) attribute a sharp decline in angina admissions in the 1990’s to more aggressive diagnosis of coronary atherosclerosis.

Secondary Diagnosis

Saver et al. in Medical Care 2009:47, pp.1106-1110

Discharge Trends

Saver et al. in Medical Care 2009:47, pp.1106-1110

Angina DX on Admission and

Discharge DX

Hypertension

• Benign hypertension included in ICD 10 code list but not ICD 9 code list (i.e. 4011)

• Questions over validity clinical grounds for PDX (The Netherlands, NZ).

Lower Extremity Amputation

• Inclusion of toe amputation

– Interpretation in relation to quality of care

– Performed in hospital and ambulatory service settings and on a day case and inpatient basis.

• Use of secondary diagnoses

– Up to 50% cases identified in SDX codes

– Availability of secondary diagnosis codes?

• Trauma code exclusions

– Adequacy of code list

Toe Amputation

• Reflects tertiary prevention

– Can reflect good quality of care as it prevents further complications (OECD 2004)

– Failed primary and secondary prevention in primary care system retain in code set?

• Only multi-day stays reflected in data.

– If ambulatory care = day cases retain in code set?

Secondary Diagnosis

Secondary Diagnosis

Consider further data collection to locate where cases are identified in SDX codes, similar to PSI in 2008-09.

Options:

1. Limit calculation to:

– countries able to report above a lower threshold number of SDX codes , and/or

– an upper threshold number of SDX codes

2. Apply adjustment methodology, similar to PSI

3. Limit the calculation to PDX only.

Trauma Exclusions

• Currently includes traumatic amputation of toe, foot and leg (ICD 9 codes 895-897)

• There may be justification for extending the exclusion code list.

– The Netherlands suggest inclusion of:

• Fracture of specified bones of lower limbs (820 to 828)

• Late effect of muscle & skin injury (905,906)

• Crushing injury of lower limb & other sites (928, 929)

• Burns of lower limbs & other sites (945, 946)

Uncontrolled Diabetes

• Potentially problematic for ICD 10 countries.

• 60% (11) countries reporting data in 2008-09 could identify uncontrolled cases:

– Just<50% by using ICD 9

– Just >50% by using ICD 10 markers.

• Austria, Canada, Finland, Korea, UK, Norway, Poland and Sweden could not identify uncontrolled cases.

• Canada establishing a marker for 2009+

Suggested Approach to Key Issues

1. Continue to examine potential confounding and explanatory factors and report key findings with the indicators in OECD publications.

2. Collect further data on hospital coverage from all countries in 2010-11 and review approach.

3. Further analysis required before proceeding with the use of composite indicators.

4. Present 3-year averages for Iceland and Luxembourg in the future to address indicator instability.

5. Consider indicator-specific issues on a case by case basis next slide.

Indicator-Specific Issues

Indicator Suggested Approach

Asthma Revise ICD 10 code list (add J448)

Chronic Obstructive Pulmonary Disease For discussion

Congestive Heart Failure No action

Angina* For discussion

Hypertension Revise ICD 9 code list (add 4011)

Diabetes : Short –term complications No action

Diabetes: Long –term complications* For discussion

Diabetes: Lower -extremity amputation For discussion

Diabetes: Uncontrolled* For discussion

* Not currently considered for publication in 2009.

The Subgroup is invited to:

–comment on these methodological issues

–endorse plans for further analysis and refinement of the indicators in preparation for the next round of data collection in 2010-11.