www.pkssk.fiPohjois-Karjalan sairaanhoito- ja sosiaalipalvelujen kuntayhtymä

WHICH RISK WHICH RISK ADJUSTMENT MODEL ADJUSTMENT MODEL SHOULD WE USE? A SHOULD WE USE? A

FINNISH POINT OF VIEWFINNISH POINT OF VIEW

16.3.2011Matti Reinikainen

North Karelia Central Hospital, Joensuu

THE FINNISH INTENSIVE CARE CONSORTIUM

1994 2007

• So far, benchmarking in the Finnish Intensive Care Consortium has been mainly based on SAPS II– Based on “The Severity Study”– 13 152 patients (720 from 7 Finnish hospitals)– Le Gall JR, Lemeshow S, Saulnier F. A new Simplified

Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA 1993; 270: 2957-63.

• APACHE II data is also collected

APACHE II vs. SAPS II

• same basic principle, values of physiologic parameters from the first 24 hrs in the ICU

• APACHE II (Acute Physiology And Chronic Health Evaluation II): the diagnostic category weight is added to the logit

• SAPS II (Simplified Acute Physiology Score II): the diagnosis is not needed; instead the type of admission (scheduled surgical, unscheduled surgical, medical) affects the score

• APACHE II - from 1985- not always easy to choose the right diagnostic category

• SAPS II- from 1993- advantage: no diagnosis needed- disadvantage: does not take into account the diagnosis

ARE THE OLD MODELS GOOD ENOUGH?

DOES THE RISK PREDICTED BY SAPS II REFLECT REALITY?

• A patient example:

- age 65 years

- no difficult chronic diseases

- a medical admission

- respiratory insufficiency, need for mechanical venti-lation, PaO2/FIO2 250 mmHg (33.3 kPa)

- HR 110/min

- SAPs 84 mmHg

-Tc 38 ºC

- consciousness, renal function, blood cell counts, electrolytes quite OK

- HCO3- 18 mmol/l

• PROBABILITY OF IN-HOSPITAL DEATH ?

DOES THE RISK PREDICTED BY SAPS II REFLECT REALITY?

• A patient example:

- age 65 years

- no difficult chronic diseases

- a medical admission

- respiratory insufficiency, need for mechanical venti-lation, PaO2/FIO2 250 mmHg (33,3 kPa)

- HR 110/min

- SAPs 84 mmHg

-Tc 38 ºC

- consciousness, renal function, blood cell counts, electrolytes quite OK

- HCO3- 18 mmol/l

• SAPS II score 32 points → probability 0.128

• SAPS II –score 32 → probability 0.128

- the database of the Finnish Consortium, 1998-2007, readmissions excluded: 2319 patients, with a SAPS II score of 32 points

- hospital mortality 8.4%

• SAPS II –score 32 → probability 0,128

- the database of the Finnish Consortium, 1998-2007, readmissions excluded: 2319 patients, with a SAPS II score of 32 points

- hospital mortality 8.4%

- diabetic ketoacidosis (n = 26): mort 0%

- drug intoxication (n = 108): mort 0.9%

- congestive heart failure (n = 49): mort 22.4%

CAN SAPS II STILL BE USED?

• It overestimates the risk of death – leads to ”grade inflation” • If most intensive care units are graduating with

honors, is it genuine quality or grade inflation? Popovich MJ, Crit Care Med 2002

• Recalibrations are needed

Tehohoidon laatupäivät Helsingissä 1.4.2008

SMR 1998 – 2007, FINNISH INTENSIVE CARE CONSORTIUM

Päivitetty 09.04.2008

SMR based on original SAPS II model

SMR based on new calibration

CAN SAPS II STILL BE USED?

• It can be used for monitoring changes in a unit’s own results

• Can be used for benchmarking purposes if the units to be compared have similar case-mix

• Should not be used to compare results of units with major differences in case-mix

SAPS 3 WAS CONSIDERED IN FINLAND TOO - IS IT A GOOD

ALTERNATIVE?

• Values of physiological parameters ± 1 h of ICU admission

• Reason for ICU admission documented more precisely than in SAPS II

• Takes into account pre-ICU care• Prognostic performance?• Quality of data collected??

The SAPS 3 StudyMetnitz et al ICM 2005: 31:1336-1344. (Part 1)Moreno et al. ICM 2005: 31:1345-1355. (Part 2)

• At first 22,791 admissions• Exclusions: readmissions (1455), < 16 yrs (628),

those without ICU admission or discharge data (1074) and those that lacked an entry in the field ”ICU outcome” (57)

- SAPS 3 basic cohort: 19,577 patients

• SAPS 3 basic cohort: 19,577 patients• More exclusions: patients with a missing entry in the

field of ”vital status at hospital discharge” (2540) and those still in hospital (253)– SAPS 3 Hospital outcome cohort: 16,784 patients

• Quality of data? – at first, 5.5% of patients excluded because of missing data; then 13% of the remaining population excluded because of missing data on vital status

The SAPS 3 StudyMetnitz et al ICM 2005: 31:1336-1344. (Part 1)Moreno et al. ICM 2005: 31:1345-1355. (Part 2)

The SAPS 3 StudyMetnitz et al ICM 2005: 31:1336-1344. (Part 1)Moreno et al. ICM 2005: 31:1345-1355. (Part 2)

• How about data completeness?– ”Data completeness was found to be satisfactory with 1

[0-3] SAPS II parameter missing per patient”

• How many SAPS 3 parameters were missing?– ???

– Were the physiological values obtained within ± 1 h?

SAPS 3 – even if data quality in the study was less than perfect, does it work?

• Ledoux D et al. SAPS 3 admission score: an external validation in a general intensive care population. Intensive Care Med 2008; 34: 1873-7.– single-centre (Belgium), 802 patients

– “the SAPS 3 … model customised for Central and Western Europe … was not significantly better than the SAPS II.”

• Poole D et al. External validation of the Simplified Acute Physiology Score (SAPS) 3 in a cohort of 28,357 patients from 147 Italian intensive care units. Intensive Care Med 2009; 35: 1916-24.– “…the SAPS 3 score calibrates inadequately in a large sample of

Italian ICU patients and thus should not be used for benchmarking, at least in Italian settings”

• Sakr Y et al. Comparison of the performance of SAPS II, SAPS 3, APACHE II, and their customized prognostic models in a surgical intensive care unit. Br J Anaesth 2008; 101: 798-803.– single-centre (Germany), 1851 patients– “… the performance of SAPS 3 was similar to that of APACHE II

and SAPS II. Customization improved the calibration of all prognostic models.”

• Metnitz B, Schaden E, Moreno R, Le Gall JR, Bauer P, Metnitz PG; ASDI Study Group. Austrian validation and customization of the SAPS 3 Admission Score. Intensive Care Med 2009; 35: 616-22.– 22 ICUs in Austria, 2060 patients– “The SAPS 3 … general equation can be seen as a framework … For

benchmarking purposes, region-specific or country-specific equations seem to be necessary...”

• 2 ICUs in Norway, 1862 patients• “The performance of SAPS 3 was satisfactory, but not

markedly better than SAPS II.”

• SAPS II showed better discrimination• SAPS 3 equations showed better calibration• “…in our experience the scoring process is more time-

comsuming and complex than that for SAPS II.”

SAPS 3, CONCLUSION:

• Does it work? – Yes!• However, prognostic performance is NOT better

than that of SAPS II• the scoring process is more time-comsuming and

complex than that for SAPS II (experience from Norway)

• on the other hand: according to many studies, the calibration of SAPS II is poor and customisation is needed

QUESTION DISCUSSED IN FINLAND:

• Should we implement a new risk-adjustment model (SAPS 3) that – is not better than the old ones

– is more time-consuming

– would require customisation

• Or should we go on with one of the old models (that also require customisation)?

FINNISH (at least temporary) SOLUTION: OWN CUSTOMISED

PREDICTION MODEL • One objective: no need to exclude patient groups

for benchmarking– neuro- and cardiac surgical patients are not excluded

• We did not want to increase the burden of data collection – no new parameters added

• SAPS II –based data collection preserved– possible to compare the results with those of previous

years – possible to describe the population using a well-known

scoring system

• Based on patients treated in 2007-2008 • Readmissions excluded• Age ≥ 18 yrs• Those discharged to another ICU excluded • n = 25 801

OWN CUSTOMISED MODEL- M Reinikainen, P Mussalo, V Kiviniemi, V Pettilä, E Ruokonen

OWN CUSTOMISED MODEL• Outcome variable (to be predicted) ”DEATH IN

HOSPITAL”• Explaining covariates:

– Emergency admission or planned beforehand – Surgical postoperative or medical– SAPS II score without admission type points– ln ((SAPS II score without admission type points) + 1)– Diagnostic groups having an independent impact on the

probability of death • First a binary variable (0,1) was made of every APACHE

III –dg group; everyone of these was tested separately• 31 dg groups with an independent effect were included in

the model

LOGISTIC REGRESSION ANALYSIS

R1

Rln logit

logit = β0 + β1X1 + β2X2 + … + βiXi

- the regression analysis produces the constant β0 and the coefficients βi

-the logit can be calculated when the parameter values Xi are known

- the logit (log odds) can also be expressed as

logit

logit

e1

eR

and thus

CALCULATING THE RISK R

R1

Rln logit

R1

R elogit

)e(R-e R logitlogit

logitlogit e)Re (1

logit

logit

e1

eR

LOGIT = -7,796 + 0,049 x (SCORE_SAPS_WITHOUT_ADM_TYPE_POINTS) + 1,013 x (ln(SAPS_WITHOUT_ADM_TYPE_POINTS + 1)) + 0,767 (if emergency admission) - 0,219 (if post-operative admission) + 1,229 (if DG_NONOP_CARDIOGENIC_SHOCK) + 0,364 (if DG_NONOP_CARDIAC_ARREST) – 0,796 (if DG_NONOP_RHYTHM_DISTURBANCE) + 0,348 (if DG_NONOP_ACUTE_MYOCARDIAL

INFARCTION) + 0,422 (if DG_NONOP_BACTERIAL_OR_VIRAL_PNEUMONIA) – 1,619 (if DG_NONOP_MECHANICAL_AIRWAY_OBSTRUCTION) + 0,306 (if DG_NONOP_OTHER_RESP_DISEASES) + 0,795 (if DG_NONOP_HEPATIC_FAILURE) + 0,703 (if DG_NONOP_GI_PERFORATION_OR_OBSTRUCTION) + 0,643 (if DG_NONOP_GI

BLEEDING_DUE_TO_VARICES) + 0,431 (if DG_NONOP_OTHER_GI_DISEASES) + 0,790 (if DG_NONOP_INTRACEREBRAL_HAEMORRHAGE) + 0,654 (if DG_NONOP_SUBARACHNOID_HAEMORRHAGE) + 0,400 (if DG_NONOP_STROKE) – 1,427 (if DG_NONOP_NEUROLOGIC_INFECTION) - 1,266 (if DG_NONOP_SEIZURE) – 0,486 (if DG_NONOP_OTHER_NEUROLOGIC_DISEASES) - 0,679 (if DG_NONOP_MULTIPLE

TRAUMA_WITHOUT_HEAD_TRAUMA) – 0,658 (if DG_NONOP_METABOLIC_COMA) – 2,126 (IF DG_NONOP_DIABETIC_KETOACIDOSIS) – 2,245 (if DG_NONOP_DRUG_OVERDOSE) – 1,150 (if DG_NONOP_OTHER_METABOLIC_DISEASES) – 0,752 (if DG_NONOP_OTHER

MEDICAL_DISEASES) + 0,340 (if DG_POSTOP_DISSECTING_OR_RUPTURED_AORTA)– 0,701 (if DG_POSTOP_CABG) + 0,701 (if DG_POSTOP_PERIPH_ARTERY_BYPASS_GRAFT) + 0,470 (if DG_POSTOP_GI_PERFORATION_OR_RUPTURE) + 0,411 (if DG_POSTOP

GI_OBSTRUCTION) – 0,522 (if DG_POSTOP_SUBDURAL_OR_EPIDURAL_HAEMATOMA)– 0,885 (if DG_POSTOP_CRANIOTOMY_FOR_NEOPLASM) – 1,620 (if DG_POSTOP_OTHER_RENAL_DISEASES)

PROB = EXP(LOGIT) / (1 + EXP(LOGIT))

PATIENT EXAMPLE

• A patient example:

- age 65 years

- no difficult chronic diseases

- a medical admission

- respiratory insufficiency, need for mechanical venti-lation, PaO2/FIO2 250 mmHg (33.3 kPa)

- HR 110/min

- SAPs 84 mmHg

-Tc 38 ºC

- consciousness, renal function, blood cell counts, electrolytes quite OK

- HCO3- 18 mmol/l

• SAPS II score 32 points → probability 0.128

• SAPS II score 32 → probability 0.128

• New customised model:

– If none of the diagnoses included in the model: probability 0.082

– dg bacterial pneumonia: probability 0.12

– dg drug intoxication: probability 0.0094

PATIENT EXAMPLE

AUROC

- APACHE II: 0.84

- SAPS II: 0.84

- new customised model: 0.87

- H-L test for new model: p = 0.127

CONCLUSIONS• SAPS 3 works, but its prognostic performance is

not better than that of SAPS II• If you want to use SAPS 3, you should probably

customise it• If you want to use SAPS II, you should probably

customise it• Idea for future research: to create a Nordic risk

adjustment model, predicting 6-month or 1-year mortality