Challenges for institutional performance measures

Responsible Data Science in health care

Nicolette de Keizer Dept Medical Informatics

Academic Medical Center, Amsterdam

2

Data science in health care

Data reuse: • Management information • Quality assurance • Research • Surveillance

3

Learning Health System

Data reuse: • Management information • Quality assurance • Research • Surveillance • Financial reimbursement

Quality registries in health care >150 registries in health care Aims: – Accountability

• Government • Insurance companies • Patients

– Quality assessment and improvement – Scientific research

Accountability

Accountability Is the data fair, accurate, transparent?

Accountability Is the data fair, accurate, transparent? Large consequences ….. – Loss of faith – Demotivation – Loss of budget

8

9

History and development

Founded in 1996 by and for intensivists

0

100000

200000

300000

400000

500000

600000

700000

800000

900000

1000000

0

10

20

30

40

50

60

70

80

90

100

1997199819992000200120022003200420052006200720082009201020112012201320142015

# admissions

# IC

Us

Quality assessment and improvement Benchmarking

Observed difference = Difference in quality of care Onverklaarde

verschillen Onverklaarde verschillen

Onverklaarde verschillen

Registratie verschillen

Patiënten kenmerken

Toeval

Kwaliteit van zorg

0,40

0,50

0,60

0,70

0,80

0,90

1,00

A B C D E F G H

indi

cato

r sco

re

ICU/institution

Variation

Observed diference

Unexplained differences

Onverklaarde verschillen

Onverklaarde verschillen

Patiënten kenmerken

Toeval

Kwaliteit van zorg

Case mix

Mortality 20% 17%

Age 68 57

Comorbidity 40% 5%

Variation

Observed difference

Unexplained differences

Onverklaarde verschillen

Case mix

Toeval

Kwaliteit van zorg

Unexplained differences

Registration, definition differences

Variation

Observed difference

Unexplained differences

Uncertainty

Quality of care

Unexplained differences

Unexplained differences

Case mix

Registration, definition differences

Mortality as quality indicator

14

Input (1st 24 hour) Output

ICU Hospital

Prognostic models: APACHE II en IV, SAPSII…

15

Standardized Mortality Ratio (SMR)

Expected mortality depends on prognostic model

Observed in-hospital mortality

Expected mortality

SMR=

Ranking institutions

A common procedure is to rank institutions by SMR, providing a league table The top/bottom 10% or 25% are sometimes labelled as excellent/poor performers

Marshall EC, Spiegelhalter DJ. BMJ 1998;316(7146):1701-4.

ICU ranking accurate?

Ranks of 40 Dutch ICUs (n=86,427) SMRs based on Apache II, SAPSII, MPM24II model Rank CIs computed by bootstrap sampling (10,000 replications) Excellent performance: with 95% certainty among top 25% institutes Very poor performance: with 95% certainty among bottom 25% institutes

Bakhshi-Raiez F et al. Crit Care Med 2007

0 10 20 30 40

215

251836223

1033408

1539141911132779

384

322

313537231729126

2016281

34243026

0 10 20 30 40

518141021258

367

3319222

39383

4030374

1527281229119

176

162431321

263413202335

0 10 20 30 40

18255

10403339363738192

2122157

1432172949

2338

1227306

3416112831351

13242620

Apache II SAPS II MPM24 II

Bak

hshi

-Rai

ez F

et a

l. C

rit C

are

Med

200

7

Results

20 ICU significantly differ in rank (2-19 positions) by 1 or more pair of models 3 ICUs rated as performance outlier by one model while others excluded this possibility with 95% certainty

215

251836223

1033408

1539141911132779

384

322

3135372317

518141021258

367

3319222

39383

4030374

1527281229119

176

18255

10403339363738192

2122157

1432172949

2338

1227306

Apache II SAPS II MPM24 II

215

251836223

1033408

1539141911132779

384

322

3135372317

518141021258

367

3319222

39383

4030374

1527281229119

176

18255

10403339363738192

2122157

1432172949

2338

1227306

Apache II SAPS II MPM24 II

215

251836223

1033408

1539141911132779

384

322

3135372317

518141021258

367

3319222

39383

4030374

1527281229119

176

18255

10403339363738192

2122157

1432172949

2338

1227306

Apache II SAPS II MPM24 II

0 10 20 30 40

81539141911132779

384

322

313537231729126

2016281

34243026

0 10 20 30 40

19222

39383

4030374

1527281229119

176

162431321

263413202335

0 10 20 30 40

192

2122157

1432172949

2338

1227306

3416112831351

13242620

Apache II SAPS II MPM24 II

Benchmark –SMR fair?

Benchmark on in-hospital mortality or long term mortality? Why choose for hospital mortality? – Sooner and easily available – Mortality not related to ICU admission

Why choose for longterm mortality? – More relevant for patients – Less influence by discharge policy

In-hospital mortality vs long term mortality

0 0,5 1 1,5 2

32

4

2

2

2

42

323

22

2

41925

2110

67

1311161824251217

32030412722281423263840153732

83519393634332942433144

SMR 3 months

0 0,5 1 1,5 2

2

3

2

34

4

2

2

3

2222

2

123456789

1011121314151617181920212223242526272829303132333435363738394041424344

SMR In-hospital

S. B

rinkm

an e

t al.

Inte

nsiv

e C

are

Med

. 201

3 N

ov;3

9(11

):192

5-31

26

Other Challenges regarding FACT

Observed difference

Unexplained differences

Uncertainty

Quality of care

Unexplained differences

Unexplained differences

Case mix

Registration, definition differences

27

Other Challenges regarding FACT The way of data collection influences SMR – Sample frequency – Coded data versus free text

Interpretation and definition of QI – Expl IGZ: Mean duration of mechanical ventilation

• Different types of ventilation • Duration in hours or calendar days • Mean based on all ICU patients or ventilated patients

28

Challenges Learning Health System

Formalisation of indicator definitions

Information models Terminological systems

29

Conclusion Performance measurement of health facilities might be biased by – Data source – Case mix correction model – Definitions (of endpoints) Need for methods to unambiguously capture health data, formalize indicators and make health data transparent for different reuse purposes