50 shades of grey - australian and new zealand college of ... · 50 shades of grey how (in)accurate...

24
50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising risk Perioperative SIG, Grand Hyatt, Melbourne October 27 th 2018

Upload: others

Post on 22-Jan-2020

7 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising

50 shades of greyhow (in)accurate are risk prediction scores

Dr Doug CampbellMeasuring, managing and minimising riskPerioperative SIG, Grand Hyatt, Melbourne

October 27th 2018

Page 2: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising

Declarations and Acknowledgements

• No relevant conflicts of interest• Funded by Precision Driven Health

Page 3: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising
Page 4: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising
Page 5: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising

Those who make many species are the 'splitters,' and those who make few are the 'lumpers.'

Page 6: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising

Myth of individualised prediction

…with apologies to XKCD

Page 7: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising

Validation 101- easy as ABCD

Page 8: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising

Validation 101- easy as ABCD

A=0B=0.96C=0.922

Page 9: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising

nzRISK – CalibrationIntercept = 0Slope = 0.97McFadden’s R2 = 0.28AUROC = 0.92

Intercept = 0Slope = 0.96McFadden’s R2 = 0.31AUROC = 0.91

Intercept = 0Slope = 0.98McFadden’s R2 = 0.37AUROC = 0.90

1 month

1 year 2 year

Page 10: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising

nzRISK – DiscriminationCentile Total Predicted Observed Predicted Observed

N (%) N % % N N

1 9481 0 0 0.9 1

2 8528 0 0 1.5 0

3 9303 0 0 2.9 1

4 8713 0.1 0.1 5.4 5

5 9026 0.1 0.1 9.5 5

6 9192 0.2 0.1 14.8 8

7 (0.2-0.3) 8784 0.2 0.3 21.6 26

8 (0.3—0.5) 9008 0.4 0.2 35.9 21

9 (0.5 – 1.5) 9004 0.9 0.9 77.5 78

10 (>1.5) 8996 5.6 5.4 508.1 489

Page 11: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising

nzRISK – DiscriminationCentile Total Predicted Observed Predicted Observed

N (%) N % % N N

1 9481 0 0 0.9 1

2 8528 0 0 1.5 0

3 9303 0 0 2.9 1

4 8713 0.1 0.1 5.4 5

5 9026 0.1 0.1 9.5 5

6 9192 0.2 0.1 14.8 8

7 (0.2-0.3) 8784 0.2 0.3 21.6 26

8 (0.3—0.5) 9008 0.4 0.2 35.9 21

9 (0.5 – 1.5) 9004 0.9 0.9 77.5 78

10 (>1.5) 8996 5.6 5.4 508.1 489

Low risk, NZRISK < 0.5%, 11% of deaths

High risk, NZRISK > 1.5%, 77% of deaths

Intermediate risk, NZRISK 0.5-1.5%, 12% of deaths

Page 12: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising

Decision curve analysis

Glance et al. Impact of the choice of model for identifying low-risk patients using the 2014 ACC/AHA perioperative guidelines. Anesthesiology 2018; 129(5): 889-900

Page 13: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising

Generalisability

SORT internal validation

AUROC = 0.91

AUROC = 0.90

SORT external validation

Page 14: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising

Generalisability

SORT internal validation

AUROC = 0.91

AUROC = 0.90

SORT external validation

Protopapa KL et al. Development and validation of the Surgical Outcome Risk Tool (SORT). Br J Surgery 2014; 101: 1774-83Campbell D et al. Development and validation of a multivariable prediction model of perioperative mortality in noncardiac surgery (NZRISK). Br J Surgery 2018; in press

Page 15: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising

What risk tool for CADENZA?

Eugene N et al. Development of the NELA risk model. BJA 2018; 121(4): 739-48

Page 16: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising

ACS-NSQIP - morbidity

Page 17: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising

DAH90 – the median is not the message

Campbell D, Boyle L, Djamali K, Cumin D, Weller J, Short T, Merry A. Unpublished data.

New Zealand 2011-6, n = 141331-month mortality = 4.2%3-month mortality = 5.7%

83 days48 days

Page 18: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising

Checklist for risk tools

• Does it model the correct outcome?– 1-month mortality, morbidity, long-term mortality

• Is it accurate?– calibration (SHARED DECISION-MAKING)– discrimination (TRIAGE)

• Is it generalizable to my patient?– validated in Australia or New Zealand?

Page 19: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising

www.nzrisk.com

Page 20: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising

www.nzrisk.com

Page 21: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising

www.nzrisk.com

Page 22: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising
Page 23: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising

Comparison - general

Calculator Covariates Country N AUROC Calibration

ACS-NSQIP 22 US 1.4 million 0.944 Brier 0.011POSPOM 17 France 5.5 million 0.929 Brier 0.004nzRISK 8 NZ 360140 0.922 Brier 0.007SORT 6 UK 16,788 0.91 H-L 12.16P-POSSUM 18 UK 10,648 total 0.83 (0.68-0.92)

H-L = Hosmer-Lemeshow statistic, McF = McFaddens pseudo rho-squared statistic, Brier = Brier score, O:E observed to expected ratio

Page 24: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising

High risk calibration