parvin tajik, md phd candidate department of clinical epidemiology & biostatistics
DESCRIPTION
FHCRC 2014 Risk Prediction Symposium June 11, 2014. Illustration of the evaluation of risk prediction models in randomized trials Examples from women’s health studies. Parvin Tajik, MD PhD candidate Department of Clinical Epidemiology & Biostatistics Department of Obstetrics & Gynecology - PowerPoint PPT PresentationTRANSCRIPT
Illustration of the evaluation of risk prediction models in randomized trialsExamples from women’s health studies
Parvin Tajik, MDPhD candidateDepartment of Clinical Epidemiology & BiostatisticsDepartment of Obstetrics & GynecologyAcademic Medical Center, University of Amsterdam, the Netherlands
FHCRC 2014 Risk Prediction SymposiumJune 11, 2014
Clinical Problem I
Pre-eclampsia
fullPIERS model
Lancet, 2011
Development Method
• Patients: • 2000 women admitted in hospital for pre-eclapmsia
(260 event)
• Outcome: • Maternal mortality or other serious complications of
pre-eclampsia
• Logistic regression model with stepwise backward elimination
Final model
Logit P(D) = 2.68 – (0.054 × gestational age at eligibility) + (1.23 × chest pain or dyspnoea) – (0.027 × creatinine) + (0.21 × platelets) + (0.00004 × platelets2) + (0.01 × AST) – (0.000003 × AST2) + (0.00025 × creatinine × platelet) – (0.00007 × platelets × AST) – (0.0026 × platelets × SpO2)
Performance of full-PIERS model
Reported good risk discrimination and calibration
Online calculator
HYPITAT trial (2005-2008)
• PP Women at 36-41 wks of pregnancy with mild pre-eclampsia (n=750)
• I I Early Induction of labor (LI)
• C C Expectant monitoring (EM)
• O O Composite measure of adverse maternal outcomes
HYPTAT Results
(relative risk 0.71, 95% CI 0.59–0.86, p<0·0001)
ManagementManagement Adverse maternal Adverse maternal outcomesoutcomes
TotalTotal
Labor induction 117 (31%) 377Expectant monitoring 166 (44 %) 379
Modeling
Logit P(D=1|T,Y) = β0 + β1T + β2Y + β3TY
•D = 1 Adverse maternal outcome•Y = fullPIERS score•T = Treatment
• 1 Labor induction • 0 Expectant monitoring
FullPIERS for guiding labor induction
P for interaction: 0.93
fullPIERS score
Clinical Problem II
Preterm birth
Cervical pessary• Medical device inserted to vagina• to provide structural support to cervix
ProTWIN trial (2009-2012)
• P Women with multiple pregnancy (twin or triplet) between 12 & 20 weeks pregnancy
• I Cervical Pessary (n = 403)• C Control (n = 410)
• O Primary: Composite Adverse perinatal outcome
ProTWIN Results
(relative risk 0.98, 95% CI 0.69–1.39)
ManagementManagement Composite adverse Composite adverse perinatal outcomeperinatal outcome
TotalTotal
Pessary 53 (13%) 401No pessary 55 (14 %) 407
Pre-specified subgroup analysis
Cervical length (<38 mm vs >= 38 mm)
Pre-specified subgroup analysis
Trial Conclusion: Clinicians should consider a cervical pessary in women with a multiple pregnancy and a short
cervical length.
Cervical length Pessary group
Control group
RR (95%CI)
CxL < 38 mm 12% 29% 0.42 (0.19-0.91)CxL >= 38 mm 13% 10% 1.26 (0.74-2.15)
(P for interaction 0.01)
Other Markers
1. Obstetric history (parity) • Nulliparous• Parous with no previous preterm birth• Parous with at least one previous preterm birth
2. Chorionicity• Monochorionic• Dichorionic
3. Number of fetuses• Twin• Triplet
One marker at a time analysis
Other Potential Treatment Selection Factors
% Poor Outcome Odds Ratio (95% CI)
Odds Ratio (95% CI)
Int. P-value
Pessary Control
Cervical length
< 38 mm 11.54 29.09 0.32 (0.13-0.79) 0.010
≥ 38mm 12.85 10.13 1.31 (0.75-2.30)
Chorionicity
Monochorionic 13.79 26.00 0.46 (0.21-0.97) 0.015
Dichorionic 13.06 9.51 1.43 (0.86-2.37) Obstetric history
Nulliparous 13.12 18.30 0.67 (0.40-1.13) 0.212
Parous with no previous preterm birth 9.93 8.28 1.22 (0.56-2.66)
Parous with at least one previous preterm birth
31.03 3.85 11.25 (1.31-96.4) 0.012
Number of foetuses
Twin 12.50 13.32 0.98 (0.61-1.41) 0.301
Triplet 44.44 22.22 2.8 (0.36-21.73)
Modeling
Logit P(D=1|T,Y) = β0 + β1T + Σ βiYi + Σ βjTYj
•D = 1 composite poor perinatal outcome•Y = Markers•T = Treatment
• 1 pessary• 0 control
- Internal validation by bootstrapping
Multi-marker modelPredictor OR (95% CI) Beta*
P-value
Intercept
-2.08
<0.001 Main terms Pessary 1.13 (0.57-2.24) 0.12 0.426
Cervical length <38 mm 2.20 (1.09-4.46) 0.79 <0.001
Monochorionic 2.44 (1.33-4.47) 0.89 <0.001
Parous with no previous preterm birth 0.53 (0.27-1.06) -0.63 0.031
Parous with at least one previous preterm birth 0.34 (0.04- 2.63) -1.09 0.165
Triplet 1.49 (0.28- 8.05) 0.40 0.010
Interaction terms
Pessary × Cervical length <38 mm 0.52 (0.19-1.42) -0.65 0.058
Pessary × Monochorionic 0.41 (0.16-1.05) -0.89 0.009
Pessary × Parous with no previous preterm birth 1.52 (0.58-3.98) 0.42 0.312
Pessary × Parous with at least one previous preterm birth 7.24 (0.78-67.65) 1.98 0.020
* Shrunken with an average shrinkage factor of 0.76c-stat : 0,71 (95%CI: 0,66-0,77); optimism-corrected c-stat: 0,69 (95%CI: 0,63-0,74)
How can the model be used in practice?
Predicted benefit from pessary
050
100
150
200
250
Stu
dy
Pa
rtic
ipa
nts
, %
* ** *** ** ** ** **** * ** ** **** *** **** ***** * *** ** *** **** * ** *** ** **** * ***** ***** ** ** ** ** ***** ****
-0.2 -0.1 0.0 0.1 0.2
Predicted Difference (Control-Pessary) in Poor Perinatal Outcome
Favors Control Favors Pessary
Calibration of the predicted benefit
-30 -20 -10 0 10 20 30 40
-30
-20
-10
01
02
03
04
0
Expected Treatment Effect
Ob
serv
ed
Tre
atm
en
t Effe
ct
Model performance
-30 -20 -10 0 10 20 30 40
-30
-20
-10
01
02
03
04
0
Expected Treatment Effect
Ob
serv
ed
Tre
atm
en
t Effe
ct
Conclusion
• Common assumption for application of risk prediction models for treatment selection:“Being at higher risk of outcome implies a
larger benefit from treatment” • Not necessarily true
• Developing models using trial data and modeling the interaction between markers and treatment might be a more optimal strategy
Open Research Questions
• Optimal modeling strategy?
• Optimal algorithm for variable selection?
• Optimal method for optimism correction?
Thanks!Any Questions?
Multimarker vs. CxL only
Multimarker + Multimarker -
Short cervix 174 9
Long cervix 120 505
Two examples