calibration plots for risk prediction models in the presence of...

Calibration plots for risk prediction models in the

presence of competing risks

Thomas A Gerds, Thomas H Scheike, Per K Andersen andMichael W Kattan

June 26, 2014

1 / 28

Motivation: patient counseling

Using a statistical model, a database can be queried to obtain atailored prediction for the present patient.

A predicted risk of 17% is called reliable, if it can be expected thatthe event will occur to about 17 out of 100 patients who allreceived a predicted risk of 17%.

A statistical model that predicts the absolute risk of an eventshould be calibrated in the sense that it provides reliable predictionsfor all subjects.

A calibration plot displays how well observed and predicted eventstatus connect on the absolute probability scale.

2 / 28

Calibration plot

Predicted event probability

0 % 25 % 50 % 75 % 100 %

Obs

erve

d ev

ent s

tatu

s

0 %

25 %

50 %

75 %

100 %

Cause−specific Cox regression

Fine−Gray regression

3 / 28

Predicting absolute risks in time-to-event analysis

First pick a time origin at which it is of interest to predict thefuture status of a patient.

Until time t after the time origin three things can happen:

1. the event has occurred

2. a competing event has occurred

3. the patient is alive and event-free.

The patient needs to know the absolute risks of all events (death,disease, recurrence, etc.).

4 / 28

John Klein's data from bone marrow transplant patients

A data frame with 1715 observations1

Transplant

Relapse Death

n= 557

n= 311

The remaining n = 847 patientswere in remission by the end ofthe follow-up period.

We are interested in predictingthe cumulative incidences ofrelapse and death.

1Szydlo, Goldman, Klein et al. Journal of Clinical Oncology, 1997.5 / 28

Observed outcome

Months since transplantation

Cum

ulat

ive

inci

denc

e

0 12 36 60 84

0 %

25 %

50 %

75 %

100 %

Aalen−Johansen estimate

Event

RelapseDeath without relapse

Months since transplantationC

umul

ativ

e in

cide

nce

0 12 36 60 84

0 %

25 %

50 %

75 %

100 %

Kaplan−Meier estimateof censoring probability

Without covariates the marginal Aalen-Johansen estimate is thebest prediction model.

6 / 28

Formula I

Let X be a vector of covariates:

F1(t|X ) = Cumulative incidence of event 1∫ t

0

exp

(−∫ s

0

{λ1(u|X ) + λ2(u|X )}du)

︸︷︷︸No event of any cause until s

λ1(s|X )︸︷︷︸Event type 1 at s

ds.

Requires a regression model for the hazard of the competing risksor a regression model for the event-free survival probability.

7 / 28

Formula II

Transformation model

h(F1(t|X )) = β01(t) + β1X1 + · · ·+ βKXK

I h(p) = log(-log(p)) (Fine-Gray model)

I h(p) = log(p/(1-p)) (Logistic model)

I h(p) = log(p) (Log-binomial model)

Requires a regression model for the cumulative probability of beinguncensored: G(t|X) = P(T>t|X)

in what follows: G(t|X)=G0(t).

8 / 28

Interpretation crisis in competing risks

Problems:

I The hazard ratios obtained by cause-speci�c Cox regressionmodels are not directly related to the prediction of thecumulative incidence.

I The absolute values of the regression coe�cients in theFine-Gray model have no direct interpretation.

Proposal: We are interested in regression models for the absoluterisk of relapse in which the regression coe�cients have thefollowing interpretation:

The 5-year risk of relapse changes with a factor exp(β1) for a one

unit change of X1 and given values for the other predictor variables

(X2, ...,XK ).

9 / 28

Absolute risk regression

The regression parameters in the log-binomial model have thedesired interpretation:

F1(t|X ) = exp(β01(t)) exp(β1X1 + · · ·+ βKXK )

A one unit change of the kth covariate:

F1(t|X1, . . . ,Xk = xk , . . . ,XK )

F1(t|X1, . . . ,Xk = (xk + 1), . . . ,XK )= exp{βk(xk − xk + 1)}

= exp(βk).

10 / 28

Bone marrow transplant data: absolute risk of relapse

Factor exp(β) CI.95 P-value

disease:ALL � � �disease:AML 0.86 [0.68;1.08] 0.1982292disease:CML 0.58 [0.44;0.76] 0.0001017karnofsky 1.3 [1.03;1.68] 0.0253975donor:sibling � � �donor:matched 0.72 [0.55;0.95] 0.0222663donor:mismatched 0.27 [0.13;0.57] 0.0006294stage:early � � �stage:intermediate 1.8 [1.37;2.46] < 0.0001stage:advanced 3.1 [2.47;4.02] < 0.0001timedxtx 0.99 [0.98;1] 0.0219938

E.g., The risk of relapse was estimated as 1.8 times higher for disease stage

intermediate compared to disease stage early.

11 / 28

Does this model �t?

Comparison with common alternatives:

I Combination of cause-speci�c Cox regressions (Formula I)

I Fine-Gray regression model (Formula II: di�erent link function)

I Flexible absolute risk regression: allow time-dependentcovariate e�ects βk(t)

Focus: the validity of the model for prediction

I Personalized: re-classi�cation of predicted probabilities

I Calibration plot: distance between predicted expectedprobabilities

I Brier score: mean squared error for predicted probabilities

12 / 28

Comparison of predicted probabilities

Predicted risk of relapse within 3 year after transplantation


Gra

y−F

ine

regr

essi

on

0 % 25 % 50 %

0 %

25 %

50 %

●

●

●

●●

●

●●

●●

●●●●●●

●

●●

●●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●●●

●

●

●●

●

●●

●

●

●

●●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●●●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●●●

●

●

●

●●●●

●●

●

●●●

●●

●

●●

●

●

●

●

●

●●

●●●

●

●

●

●

●

●

●●●

●

●

●●●●

●●

●

●

●

●

●

●●

●

●

●●●

●

●

●

●

●●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●●

●

●●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●●

●●

●

●

●

●●

●

●

●●

●●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●●●

●

●

●

●

●●●●

●●●●

●

●

●

●

●●

●●

●

●

●

●●

●●

●

●●

●●

●●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●●●

●

●●

●

●●●

●

●

●

●

●

●●

●

●

●

●●●●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●

●

●

●

●

●

●●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●●●

●

●

●

●●

●

●

●

●●

●

●●

●

●

●

●●

●●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●●●

●

●

●●●

●

●

●

●●

●●

●

●●

●

●

●

●

●

●

●●

●

●●●●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●●

●

●

●

●●

●

●●●●

●

●●

●

●●

●●

●

●

●●●

●

●●●

●

●

●●●●

●●

●●

●●●●

●●

●

●

●

●●

●

●

●

●●

●●●

●

●●●●

●

●

●

●●●●●

●●●

●●

●●

●●

●

●

●

●●●

●

●

●●

●

●

●●

●

●

●

●●●●

●●●

●

●

●

●●●●●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●●

●

●

●●

●

●

●●●

●●

●●●

●

●

●

●●

●

●

●

●●

●●

●●

●

●

●

●●

●

●

●●

●●●

●

●

●●

●

●

●●●

●●

●

●

●

●

●●●

●

●●●

●

●

●

●

●

●

●

●●

●

●

●

●●●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●●

●●

●

●

●

●

●

●

●●

●●

●

●●

●

●

●●

●●

●●

●

●

●

●

●●

●

●

●●●

●●

●●●

●

●●

●

●

●

●

●

●

●

●

●

●

●●●●

●

●

●

●

●●

●●

●

●

●●

●

●

●

●

●

●●●

●

●

●

●

●

●●●

●

●

●

●●

●

●

●

●

●

●●

●

●

●●

●●●

●

●

●

●

●

●●

●

●●●●

●

●

●●●●

●

●●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●●●

●

●●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●●●

●

●

●

●

●

●●●

●

●

●

●

●

●●●●

●

●

●

●

●●

●●●

●●

●

●

●●

●

●●●●

●

●●●●

●

●●●

●

●

●

●

●

●●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●●●

●

●●

●

●●●●●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●●

●

●●

●●

●

●

●

●●●

●

●

●●●

●

●

●

●

●

●

●

●

●●

●●

●

●

●●

●

●●

●●

●

●

●

●

●

●●●

●

●●

●

●●

●

●●●

●

●

●

●●

●●

●

●

●

●

●

●●

●●

●

●●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●●

●

●

●●●

●●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●●

●

●

●

●●●

●

●

●

●

●●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●●●●

●

●

●●

●

●●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●●

●

●

●

●

●●

●

●

●

●●

●

●●

●

●

●●●

●

●●

●

●●

●

●●●●

●

●●●

●

●

●

●●●●

●

●

●

●

●

●

●

●●●●

●

●

●●

●

●●

●●

●

●

●

●●●

●

●●

●

●

●●

●

●

●

●

●

●●

●●

●●

●

●

●

●

●

●

●

●

●

●●

●●●●

●

●

●

●

●

●

●

●

●

●●

●●

●●

●

●

●●

●

●●

●

●

●

●●●

●

●●●

●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●●

●●

●

●●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●●

●●

●

●●●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●●

●

●

●●

●

●

●●

●

●

●

●

●●

●●●●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●●

●

●

●

●

●●

●

●

●●

●

●

●●●

●

●

●●

●●

●

●

●

●●●

●

●

●●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●●

●

●

●●

●●

●●●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●


Cau

se−

spec

ific

Cox

reg

ress

ion

0 % 25 % 50 %

0 %

25 %

50 %

●

●

●

●

●

●

●

●

●

●

●

●●●●●

●

●

●

●●

●

●

●

●

●

●●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●●

●

●●

●

●●

●

●

●

●●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●●●●

●

●

●●●●

●

●

●

●●●

●●

●

●●

●

●

●

●

●

●●●

●●

●

●●

●

●

●●●●

●● ●●●●●●●●

●

●●

●

●

●● ●●●

●

●

●

●

●●

●

●●

●●

●

●

●●

●

●

●

●

●

●● ●●●

●

●

●

●

●

●

●●

●●

●●

●● ●●

●

●

●

●●

●

●

●●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●●

●

●

●●●●●

● ●

●

●

●

●

●

●

●

●

●●

●

●

●●●●

●

●

●

●

●●●●●●●●●

●●

●

●● ●●

●

●

●

●●

●●

●●●

●●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●● ●●●●●

●●

●

●●

●

●

●

●

●

●

●●

●

●

●

●●●●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●

●

●

●

●

●

●

●●

●

●●●

●

● ●

●●

●

●

●

●●

●

●●

●

●

●●

●●

●

●●

●●

●●●

●

●

●

●●●

●

●

●●●

●●●

●

●

●

●

●●

●

●●●

●

●●

●

●

●

●

●

●●

●●

●

●

●

●

●

●●●

●●●

●

●

●●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●●

●

●●●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●● ●●●●● ●●●

●●

●

●

●●

●●

●●

●

● ●●●● ●●●

●

●

●●●● ●●●● ●●●● ●

●

●

●

●

●●

●●

●

●●

●

●

●

● ●●●●

●

●●

●●●●●

●●●

●

●

●●

●●● ●● ●●

●

●

●●●

●

●

●●

●

●

●●●●●●

●● ●

●

● ●●●●●●●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●● ●●

●

●

●●

●

●

●●●

●●●●●

●

●

●●●

●

●

●

●●

●●

●

●

●●

●●●

●

●

●●

●●●

●●

●●

●

●

●●●

●●

●●

●

●

●

●

●

●

●●●

●

●

●● ●

●

●

●●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●●

●

●

●

●

●

●●

●●

●

●●

●

●

●●

●●

●●

●

●

●

●

●●

●

●

●●●

●●

●●●● ●●

●

●

●

●

●

●

●

●

●●

●●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●●●

●

●

●

●

●

●●●

●

●

●

●●

●

●

●

●

●

●●●

●

●●

●●●

●

●

●

●

●

●●

●

●

●●

●

●

●

●●●●

●

●

●

●

●

●

● ●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●●●

●

●●

●

●●●●

●●

●

●

●

●●●●

●

●

●

●

●●

●●●

●●●

●

●●

●

●●●●

●

●●●●

●

●●●

●

●

●

●●

●●●

●●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●●

●●●

●

●●

●

●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●●

●

●●

●●

●

●

●

●●●

●

●

●●●

●

●

●

●

●●

●

●

●●

●●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●●

●

●● ●

●

●●

●●

●●

●●

●●

●●

●

●

●●

●

●●

●●

● ●●●●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●●

●

●

●●●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●●●

●

●

●

●●●

●

●●

●

●●

●

●

●

●

●●

●

●●

●

●

●●

●

●

●●●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

● ●

●

●

●

●

●●

●●●

●

●●

●

●

●

●

●

●●

●

● ●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●●●●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●●●

●●

●

●

●

●

●●●●●

●

●●●

●

●

●

●●●●

●

●

●

●

●

●

●

●●●●

●

●

●●

●

●●

●●

●

●

●

●●●

●

●●

●

●

●●

●

●

●

●

●

●●

●●

●●

●

●

●●

●

●

●

●

●

●

●

●●●●●

● ●

●

●

●

●

●●

●●

●

●

●●

●

●

●●

●

●

●

●

●

●

●●

●

●

●●● ●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●●

●●

●

●●

●

●

●

●

●

●

●●●

●

●

●

● ●●

●●●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●●

●

●

●

●

●

●

●●●

●

● ●●●

●

●

●

●●●

●●

●

●●●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●●

●

●●

●

● ●●●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●


Tim

e−de

pend

ent e

ffect

s

0 % 25 % 50 %

0 %

25 %

50 %

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●●

●

●

●

●

●

●●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●●●●

●

●

●

●●●

●●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

● ●

●

●

●●

●

●

●●●

●●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●●●●

●

●●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

● ●

●●●

●

●

●

●

●

●

● ●

●

●

●

●

●

●●

●

●●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

● ●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●●

●

●

●●●

● ●●● ●●

●●

●

●

●●●

●

●●●

●

●

●●●

●

●

●

●●

●●

●●

●

●

●

●

●●●●

●●

●

●

●

●

●

●

●●●●

●

●

●

●●●●●

●●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●●

●●●

●

●

●●

●●●

●

●

●

●●●

●

●

●●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●●

●

●●

●

●●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●●

●

●

●●

●●●●

●

●●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●●●

●

●

●●

●

●

●●●

●

● ●

●

●

●

●

●●●

●●

●●●

●

●

●●

●

●

●

●●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

● ●

●

●

●

●

●●

●

●●

●

●

●

●

●

●●

●●●

●

●

●

●●●●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

● ●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●●

●

●● ●

●●

●●●

●

●

●

●

●●

●

●●

●

●

●

●●●

●

●

●●●

●

●

●

●

●

●

●●

●●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●● ●●●

●●

●

●

●●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●●

●

●●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●●● ●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●●

●

●

●●●

●

●●●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

● ●

●

●

●

●●●

●

●

●

●●

●

●●●

●●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●●

●

●

●

●●

●

●

●●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●●

●●●●●

●

● ●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

● ●●

●

●●

●

●

●

●

●

●

●●●

●

●

●

● ●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

Risk re-classi�cation plots13 / 28

Calibration curve

Ingredients:

I The event status indicator variable:

N(t) = 1{T ≤ t,D = 1}

I The risk prediction model:

r(t|X ) ∈ [0, 1]

I The risk group at p ∈ [0, 1]

Gr (t; p) = {x ∈ Rd : r(t|x) = p}

The calibration curve at time t:

p 7→ C (p, t, r) = E{N(t) | r(t|X ) = p}.= E{N(t) | X ∈ Gr (t; p)}

14 / 28

Estimation

To obtain the graph we need to estimate the expectation

E{N(t) | X ∈ Gr (t; p)}

Three often encountered practical problems arise:

I Right censoring: if patient i is not followed until time t, thestatus Ni (t) is unknown.

I Continuity: the size of the sets Gr (t; p) may be small and itmay happen that a set includes only a single patient.

I Generalizability: we would like to know if the model will bereliable for new patients, not those in the data set which wasused to specify and estimate the models.

15 / 28

Estimation approach

I Right censoring: if patient i is not followed until time t, thestatus Ni (t) is unknown:JACKNIFE PSEUDO-VALUES

I Continuity: the size of the sets Gr (t; p) may be small and itmay happen that a set includes only a single patient:NEAREST NEIGHBORHOOD SMOOTHING

I Generalizability: we would like to know if the model will bereliable for new patients, not those in the data set which wasused to specify and estimate the model:BOOTSTRAP-CROSSVALIDATION

16 / 28

Estimated calibration curve

Can,B(p, t, r) =1

n

n∑i=1

1

mi

∑b:i∈Vb

Ni(t)Kan(p, rb(t|Xi)) .

17 / 28


Can,B(p, t, r) =1

n

n∑i=1

1

mi

∑b:i∈Vb


I Ni (t) = jacknife pseudo value for event status at time t based onAalen-Johansen estimate of E(N(t))

I Kan(p,q)= smoothing kernel

I an = bandwidth

I B = number of bootstrap splits: Data = Lb + Vb

I rb = model �tted in learning sample Lb

I mi = the number of splits where patient i is in Vb

I rb(t,Xi ) = prediction for patient in validation sample Vb.

18 / 28


Can,B(p, t, r) =1

n

n∑i=1

1

mi

∑b:i∈Vb



I Kan(p,q) = smoothing kernel

I an = bandwidth





19 / 28


Can,B(p, t, r) =1

n

n∑i=1

1

mi

∑b:i∈Vb



I Kan(p,q)= smoothing kernel

I an = bandwidth





20 / 28

E�ect of censoring: 3 months after transplantation

Relapse


0 % 25 % 50 % 75 % 100 %

Pse

udo−

obse

rved

eve

nt s

tatu

s

0 %

25 %

50 %

75 %

100 % ● ●●●●● ●●●● ●● ● ●●● ●●●●

●●

●

●●●●

●● ●

●

●

● ●●●

●

●●● ●

●

● ●●●

●● ●

●●● ●

●

● ● ●● ●● ●●

● ● ●●

●●

●

●● ●●● ● ●●● ●

●

●● ●●● ●

●●

●●

●

●

●●● ●●●

●●

●●● ●●● ●●

●

●●● ●●●●● ●

●

●●● ●●●●●

● ●

● ●●● ●●●●●●●

● ●●

●● ●●●●●

●●● ●●

● ●●●

●

● ●●● ●●● ●●

●

● ●

●

● ●●●

●

●

●

●●● ●

● ●

●●●●● ●

●

●

●

● ●●

●

●● ●●● ● ●●●●●

●

●●

●

●

●●

●

●

●● ●●● ●●●●●● ●●● ●●

●

●●●● ●●●

●●

●

●●

●

●

●●●

●●●●●●

●

● ●●●● ●●

●

●

●

●●

● ●

●●● ●●●

●

●●● ●●●

●

●

●

●

●

●

●

●●●

● ●●●●

●

●● ●

●

●●●●

●●

●●● ●● ●●●

●●

●

●●●● ●●

●

●● ●●● ●●●

●

●●●●●● ●●●

●

●● ● ●

●

●●●●● ●●●●

●

●●

●●

●●●●●

●

●

●●

●●

●

●●● ●●●

●

●

●

● ● ●

●

●●●● ●●

●

●

●●

●

●

●●●

●

●●●

●

●

●

● ●

●

●●

●●

● ●●●● ●●●●●

●

●

●

●

●

●

●●

● ●

●

● ●● ●●●●●● ●●● ●● ●●●● ●●● ●●● ●●● ●● ● ●● ● ●●●● ●●● ●●●● ●●● ●●● ●● ●● ●● ●●●● ●●● ●●●●● ●●●● ● ●●●●● ● ●●● ● ●●●●● ●●●●●●●●● ●●●●●● ●●●●● ●●● ●●●●●● ● ●●●●●● ●●●●● ●● ●●●● ●●●●● ●● ● ●● ●● ● ●●●●● ●●●● ●●●● ●●●● ●●●●● ● ●●●●●●●●●●● ●●●●●●● ●●●● ●● ●● ●● ●● ● ●●● ● ●●●● ● ●● ●● ●● ●●● ●● ●●●● ●●●●●●●●● ●●● ● ●● ●● ●●●● ●●● ●● ●●●●● ●●●● ●●●●● ●● ●●● ●●●●●● ●●●●● ● ●●● ●●●●●● ● ●●● ●● ●● ●●●● ●●●● ●●● ● ●● ●●●● ●● ●●●●● ●●● ● ●●●●●●●● ●●●●●●●● ● ●● ●●●●●●● ●●●● ●● ●●●●● ●● ●●●●● ●●●● ●● ●●●●●● ●●● ●●● ●●●● ●● ●● ●●● ●●● ●●●●● ●● ●●●●●●● ●●● ●● ● ●●● ●● ●●●● ●●● ●● ●●● ●● ●● ●●●● ●●●●● ●●●● ● ●●● ●● ●●●● ●●●●●●●●●●●● ●●●● ●●●● ●●● ●●●● ●●● ●●●●● ●●●●●●●●●●●●● ●●●●●● ●●●●● ●●●●● ●●●● ●● ●●● ●●●● ●●●●●●● ●●●●●●●● ●●●●●●● ●● ●●● ●● ● ●●●● ●● ●●●● ●● ●●●● ●●● ●●●●●●● ●●●● ●●●● ●●●●●●●●●●●●●● ●● ●● ●●●● ●●●● ●●●●●●●●●● ●●●● ●●●●●● ●●●●● ●●●●●●●●● ●●●● ●●●●●●● ●● ●●●●●●●● ●●●● ● ●●●●●●●●●●● ●●●●●●●● ●●●●●●●●●●●●● ●● ●● ●●●●●●● ●●●●● ● ●● ●● ●●●● ●●●● ●● ●●●●●● ●●● ●● ●● ●●●●●●●●●● ●●●●●●●●●● ●●● ●●●●● ● ●●●●●●●●●●●● ●●●●●●●●● ●●● ●●●●●● ●●●● ●●●●●●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●●●●● ●● ●●●●●●●● ●●●●●●●●●●●● ●● ●●● ●●● ●● ●●●●● ●●●● ●●●●● ● ●● ●●●●●● ●●●●●●●●●●●● ●●●● ●●●●●●●●●●●●● ●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●● ●●● ●●●●●● ● ●●●●●●●●●● ●●● ●●●●● ●●● ●●●●●●●● ●●● ●●●●●●●●●●●●●●●● ●● ● ●● ●●● ●●●●●●●●● ●●● ●●● ●●●●●●●● ●●●●● ●● ●●●● ●● ●●● ●●●●●● ●●● ●●●●● ●● ●●●●●●● ●●●●●●●●●●●● ●●●●● ●●●●●●●●●●● ●● ●●● ●●● ●●●●●● ● ●●●●● ●●● ●●● ●●●●●● ●●●●● ●●●● ●●●●● ●●●●●●●●●●●● ●●●●●● ●● ●●●●●● ●●●●●●●●●●●●●●●●● ●●●●●●●●●●● ●●● ●●●●●●●●●●● ●● ●●●●● ●●●●●● ●

Death without relapse


0 % 25 % 50 % 75 % 100 %

Pse

udo−

obse

rved

eve

nt s

tatu

s

0 %

25 %

50 %

75 %

100 %

● ●●● ●●●●●●● ●●●● ●●● ● ●

●●

●

●●● ●

● ●●

●

●

●●● ●

●

●●●●

●

● ●●●

●● ●

● ●●●

●

●●●● ● ●●●

● ●●●

●●

●

● ● ●● ●●● ●● ●

●

● ●●● ●●

● ●

● ●

●

●

● ●●●●●

●●

● ● ●●● ●● ●

●

● ●● ●● ●● ●●

●

● ●● ● ●● ●●

●●

●● ●●● ●●● ●●

●

●●

●

● ●● ●● ●

●

● ●●●

●

●● ●●

●

●● ●●● ●●● ●

●

● ●

●

●●

●

●

●

●

●

●● ●●

● ●

●●

●

●●●

●

●

●

●● ●

●

●●● ● ●●●● ●● ●

●

●

●

●

●

●●

●

●

●●●● ●●●● ●● ●● ●● ● ●

●

●

● ●●● ●●

●

●

●

●●

●

●

●● ●

●● ●●●●

●

● ●● ● ●● ●

●

●

●

● ●

● ●

● ● ●●● ●

●

●●●● ●●

●

●

●

●

●

●

●

● ●

●

●●●●●

●

●●●

●

●●●●

●●

●● ●● ●● ●

●

● ●

●

●● ●●● ●

●

●●●●●● ●●

●

●● ●●● ●●● ●

●

● ●● ●

●

●●●● ●●●● ●

●

●●

●●

● ●● ●●

●

●

●●

● ●

●

●● ●● ●●

●

●

●

●●●

●

● ● ●●● ●

●

●

●●

●

●

● ● ●

●

●●●

●

●

●

●●

●

●●

●●

●●● ●●● ●●● ●

●

●

●

●

●

●

● ●

●●

● ●●●● ●●● ●● ●● ●●●●● ●●● ●● ●● ●● ● ●● ●●● ●● ● ●●● ● ●●●● ●●●●● ●● ●●● ●● ●●● ●●●● ●●● ●●●● ●●● ●●●● ●●●●●●●●● ● ●●●● ●●● ● ●●●●●● ●● ●● ●●●● ●● ● ●● ●●● ● ●●● ● ●●● ●● ●●●● ●●●● ● ●●● ●● ●● ●● ●●●● ●●● ●● ●●● ●●● ●●●● ●●●● ●●● ●● ● ● ●●●●●●● ●●●●●● ●●●●●● ●●●● ●●●●● ●●● ●● ●●● ● ●● ●●● ●●● ●●● ●●● ●● ● ● ●●●● ● ● ●●●● ●●● ● ●● ●●● ●●● ●●●●● ●● ●●● ●●●●● ●●● ●● ●●●● ●● ●●●●● ●● ●●● ● ●●● ● ●● ● ●● ● ●● ●● ●●● ●● ●●● ● ●● ●●● ●● ●● ●● ●● ●●●● ●● ●●● ●●●●●●●● ● ●●● ●●●●● ●●● ●●● ●●● ●● ●●●● ●●●●● ●●●● ●●● ●● ●● ●● ●● ● ●● ●● ● ●●●● ●●● ●● ●●● ● ●● ●● ●● ●● ● ●●● ●●●●●●●● ●●● ●● ●●●●●● ●● ●● ●●●● ● ●●● ●● ●●● ●●● ●●●● ●●●● ●● ●● ●● ● ●●● ●● ●●● ● ●● ●●● ● ●●● ● ●● ●●● ●●●●●● ●●●● ●●●● ●●●● ●●●●● ●●●● ●● ●● ●● ●●●● ●● ●●● ●●● ● ●● ●● ● ●●● ●●●● ● ●●●● ● ●●● ●●●● ●●● ●● ●● ●● ●●●●●●●●●● ●●● ●●● ●●●● ● ●●●● ● ●● ●● ●●●● ● ● ●● ●●● ●●●●●● ●●●● ● ●●● ● ●●● ●● ●●● ●●●●●● ●●● ●●● ●● ● ●● ● ●● ●●●● ● ●● ●●●●●● ● ●●● ●● ●●●●● ●●●●●●● ● ●●●● ● ●●●● ●● ●●●●● ●●●● ●● ●●●●●● ●●● ● ●●●●● ●●● ●●● ● ●● ●●● ● ● ●●● ●● ● ●●●●● ●●●● ● ●● ●● ●●●● ●● ●● ●● ●●●●●● ●●●●●● ●●●●●● ●●● ●●● ●●●● ●● ●● ●● ●●●● ●● ●●● ●● ●● ●● ●●●● ●● ●●●● ●● ●●● ● ●●●●●● ●●●● ● ●●●● ● ●●●● ●● ●●● ●● ● ●●● ●●●● ●●●●● ●●●● ●●●● ● ●●●● ●●●●● ●● ● ● ● ●● ●●● ●●● ●● ●●● ●● ●●● ●●●●●●●●●● ●● ●●● ●● ●●●●●● ●● ●●● ●●●●● ●●●● ●● ●● ●● ● ●●● ●●● ●●● ●●●●●● ●●●●●● ●●●● ● ●●● ●●● ●●●●●●●● ●●● ●● ●●● ● ●●● ● ● ●●● ●● ●● ● ●●● ●● ●●●●●●●●●● ●● ●●● ●●●●●● ●●● ●●● ●●●●●●● ●●● ●●● ●●●● ●●● ● ●●● ●●● ● ●● ●● ●●● ●●●● ●●●●● ● ●●●●●●● ●●● ●● ● ●●●●●● ●●●● ●●● ●● ●●●● ●● ●●● ●●●● ●● ●● ●● ●●●● ●●● ●● ●●● ●● ●● ●●● ●●● ●●● ●●● ●●●●●●●●● ● ●● ●● ●●●● ●●● ●●●●●● ●●●● ●●● ●●●●● ●●●●● ●●●● ●● ●●● ●●●● ●●●● ● ●● ●●● ●●●● ●●●●● ● ●●● ●● ●● ●● ●●● ●●●●● ●● ●

Figure: Risks predicted by two independent absolute risk regression

models, one for relapse and one for death without relapse.

21 / 28

E�ect of censoring: 1 year after transplantation

Relapse


0 % 25 % 50 % 75 % 100 %

Pse

udo−

obse

rved

eve

nt s

tatu

s

0 %

25 %

50 %

75 %

100 % ● ●●●●● ● ●●● ●● ● ●●● ●●●●

●●

●

● ●●●

●● ●

●

●

● ●●●

●

●●● ●

●

● ● ●●

●● ●

●● ● ●

●

● ● ●● ●● ● ●

● ● ●●

●●

●

●● ●●● ● ●●● ●

●

●● ●●● ●

●●

●●

●

●

●●● ●●●

●●

●●● ●●● ●●

●

● ●● ●● ●●● ●

●

● ●● ●● ●●●

● ●

● ●●● ●●● ●●●

●

● ●

●

● ● ●●●●

●

●●● ●

●

● ●●●

●

● ●●● ●●● ●●

●

● ●

●

● ●

●

●

●

●

●

●●● ●

● ●

●●

●

●● ●

●

●

●

● ●●

●

●● ●●● ● ●●●●●

●

●

●

●

●

●●

●

●

●● ●●● ●●●●● ● ●●● ●●

●

●

●●● ●●●

●

●

●

● ●

●

●

● ●●

●● ●●● ●

●

● ●● ●● ●●

●

●

●

●●

● ●

●●● ●● ●

●

●●● ●●●

●

●

●

●

●

●

●

●●

●

● ●●●●

●

● ● ●

●

●● ●●

● ●

●●● ●● ●●

●

● ●

●

●●●● ●●

●

●● ●●● ●●●

●

●●●●●● ●●●

●

●● ● ●

●

● ●●●● ●●●●

●

● ●

●●

●●●●●

●

●

●●

●●

●

●●● ●●●

●

●

●

● ● ●

●

●●●● ●●

●

●

●●

●

●

●●●

●

●●●

●

●

●

● ●

●

● ●

●●

● ●●●● ●●● ●●

●

●

●

●

●

●

●●

● ●

●

● ●

● ●

● ●●● ●

●

●● ●

● ●●

●● ●●●

●

●● ●

●

●

●

● ●

●

● ●

●

●●

●

●

●

● ● ●●

●

●

●

● ●●

●

●

●

●●

●

● ●●●● ●●● ●

●

●●

● ●●●

●

● ●

●

●

●●

● ●

●●

● ●

●

●●

●

●●

●

●● ●●

●● ●

●●● ●

● ●

●●●● ●

●

●

●

● ●

●

●●

●

●

●●

●●

●

●

●●●● ●

●

●

●●

●

●●

●

●●

●●

●

●

●

●

● ●

●

●● ●●

●●

●

●

●●

●● ●● ●

●

●●●

●

● ● ●●

● ●

●●●

●

●

●●

●

●

●

●

●● ●

●

● ●

●

●

● ●

●

●

●

●● ● ●

●●

●

●

●

●● ●

●

● ●

●

●

●

●●

●

●

● ●●

●●

● ●●

●

●

●

●●

●

●●

●

●

●

●

●●

●

●

●

● ●●

●

●

● ●

●

●

●

●

●

●●

●

●●●

●

●

●

●

●

●●

●

●●

●●

●

●

●

●

●

●

●

● ●

● ●●

● ●●

● ●

●

●

● ●

● ●

●

●●●●

●●

●● ●●● ●

●

● ●●●● ●

●

●

● ●●●

●

●

●

●

●●●

● ●

●●●

●

●● ●●

●

●● ●

●●

●●

●● ●

●

●

●

●

●

●●● ●●●●● ●●

●

●

●

●

● ●

●

●

●

●

● ●● ●●

●●

●●

● ●●●

●

● ●●

●

●

●●

●●

●

●

●

● ●

●

●●●

●

●

●

●

●

●●●

● ●●

●

●● ●

●

●

● ●● ●●

●

● ●

●

● ●● ●● ● ●● ●● ●●●● ● ●● ●● ●●●● ● ●●● ●● ●● ●● ●●●●●● ● ●● ● ●● ●●●● ●●●● ●● ● ●● ●● ●● ● ●●●●● ● ●● ● ●● ●●● ●● ●● ●● ●●●● ●●●●● ●● ●●● ●●● ● ●● ●●● ●●●● ●●●● ●●● ● ●●●● ●●● ●●●●● ●● ●● ●● ● ●● ● ●●●● ●● ●●●● ●● ●●●● ●●● ●●● ●●●● ●●● ● ●● ●● ●●● ●●● ●●●●● ●●● ●● ●● ●● ●● ●●● ● ●● ● ●●●●●●● ●●●● ●●● ●●● ● ●●●● ● ●●●●● ●● ● ●●● ● ●●●● ●● ● ●● ●●●●●●●● ●● ●● ● ●●● ●●●●● ●●● ●●●● ●●●● ●●● ●● ●●●●● ●●● ●● ●● ●●●● ● ●● ●●● ●● ● ●● ●● ●●●● ●●●● ●● ●●●● ●● ●●● ●● ●● ●●●● ●●●●●● ●●●● ●●●●●● ●●● ● ●● ●● ● ●● ●●● ●●● ●●● ● ●●● ●● ●●●● ●● ● ●● ●●●● ●●●● ●●●●● ●●●●●●● ●● ● ●●●●●● ●● ● ●● ●● ●●●●● ●●●● ●● ●●● ●●●●● ●●●●●●●●●● ●● ●● ●●● ●●● ●● ●●●●● ●● ●● ●●●●● ● ●● ●●●● ●● ●●● ●● ●●● ●●● ● ●●●● ●● ●● ●● ●●● ●● ●● ●● ●● ●● ●●●●●● ●●●●●● ●●● ●●●●●● ●●●● ●●● ●●● ●●● ● ●●●● ● ●●●●● ●●● ●●● ●● ●●● ●●●●●●●● ●●● ●● ●●●●●●● ●●● ● ●●● ●● ● ●● ●● ● ●●●● ●●●●● ●●● ●●● ●●●● ● ●●● ●●●● ● ●● ●●●● ●● ●●● ●● ●● ●● ●●● ●●●●● ●● ●●●● ● ●● ●●●● ●●●● ● ●● ● ●● ●●● ● ●●●●●● ●●●● ●● ●●● ●●● ●●●●●● ● ●●● ● ● ●●● ●●● ●●● ●●● ●●●●● ●● ●● ●●● ●● ●●● ●● ● ●●●● ●● ●●● ●●● ●● ● ●●●●● ●● ● ●●●● ●●●● ●●● ●●● ●●●●● ● ●●●●● ●●● ●●● ●● ●●●●●● ●● ●●●●● ●●●●● ● ●



0 % 25 % 50 % 75 % 100 %

Pse

udo−

obse

rved

eve

nt s

tatu

s

0 %

25 %

50 %

75 %

100 %

● ●●● ●●●●●●● ●●●● ●●● ● ●

●●

●

●●● ●

● ●●

●

●

●●● ●

●

●● ●●

●

● ●●●

●● ●

● ●●●

●

●●●● ● ●●●

● ●●●

● ●

●

● ● ●● ●●● ●● ●

●

● ●●● ●●

● ●

● ●

●

●

● ●●●●●

●●

● ● ●●● ●● ●

●

● ●● ●● ●● ●●

●

● ●● ● ●● ●●

●●

●● ●●● ●●● ●●

●

●●

●

● ●● ●● ●

●

● ●●●

●

●● ●●

●

●● ●●● ● ●● ●

●

● ●

●

●●

●

●

●

●

●

●● ●●

● ●

●●

●

●●●

●

●

●

●● ●

●

● ●● ● ●●●● ●● ●

●

●

●

●

●

●●

●

●

●●●● ●●●● ●● ●● ●● ● ●

●

●

● ●●● ●●

●

●

●

●●

●

●

●● ●

●● ● ●●●

●

● ●● ● ●● ●

●

●

●

● ●

● ●

● ● ● ●● ●

●

● ●●● ●●

●

●

●

●

●

●

●

● ●

●

●● ●●●

●

●●●

●

●●●●

●●

● ● ●● ●● ●

●

● ●

●

●● ●●● ●

●

●●●●●● ●●

●

●● ●●● ●●● ●

●

● ●● ●

●

●●●● ●●●● ●

●

●●

●●

● ●● ● ●

●

●

●●

● ●

●

●● ●● ●●

●

●

●

●●●

●

● ● ●●● ●

●

●

●●

●

●

● ● ●

●

●●●

●

●

●

●●

●

●●

●●

●●● ●●● ●●● ●

●

●

●

●

●

●

● ●

●●

●

●●

●●

●●● ●●

●

● ●●

●●●

● ●● ●●

●

● ●●

●

●

●

●●

●

●●

●

●●

●

●

●

●●● ●

●

●

●

● ●●

●

●

●

●●

●

●● ●●●● ●●●

●

●●

● ●●●

●

●●

●

●

●●

●●

●●

●●

●

●●

●

● ●

●

● ● ●●

●● ●

● ●● ●

● ●

●●● ●●

●

●

●

●●

●

● ●

●

●

● ●

●●

●

●

●●●● ●

●

●

● ●

●

●●

●

● ●

● ●

●

●

●

●

● ●

●

● ●● ●

●●

●

●

● ●

●●● ●●

●

● ●●

●

●● ● ●

●●

●●●

●

●

●●

●

●

●

●

●●●

●

●●

●

●

● ●

●

●

●

●● ●●

● ●

●

●

●

● ● ●

●

●●

●

●

●

● ●

●

●

●● ●

●●

● ● ●

●

●

●

● ●

●

●●

●

●

●

●

● ●

●

●

●

● ●●

●

●

●●

●

●

●

●

●

●●

●

●●●

●

●

●

●

●

● ●

●

●●

●●

●

●

●

●

●

●

●

●●

● ● ●

●● ●

●●

●

●

● ●

●●

●

● ●●

●

●●

●

●● ● ●●

●

●● ●

● ●●

●

●

●● ●

●

●

●

●

●●●

●

●●

●

●●●

●

● ● ●

●●

●●

●●● ●

●● ●

●●

●●

●●

● ●●●● ●●●●

●

●

●●

● ●●

●

●

● ●● ●●

●

●

●

●● ●●

● ●

●●●

●

●

●

●●

●

●●

●

●●

●

● ●●

●

● ● ●●●

●

●●●●●●● ●●

●

●●

●

●

●

●●

●

●● ●● ●●●● ● ● ●● ●● ●●● ●●● ●●●● ●●●● ●● ●● ●● ● ●●● ●● ●●● ● ●● ●●● ● ●●● ● ●● ●●● ●● ●●●● ●● ● ● ●●● ● ●●●● ●●●●● ●●●● ●● ●● ●● ●●●● ●● ●●● ●●● ● ●● ●● ● ●●● ●●●● ● ●●●● ● ●●● ●●●● ●●● ●● ●● ●● ●●● ●● ●●●●● ●●● ●●● ●●●● ● ●●●● ● ●● ●● ●●●● ● ● ●● ●●● ●●●●●● ●●●● ● ●●● ● ●●● ●● ●●● ●●●●●● ●●● ●●● ●● ● ●● ● ●● ●●●● ● ●● ●●●● ●● ● ●●● ●● ●●●●● ●● ●●●● ● ● ●●●● ● ●●●● ●● ●●●●● ●●●● ●● ●●● ●●● ●●● ● ●● ● ●● ●●● ●●● ● ●● ●●● ● ● ●●● ●● ● ●●●●● ●●●● ● ●● ●● ●● ●● ●● ●● ●● ●●●●●● ●● ● ●●● ●● ●●●● ●●● ●●● ●●● ● ●● ●● ●● ●●●● ●● ●●● ●● ●● ●● ●●●● ●● ●●●● ●● ●●● ● ●●●●●● ●●●● ● ●●●● ● ●●●● ●● ●●● ●● ● ●●● ●●●● ●●●● ● ●●●● ●● ●● ● ●●●● ●●●●● ●● ● ● ● ●● ●●● ●●● ●● ●●● ●● ●●● ●●●●●●● ●●● ●● ●●● ●● ●●●● ●● ●● ●●● ●●●●● ●●●● ●● ●● ●● ● ●●● ●●● ●●● ● ●●●●● ●●●●●● ●●●● ● ●●● ●●● ●●●● ● ●● ● ●●● ●● ●●● ● ●●● ● ● ●●● ●● ●● ● ●●● ●● ●● ●●● ●●●●● ●● ●●● ●●● ●●● ● ●● ●●● ●●●●●●● ●●● ●●● ●●●● ●●● ● ●●● ●●● ● ●● ●● ●●● ●●●● ● ●●●● ● ●●●●●●● ●● ● ●● ● ●●●●●● ●●●● ●●● ●● ●●●● ●● ●● ● ●●●● ●● ●● ●● ●●●● ●●● ●● ●●● ●● ●● ●●● ●●● ●●● ●●● ●● ●●●● ●●● ● ●● ●● ●●●● ●●● ●●●●●● ●●●● ●●● ●●●●● ●●●●● ●●●● ●● ●●● ●●●● ● ●●● ● ●● ●●● ●●●● ●●●●● ● ●●● ●● ●● ●● ●●● ●●● ●● ●● ●



22 / 28

E�ect of censoring: 3 years after transplantation

Relapse


0 % 25 % 50 % 75 % 100 %

Pse

udo−

obse

rved

eve

nt s

tatu

s

0 %

25 %

50 %

75 %

100 % ● ●●● ●● ● ●● ● ●● ● ●●● ●●●●

●●

●

● ●●●

●● ●

●

●

● ●● ●

●

●●● ●

●

● ● ●●

●● ●

●● ● ●

●

● ● ●● ●● ● ●

● ● ●●

●●

●

●● ●●● ● ●●● ●

●

●● ●●● ●

●●

●●

●

●

●●● ●●●

●●

●●● ●●● ●●

●

● ●● ●● ●●● ●

●

● ●● ●● ●●●

● ●

● ●●● ●●● ●●●

●

● ●

●

● ● ●●●●

●

●●● ●

●

● ●●●

●

● ●●● ●●● ●●

●

● ●

●

● ●

●

●

●

●

●

●●● ●

● ●

●●

●

●● ●

●

●

●

● ●●

●

●● ●●● ● ●●●●●

●

●

●

●

●

●●

●

●

●● ●●● ●●●●● ● ●●● ●●

●

●

●● ● ●●●

●

●

●

● ●

●

●

● ●●

●● ●●● ●

●

● ●● ●● ●●

●

●

●

●●

● ●

●●● ●● ●

●

●●● ●●●

●

●

●

●

●

●

●

●●

●

● ●●●●

●

● ● ●

●

●● ●●

● ●

●●● ●● ●●

●

● ●

●

●●● ● ●●

●

●● ●●● ●●●

●

●●●●●● ●●●

●

●● ● ●

●

● ●●●● ●●●●

●

● ●

●●

●● ●●●

●

●

●●

●●

●

●●● ●●●

●

●

●

● ● ●

●

● ●●● ●●

●

●

●●

●

●

●●●

●

●●●

●

●

●

● ●

●

● ●

●●

● ●●●● ●●● ●●

●

●

●

●

●

●

●●

● ●

●

● ●

● ●

● ●●● ●

●

●● ●

● ●●

●● ●●●

●

●● ●

●

●

●

● ●

●

● ●

●

●●

●

●

●

● ● ●●

●

●

●

● ●●

●

●

●

●●

●

● ●●●● ●● ● ●

●

●●

● ●●●

●

● ●

●

●

●●

● ●

●●

● ●

●

●●

●

●●

●

●● ●●

●● ●

●●● ●

● ●

●●●● ●

●

●

●

● ●

●

●●

●

●

●●

●●

●

●

●● ●● ●

●

●

●●

●

●●

●

●●

●●

●

●

●

●

● ●

●

●● ●●

●●

●

●

●●

●● ●● ●

●

●●●

●

● ● ●●

● ●

●●●

●

●

●●

●

●

●

●

●● ●

●

● ●

●

●

● ●

●

●

●

●● ● ●

●●

●

●

●

●● ●

●

● ●

●

●

●

●●

●

●

● ●●

●●

● ●●

●

●

●

● ●

●

●●

●

●

●

●

●●

●

●

●

● ●●

●

●

● ●

●

●

●

●

●

●●

●

●●●

●

●

●

●

●

●●

●

●●

●●

●

●

●

●

●

●

●

● ●

● ●●

● ● ●

● ●

●

●

● ●

● ●

●

●●●

●

●●

●

● ●●● ●

●

● ●●

●● ●

●

●

● ●●

●

●

●

●

●

●●

●

● ●

●

●●

●

●

● ●●

●

●

● ●

●●

●●

●● ●

●

●

●

●

●

●

●● ●● ●●● ●●

●

●

●

●

● ●

●

●

●

●

● ●● ●

●

●

●

●●

● ●

●●

●

● ●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

● ●●

●

●● ●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

● ●

●● ●

●

●

●●●

● ●

●

●

● ●●

●●

●

●

●● ●

●

●

● ●

●

●

●●

●

●● ● ●

●

●

●

● ●●

●● ●

●●

●●

●

●

●●●

●●●

● ●●●

●

●●

●

● ●●●

●●● ●● ●● ●●

●

●●● ●●

●● ●

●

● ●

●

● ●

●

●● ●● ●

●

● ●●● ●

●●●● ●●● ● ●

●●● ●

●

●

●●

●●● ●

●

●● ●● ●●

●

●

●

●●● ●● ●●● ● ●●

●

● ●● ●●●●●

●

●

● ●●●

●

●

●

●

●●●

●

●● ●●

● ●

●●●●●

●● ●●

●● ●● ●● ●●● ● ●● ● ●●●

●

●●

● ●

●●● ●●● ●●● ● ●●●● ● ●

●●

●● ●● ●

●

●● ●

●●

●

●

●● ● ●●●

● ●

●●●●

●

●

●

●

● ●

●●● ●●●●● ●●● ●

●

●● ●●●● ●●● ●● ●

●

●●● ●●● ●● ●● ●●●●

●

●● ●●● ●● ● ●● ●● ●●●● ●●●● ●●

●

●●● ●● ●●● ●● ●● ●●●● ● ●●●●● ●●●● ●●●●●● ●●● ●

●

● ●● ●

●

● ●●● ●●● ●●● ● ●●● ●● ●●●● ●● ● ●● ●●●● ●●●● ●

●

●●● ●●●●●●● ●● ● ●●●●●● ●● ● ●● ●● ●●●●● ●●●● ●● ●●● ●●●●● ●●●●●●●●●● ●● ●● ●●● ●●● ●● ●●●●● ●● ●● ●●●●● ● ●● ●●●● ● ● ●●● ●● ●●● ●●● ● ●●●● ●● ●● ●● ●●● ●● ●● ●● ●● ●● ●●●●●● ●●●●●● ● ●● ●●●●●● ●●●● ●●● ●●● ●●● ● ●●●● ● ●●●●● ●●● ● ●● ●● ●●● ●●●● ●●●● ●●● ●● ●●●●●●● ●●● ● ●●● ●● ● ●● ●● ● ●●●● ●●●●● ●●● ●●● ●●●● ● ●●● ●● ●● ● ●● ●●●● ●● ●●● ●● ●● ●● ●●● ●●●●● ●● ●●●● ● ●● ●● ●● ●●●● ● ●● ● ●● ●●● ● ●●● ●●● ●●●● ●● ●●● ●●● ●●●●●● ● ●●● ● ● ●●● ●●● ●●● ●●● ●●●●● ●● ●● ●●● ●● ●●● ●● ● ●●●● ●● ●●● ●●● ●● ● ●●●●● ●● ● ●● ●● ●●●● ●●● ●●● ●●●●● ● ●●●●● ●●● ●●● ●● ●●● ●●● ●● ●● ●●● ● ●●●● ● ●



0 % 25 % 50 % 75 % 100 %

Pse

udo−

obse

rved

eve

nt s

tatu

s

0 %

25 %

50 %

75 %

100 %

● ●●● ●●●●●●● ●●●● ●●● ● ●

●●

●

●●● ●

● ●●

●

●

●●● ●

●

●● ●●

●

● ●●●

●● ●

● ●●●

●

●●●● ● ●●●

● ●●●

● ●

●

● ● ●● ●●● ●● ●

●

● ●●● ●●

● ●

● ●

●

●

● ●●●●●

●●

● ● ●●● ●● ●

●

● ●● ●● ●● ●●

●

● ●● ● ●● ●●

●●

●● ●●● ●●● ●●

●

●●

●

● ●● ●● ●

●

● ●●●

●

●● ●●

●

●● ●●● ● ●● ●

●

● ●

●

●●

●

●

●

●

●

●● ●●

● ●

●●

●

●●●

●

●

●

●● ●

●

● ●● ● ●●●● ●● ●

●

●

●

●

●

●●

●

●

●●●● ●●●● ●● ●● ●● ● ●

●

●

● ●●● ●●

●

●

●

●●

●

●

●● ●

●● ● ●●●

●

● ●● ● ●● ●

●

●

●

● ●

● ●

● ● ● ●● ●

●

● ●●● ●●

●

●

●

●

●

●

●

● ●

●

●● ●●●

●

●●●

●

●●●●

●●

● ● ●● ●● ●

●

● ●

●

●● ●●● ●

●

●●●●●● ●●

●

●● ●●● ●●● ●

●

● ●● ●

●

●●●● ●●●● ●

●

●●

●●

● ●● ● ●

●

●

●●

● ●

●

●● ●● ●●

●

●

●

●●●

●

● ● ●●● ●

●

●

●●

●

●

● ● ●

●

●●●

●

●

●

●●

●

●●

●●

●●● ●●● ●●● ●

●

●

●

●

●

●

● ●

●●

●

●●

●●

●●● ●●

●

● ●●

●●●

● ●● ●●

●

● ●●

●

●

●

●●

●

●●

●

●●

●

●

●

●●● ●

●

●

●

● ●●

●

●

●

●●

●

●● ●●●● ●●●

●

●●

● ●●●

●

●●

●

●

●●

●●

●●

●●

●

●●

●

● ●

●

● ● ●●

●● ●

● ●● ●

● ●

●●● ●●

●

●

●

●●

●

● ●

●

●

● ●

●●

●

●

●●●● ●

●

●

● ●

●

●●

●

● ●

● ●

●

●

●

●

● ●

●

● ●● ●

●●

●

●

● ●

●●● ●●

●

● ●●

●

●● ● ●

●●

●●●

●

●

●●

●

●

●

●

●●●

●

●●

●

●

● ●

●

●

●

●● ●●

● ●

●

●

●

● ● ●

●

●●

●

●

●

● ●

●

●

●● ●

●●

● ● ●

●

●

●

● ●

●

●●

●

●

●

●

● ●

●

●

●

● ●●

●

●

●●

●

●

●

●

●

●●

●

●●●

●

●

●

●

●

● ●

●

●●

●●

●

●

●

●

●

●

●

●●

● ● ●

●● ●

●●

●

●

● ●

●●

●

● ●●

●

●●

●

●● ● ●●

●

●● ●

● ●●

●

●

●● ●

●

●

●

●

●

● ●

●

●●

●

●●

●

●

● ● ●

●

●

●●

●●

● ●

●● ●

●

●

●

●

●

●

● ●●●● ●●●●

●

●

●

●

● ●

●

●

●

●

●● ●●

●

●

●

●●●●

● ●

●●●

●

●

●

●●

●

●●

●

●●

●

●●●

●

●●

●●

●

●

●●●●●

●● ●●

●

●●

●

●

●

●●

●

●●

●●

●

●●●● ● ●●

●● ●●

● ●

●

●

●●●

● ●

●●

● ●●

●

●●●●

●●●

●● ●●●

●

●●

●●●● ●●

● ●

●

●

●

●

● ●

●

●

●●

● ●

● ●●

●

●●

● ●

●●

● ●

●●●● ●●●●

●● ●●

●

● ●●●● ●●

●●●

●

●● ● ●●

●●

● ●●

● ●

●

●● ● ●●●

●

● ●●● ●●●● ●●●●

● ●● ●●

●

●● ●● ●●●●● ●●● ●●● ●●●● ● ●

●●

● ● ●● ●

●

●●●●

●

● ●● ●●● ●●●●●● ●

●

●● ●

●

●● ● ●●● ●● ●●● ●●●●●● ●●● ●●● ●● ● ●● ● ●● ●

●

●● ● ●● ●

●

●● ●● ● ●●● ●● ●●●●● ●●

●

●●● ● ● ●●●● ● ●●●● ●● ●●●●● ●●●● ●● ●●● ●●● ●●● ● ●● ● ●● ●●● ●●● ● ●● ●●● ● ● ●●● ●● ● ●

●

●●● ●●●● ● ●● ●● ●● ●● ●●

●

● ●● ●●●●●● ●● ● ●●● ●● ●●●● ●●●

●

●● ●●● ● ●● ●● ●● ●

●

●● ●● ●●● ●● ●● ●●

●

●●● ●● ●●●● ●● ●●● ● ●●●●●● ●●●● ● ●●●● ● ●●●● ●● ●●● ●● ● ●●● ●●●● ●●●● ● ●●●● ●● ●● ● ●●●● ●●●●● ●● ● ● ● ●● ●●● ●●● ●● ●●● ●● ●●● ●●●●●●● ●●● ●● ●●● ●● ●●●● ●● ●● ●●● ●●●●● ●●●● ●● ●● ●● ● ●●● ●●● ●●● ● ●●●●● ●●●●●● ●●●● ● ●●● ●●● ●●●● ● ●● ● ●●● ●● ●● ● ● ●●● ● ● ●●● ●● ●● ● ●●● ●● ●● ●●● ●●●●● ●● ●●● ●●● ●●● ● ●● ●●● ●●●●●●● ●●● ●●● ●●●● ●●● ● ●●● ●●● ● ●● ●● ●●● ●●●● ● ●●●● ● ●●●●●●● ● ● ● ●● ● ●●●●●● ●●●● ●●● ●● ●●●● ●● ●● ● ●●●● ●● ●● ●● ●●●● ●●● ●● ●●● ●● ●● ●●● ●●● ●●● ●●● ●● ●●●● ●●● ● ●● ●● ●●●● ●●● ●●●●●● ●●●● ●●● ●●●●● ●●●●● ●●●● ●● ●●● ●●●● ● ●●● ● ●● ●●● ●●●● ●●●●● ● ●●● ●● ●● ●● ●●● ●●● ●● ●● ●



23 / 28

E�ect of bandwidth: event= relapse, t=36 months

Calibration in the largebandwidth=1


0 % 25 % 50 % 75 % 100 %

Pse

udo−

obse

rved

eve

nt s

tatu

s

0 %

25 %

50 %

75 %

100 % ● ●●●●● ● ●●● ●● ● ●●● ●●●●

●●

●

● ●●●

●● ●

●

●

● ●●●

●

●●● ●

●

● ● ●●

●● ●

●● ● ●

●

● ● ●● ●● ● ●

● ● ●●

●●

●

●● ●●● ● ●●● ●

●

●● ●●● ●

●●

●●

●

●

●●● ●●●

●●

●●● ●●● ●●

●

● ●● ●● ●●● ●

●

● ●● ●● ●●●

● ●

● ●●● ●●● ●●●

●

● ●

●

● ● ●●●●

●

●●● ●

●

● ●●●

●

● ●●● ●●● ●●

●

● ●

●

● ●

●

●

●

●

●

●●● ●

● ●

●●

●

●● ●

●

●

●

● ●●

●

●● ●●● ● ●●●●●

●

●

●

●

●

●●

●

●

●● ●●● ●●●●● ● ●●● ●●

●

●

●●● ●●●

●

●

●

● ●

●

●

● ●●

●● ●●● ●

●

● ●● ●● ●●

●

●

●

●●

● ●

●●● ●● ●

●

●●● ●●●

●

●

●

●

●

●

●

●●

●

● ●●●●

●

● ● ●

●

●● ●●

● ●

●●● ●● ●●

●

● ●

●

●●●● ●●

●

●● ●●● ●●●

●

●●●●●● ●●●

●

●● ● ●

●

● ●●●● ●●●●

●

● ●

●●

●●●●●

●

●

●●

●●

●

●●● ●●●

●

●

●

● ● ●

●

●●●● ●●

●

●

●●

●

●

●●●

●

●●●

●

●

●

● ●

●

● ●

●●

● ●●●● ●●● ●●

●

●

●

●

●

●

●●

● ●

●

● ●

● ●

● ●●● ●

●

●● ●

● ●●

●● ●●●

●

●● ●

●

●

●

● ●

●

● ●

●

●●

●

●

●

● ● ●●

●

●

●

● ●●

●

●

●

●●

●

● ●●●● ●●● ●

●

●●

● ●●●

●

● ●

●

●

●●

● ●

●●

● ●

●

●●

●

●●

●

●● ●●

●● ●

●●● ●

● ●

●●●● ●

●

●

●

● ●

●

●●

●

●

●●

●●

●

●

●●●● ●

●

●

●●

●

●●

●

●●

●●

●

●

●

●

● ●

●

●● ●●

●●

●

●

●●

●● ●● ●

●

●●●

●

● ● ●●

● ●

●●●

●

●

●●

●

●

●

●

●● ●

●

● ●

●

●

● ●

●

●

●

●● ● ●

●●

●

●

●

●● ●

●

● ●

●

●

●

●●

●

●

● ●●

●●

● ●●

●

●

●

●●

●

●●

●

●

●

●

●●

●

●

●

● ●●

●

●

● ●

●

●

●

●

●

●●

●

●●●

●

●

●

●

●

●●

●

●●

●●

●

●

●

●

●

●

●

● ●

● ●●

● ●●

● ●

●

●

● ●

● ●

●

●●●

●

●●

●

● ●●● ●

●

● ●●

●● ●

●

●

● ●●

●

●

●

●

●

●●

●

● ●

●

●●

●

●

● ●●

●

●

● ●

●●

●●

●● ●

●

●

●

●

●

●

●● ●●●●● ●●

●

●

●

●

● ●

●

●

●

●

● ●● ●

●

●

●

●●

● ●

●●

●

● ●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●●

● ●●

●

●● ●

●

●

●●

●●

●

●

●●

●

● ●●

●

● ●

●● ●

●

●

●●●● ●

●

●● ●●●●

●

●

●● ●●

●

● ●

●

●

●●

●

●● ● ●●

●

●

● ●●

●● ●

●●

●●

●

●

●●●

●●●

● ●●●

●

●●

●

● ●●●

●●● ●● ●● ●●

●

●●● ●●

●●●

●

● ●

●

● ●

●

●● ●● ●

●

● ●●●●

●●●● ●●● ● ●

●●● ●

●

●

●●

●●● ●

●

●● ●● ●●

●

●

●

●●● ●● ●●●● ●●

●

●●● ●●●●●

●

●

●●●●

●

●

●

●

●●●

●

●● ●●

● ●

●●●●●●● ●

●●● ●● ●● ●●● ● ●● ● ●●●

●

●●

● ●

●●● ●●● ●●● ● ●●●● ● ●

●●

●● ●● ●

●

●● ●

●●

●

●

●● ● ●● ●

●●

●●●●

●

●

●

●

● ●

●●● ●●●●● ●●● ●

●

●● ●●●● ●●● ●● ●

●

●●● ●●● ●● ●● ●●●●

●

●● ●●● ●● ● ●● ●● ●●●● ●●●● ●●

●

●●● ●● ●●● ●● ●● ●●●● ●●●●●● ●●●● ●●●●●● ●●● ●

●

● ●● ●

●

● ●●● ●●● ●●● ● ●●● ●● ●●●● ●● ● ●● ●●●● ●●●● ●

●

●●● ●●●●●●● ●● ● ●●●●●● ●● ● ●● ●● ●●●●● ●●●● ●● ●●● ●●●●● ●●●●●●●●●● ●● ●● ●●● ●●● ●● ●●●●● ●● ●● ●●●●● ● ●● ●●●● ●● ●●● ●● ●●● ●●● ● ●●●● ●● ●● ●● ●●● ●● ●● ●● ●● ●● ●●●●●● ●●●●●● ●●● ●●●●●● ●●●● ●●● ●●● ●●● ● ●●●● ● ●●●●● ●●● ●●● ●● ●●● ●●●●●●●● ●●● ●● ●●●●●●● ●●● ● ●●● ●● ● ●● ●● ● ●●●● ●●●●● ●●● ●●● ●●●● ● ●●● ●●●● ● ●● ●●●● ●● ●●● ●● ●● ●● ●●● ●●●●● ●● ●●●● ● ●● ●●●● ●●●● ● ●● ● ●● ●●● ● ●●●●●● ●●●● ●● ●●● ●●● ●●●●●● ● ●●● ● ● ●●● ●●● ●●● ●●● ●●●●● ●● ●● ●●● ●● ●●● ●● ● ●●●● ●● ●●● ●●● ●● ●●●●●● ●● ● ●●●● ●●●● ●●● ●●● ●●●●● ● ●●●●● ●●● ●●● ●● ●●●●●● ●● ●●●●● ●●●●● ● ●

●

Localized calibrationbandwidth=0


0 % 25 % 50 % 75 % 100 %

Pse

udo−

obse

rved

eve

nt s

tatu

s

0 %

25 %

50 %

75 %

100 % ● ●●●●● ● ●●● ●● ● ●●● ●●●●

●●

●

● ●●●

●● ●

●

●

● ●●●

●

●●● ●

●

● ● ●●

●● ●

●● ● ●

●

● ● ●● ●● ● ●

● ● ●●

●●

●

●● ●●● ● ●●● ●

●

●● ●●● ●

●●

●●

●

●

●●● ●●●

●●

●●● ●●● ●●

●

● ●● ●● ●●● ●

●

● ●● ●● ●●●

● ●

● ●●● ●●● ●●●

●

● ●

●

● ● ●●●●

●

●●● ●

●

● ●●●

●

● ●●● ●●● ●●

●

● ●

●

● ●

●

●

●

●

●

●●● ●

● ●

●●

●

●● ●

●

●

●

● ●●

●

●● ●●● ● ●●●●●

●

●

●

●

●

●●

●

●

●● ●●● ●●●●● ● ●●● ●●

●

●

●●● ●●●

●

●

●

● ●

●

●

● ●●

●● ●●● ●

●

● ●● ●● ●●

●

●

●

●●

● ●

●●● ●● ●

●

●●● ●●●

●

●

●

●

●

●

●

●●

●

● ●●●●

●

● ● ●

●

●● ●●

● ●

●●● ●● ●●

●

● ●

●

●●●● ●●

●

●● ●●● ●●●

●

●●●●●● ●●●

●

●● ● ●

●

● ●●●● ●●●●

●

● ●

●●

●●●●●

●

●

●●

●●

●

●●● ●●●

●

●

●

● ● ●

●

●●●● ●●

●

●

●●

●

●

●●●

●

●●●

●

●

●

● ●

●

● ●

●●

● ●●●● ●●● ●●

●

●

●

●

●

●

●●

● ●

●

● ●

● ●

● ●●● ●

●

●● ●

● ●●

●● ●●●

●

●● ●

●

●

●

● ●

●

● ●

●

●●

●

●

●

● ● ●●

●

●

●

● ●●

●

●

●

●●

●

● ●●●● ●●● ●

●

●●

● ●●●

●

● ●

●

●

●●

● ●

●●

● ●

●

●●

●

●●

●

●● ●●

●● ●

●●● ●

● ●

●●●● ●

●

●

●

● ●

●

●●

●

●

●●

●●

●

●

●●●● ●

●

●

●●

●

●●

●

●●

●●

●

●

●

●

● ●

●

●● ●●

●●

●

●

●●

●● ●● ●

●

●●●

●

● ● ●●

● ●

●●●

●

●

●●

●

●

●

●

●● ●

●

● ●

●

●

● ●

●

●

●

●● ● ●

●●

●

●

●

●● ●

●

● ●

●

●

●

●●

●

●

● ●●

●●

● ●●

●

●

●

●●

●

●●

●

●

●

●

●●

●

●

●

● ●●

●

●

● ●

●

●

●

●

●

●●

●

●●●

●

●

●

●

●

●●

●

●●

●●

●

●

●

●

●

●

●

● ●

● ●●

● ●●

● ●

●

●

● ●

● ●

●

●●●

●

●●

●

● ●●● ●

●

● ●●

●● ●

●

●

● ●●

●

●

●

●

●

●●

●

● ●

●

●●

●

●

● ●●

●

●

● ●

●●

●●

●● ●

●

●

●

●

●

●

●● ●●●●● ●●

●

●

●

●

● ●

●

●

●

●

● ●● ●

●

●

●

●●

● ●

●●

●

● ●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●●

● ●●

●

●● ●

●

●

●●

●●

●

●

●●

●

● ●●

●

● ●

●● ●

●

●

●●●● ●

●

●● ●●●●

●

●

●● ●●

●

● ●

●

●

●●

●

●● ● ●●

●

●

● ●●

●● ●

●●

●●

●

●

●●●

●●●

● ●●●

●

●●

●

● ●●●

●●● ●● ●● ●●

●

●●● ●●

●●●

●

● ●

●

● ●

●

●● ●● ●

●

● ●●●●

●●●● ●●● ● ●

●●● ●

●

●

●●

●●● ●

●

●● ●● ●●

●

●

●

●●● ●● ●●●● ●●

●

●●● ●●●●●

●

●

●●●●

●

●

●

●

●●●

●

●● ●●

● ●

●●●●●●● ●

●●● ●● ●● ●●● ● ●● ● ●●●

●

●●

● ●

●●● ●●● ●●● ● ●●●● ● ●

●●

●● ●● ●

●

●● ●

●●

●

●

●● ● ●● ●

●●

●●●●

●

●

●

●

● ●

●●● ●●●●● ●●● ●

●

●● ●●●● ●●● ●● ●

●

●●● ●●● ●● ●● ●●●●

●

●● ●●● ●● ● ●● ●● ●●●● ●●●● ●●

●

●●● ●● ●●● ●● ●● ●●●● ●●●●●● ●●●● ●●●●●● ●●● ●

●

● ●● ●

●

● ●●● ●●● ●●● ● ●●● ●● ●●●● ●● ● ●● ●●●● ●●●● ●

●

●●● ●●●●●●● ●● ● ●●●●●● ●● ● ●● ●● ●●●●● ●●●● ●● ●●● ●●●●● ●●●●●●●●●● ●● ●● ●●● ●●● ●● ●●●●● ●● ●● ●●●●● ● ●● ●●●● ●● ●●● ●● ●●● ●●● ● ●●●● ●● ●● ●● ●●● ●● ●● ●● ●● ●● ●●●●●● ●●●●●● ●●● ●●●●●● ●●●● ●●● ●●● ●●● ● ●●●● ● ●●●●● ●●● ●●● ●● ●●● ●●●●●●●● ●●● ●● ●●●●●●● ●●● ● ●●● ●● ● ●● ●● ● ●●●● ●●●●● ●●● ●●● ●●●● ● ●●● ●●●● ● ●● ●●●● ●● ●●● ●● ●● ●● ●●● ●●●●● ●● ●●●● ● ●● ●●●● ●●●● ● ●● ● ●● ●●● ● ●●●●●● ●●●● ●● ●●● ●●● ●●●●●● ● ●●● ● ● ●●● ●●● ●●● ●●● ●●●●● ●● ●● ●●● ●● ●●● ●● ● ●●●● ●● ●●● ●●● ●● ●●●●●● ●● ● ●●●● ●●●● ●●● ●●● ●●●●● ● ●●●●● ●●● ●●● ●● ●●●●●● ●● ●●●●● ●●●●● ● ●

Kernel smootherautomatically selected

bandwidth=0.044


0 % 25 % 50 % 75 % 100 %

Pse

udo−

obse

rved

eve

nt s

tatu

s

0 %

25 %

50 %

75 %

100 % ● ●●●●● ● ●●● ●● ● ●●● ●●●●

●●

●

● ●●●

●● ●

●

●

● ●●●

●

●●● ●

●

● ● ●●

●● ●

●● ● ●

●

● ● ●● ●● ● ●

● ● ●●

●●

●

●● ●●● ● ●●● ●

●

●● ●●● ●

●●

●●

●

●

●●● ●●●

●●

●●● ●●● ●●

●

● ●● ●● ●●● ●

●

● ●● ●● ●●●

● ●

● ●●● ●●● ●●●

●

● ●

●

● ● ●●●●

●

●●● ●

●

● ●●●

●

● ●●● ●●● ●●

●

● ●

●

● ●

●

●

●

●

●

●●● ●

● ●

●●

●

●● ●

●

●

●

● ●●

●

●● ●●● ● ●●●●●

●

●

●

●

●

●●

●

●

●● ●●● ●●●●● ● ●●● ●●

●

●

●●● ●●●

●

●

●

● ●

●

●

● ●●

●● ●●● ●

●

● ●● ●● ●●

●

●

●

●●

● ●

●●● ●● ●

●

●●● ●●●

●

●

●

●

●

●

●

●●

●

● ●●●●

●

● ● ●

●

●● ●●

● ●

●●● ●● ●●

●

● ●

●

●●●● ●●

●

●● ●●● ●●●

●

●●●●●● ●●●

●

●● ● ●

●

● ●●●● ●●●●

●

● ●

●●

●●●●●

●

●

●●

●●

●

●●● ●●●

●

●

●

● ● ●

●

●●●● ●●

●

●

●●

●

●

●●●

●

●●●

●

●

●

● ●

●

● ●

●●

● ●●●● ●●● ●●

●

●

●

●

●

●

●●

● ●

●

● ●

● ●

● ●●● ●

●

●● ●

● ●●

●● ●●●

●

●● ●

●

●

●

● ●

●

● ●

●

●●

●

●

●

● ● ●●

●

●

●

● ●●

●

●

●

●●

●

● ●●●● ●●● ●

●

●●

● ●●●

●

● ●

●

●

●●

● ●

●●

● ●

●

●●

●

●●

●

●● ●●

●● ●

●●● ●

● ●

●●●● ●

●

●

●

● ●

●

●●

●

●

●●

●●

●

●

●●●● ●

●

●

●●

●

●●

●

●●

●●

●

●

●

●

● ●

●

●● ●●

●●

●

●

●●

●● ●● ●

●

●●●

●

● ● ●●

● ●

●●●

●

●

●●

●

●

●

●

●● ●

●

● ●

●

●

● ●

●

●

●

●● ● ●

●●

●

●

●

●● ●

●

● ●

●

●

●

●●

●

●

● ●●

●●

● ●●

●

●

●

●●

●

●●

●

●

●

●

●●

●

●

●

● ●●

●

●

● ●

●

●

●

●

●

●●

●

●●●

●

●

●

●

●

●●

●

●●

●●

●

●

●

●

●

●

●

● ●

● ●●

● ●●

● ●

●

●

● ●

● ●

●

●●●

●

●●

●

● ●●● ●

●

● ●●

●● ●

●

●

● ●●

●

●

●

●

●

●●

●

● ●

●

●●

●

●

● ●●

●

●

● ●

●●

●●

●● ●

●

●

●

●

●

●

●● ●●●●● ●●

●

●

●

●

● ●

●

●

●

●

● ●● ●

●

●

●

●●

● ●

●●

●

● ●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●●

● ●●

●

●● ●

●

●

●●

●●

●

●

●●

●

● ●●

●

● ●

●● ●

●

●

●●●● ●

●

●● ●●●●

●

●

●● ●●

●

● ●

●

●

●●

●

●● ● ●●

●

●

● ●●

●● ●

●●

●●

●

●

●●●

●●●

● ●●●

●

●●

●

● ●●●

●●● ●● ●● ●●

●

●●● ●●

●●●

●

● ●

●

● ●

●

●● ●● ●

●

● ●●●●

●●●● ●●● ● ●

●●● ●

●

●

●●

●●● ●

●

●● ●● ●●

●

●

●

●●● ●● ●●●● ●●

●

●●● ●●●●●

●

●

●●●●

●

●

●

●

●●●

●

●● ●●

● ●

●●●●●●● ●

●●● ●● ●● ●●● ● ●● ● ●●●

●

●●

● ●

●●● ●●● ●●● ● ●●●● ● ●

●●

●● ●● ●

●

●● ●

●●

●

●

●● ● ●● ●

●●

●●●●

●

●

●

●

● ●

●●● ●●●●● ●●● ●

●

●● ●●●● ●●● ●● ●

●

●●● ●●● ●● ●● ●●●●

●

●● ●●● ●● ● ●● ●● ●●●● ●●●● ●●

●

●●● ●● ●●● ●● ●● ●●●● ●●●●●● ●●●● ●●●●●● ●●● ●

●

● ●● ●

●

● ●●● ●●● ●●● ● ●●● ●● ●●●● ●● ● ●● ●●●● ●●●● ●

●

●●● ●●●●●●● ●● ● ●●●●●● ●● ● ●● ●● ●●●●● ●●●● ●● ●●● ●●●●● ●●●●●●●●●● ●● ●● ●●● ●●● ●● ●●●●● ●● ●● ●●●●● ● ●● ●●●● ●● ●●● ●● ●●● ●●● ● ●●●● ●● ●● ●● ●●● ●● ●● ●● ●● ●● ●●●●●● ●●●●●● ●●● ●●●●●● ●●●● ●●● ●●● ●●● ● ●●●● ● ●●●●● ●●● ●●● ●● ●●● ●●●●●●●● ●●● ●● ●●●●●●● ●●● ● ●●● ●● ● ●● ●● ● ●●●● ●●●●● ●●● ●●● ●●●● ● ●●● ●●●● ● ●● ●●●● ●● ●●● ●● ●● ●● ●●● ●●●●● ●● ●●●● ● ●● ●●●● ●●●● ● ●● ● ●● ●●● ● ●●●●●● ●●●● ●● ●●● ●●● ●●●●●● ● ●●● ● ● ●●● ●●● ●●● ●●● ●●●●● ●● ●● ●●● ●● ●●● ●● ● ●●●● ●● ●●● ●●● ●● ●●●●●● ●● ● ●●●● ●●●● ●●● ●●● ●●●●● ● ●●●●● ●●● ●●● ●● ●●●●●● ●● ●●●●● ●●●●● ● ●

Kernel smootherbandwidth=0.1


0 % 25 % 50 % 75 % 100 %

Pse

udo−

obse

rved

eve

nt s

tatu

s

0 %

25 %

50 %

75 %

100 % ● ●●●●● ● ●●● ●● ● ●●● ●●●●

●●

●

● ●●●

●● ●

●

●

● ●●●

●

●●● ●

●

● ● ●●

●● ●

●● ● ●

●

● ● ●● ●● ● ●

● ● ●●

●●

●

●● ●●● ● ●●● ●

●

●● ●●● ●

●●

●●

●

●

●●● ●●●

●●

●●● ●●● ●●

●

● ●● ●● ●●● ●

●

● ●● ●● ●●●

● ●

● ●●● ●●● ●●●

●

● ●

●

● ● ●●●●

●

●●● ●

●

● ●●●

●

● ●●● ●●● ●●

●

● ●

●

● ●

●

●

●

●

●

●●● ●

● ●

●●

●

●● ●

●

●

●

● ●●

●

●● ●●● ● ●●●●●

●

●

●

●

●

●●

●

●

●● ●●● ●●●●● ● ●●● ●●

●

●

●●● ●●●

●

●

●

● ●

●

●

● ●●

●● ●●● ●

●

● ●● ●● ●●

●

●

●

●●

● ●

●●● ●● ●

●

●●● ●●●

●

●

●

●

●

●

●

●●

●

● ●●●●

●

● ● ●

●

●● ●●

● ●

●●● ●● ●●

●

● ●

●

●●●● ●●

●

●● ●●● ●●●

●

●●●●●● ●●●

●

●● ● ●

●

● ●●●● ●●●●

●

● ●

●●

●●●●●

●

●

●●

●●

●

●●● ●●●

●

●

●

● ● ●

●

●●●● ●●

●

●

●●

●

●

●●●

●

●●●

●

●

●

● ●

●

● ●

●●

● ●●●● ●●● ●●

●

●

●

●

●

●

●●

● ●

●

● ●

● ●

● ●●● ●

●

●● ●

● ●●

●● ●●●

●

●● ●

●

●

●

● ●

●

● ●

●

●●

●

●

●

● ● ●●

●

●

●

● ●●

●

●

●

●●

●

● ●●●● ●●● ●

●

●●

● ●●●

●

● ●

●

●

●●

● ●

●●

● ●

●

●●

●

●●

●

●● ●●

●● ●

●●● ●

● ●

●●●● ●

●

●

●

● ●

●

●●

●

●

●●

●●

●

●

●●●● ●

●

●

●●

●

●●

●

●●

●●

●

●

●

●

● ●

●

●● ●●

●●

●

●

●●

●● ●● ●

●

●●●

●

● ● ●●

● ●

●●●

●

●

●●

●

●

●

●

●● ●

●

● ●

●

●

● ●

●

●

●

●● ● ●

●●

●

●

●

●● ●

●

● ●

●

●

●

●●

●

●

● ●●

●●

● ●●

●

●

●

●●

●

●●

●

●

●

●

●●

●

●

●

● ●●

●

●

● ●

●

●

●

●

●

●●

●

●●●

●

●

●

●

●

●●

●

●●

●●

●

●

●

●

●

●

●

● ●

● ●●

● ●●

● ●

●

●

● ●

● ●

●

●●●

●

●●

●

● ●●● ●

●

● ●●

●● ●

●

●

● ●●

●

●

●

●

●

●●

●

● ●

●

●●

●

●

● ●●

●

●

● ●

●●

●●

●● ●

●

●

●

●

●

●

●● ●●●●● ●●

●

●

●

●

● ●

●

●

●

●

● ●● ●

●

●

●

●●

● ●

●●

●

● ●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●●

● ●●

●

●● ●

●

●

●●

●●

●

●

●●

●

● ●●

●

● ●

●● ●

●

●

●●●● ●

●

●● ●●●●

●

●

●● ●●

●

● ●

●

●

●●

●

●● ● ●●

●

●

● ●●

●● ●

●●

●●

●

●

●●●

●●●

● ●●●

●

●●

●

● ●●●

●●● ●● ●● ●●

●

●●● ●●

●●●

●

● ●

●

● ●

●

●● ●● ●

●

● ●●●●

●●●● ●●● ● ●

●●● ●

●

●

●●

●●● ●

●

●● ●● ●●

●

●

●

●●● ●● ●●●● ●●

●

●●● ●●●●●

●

●

●●●●

●

●

●

●

●●●

●

●● ●●

● ●

●●●●●●● ●

●●● ●● ●● ●●● ● ●● ● ●●●

●

●●

● ●

●●● ●●● ●●● ● ●●●● ● ●

●●

●● ●● ●

●

●● ●

●●

●

●

●● ● ●● ●

●●

●●●●

●

●

●

●

● ●

●●● ●●●●● ●●● ●

●

●● ●●●● ●●● ●● ●

●

●●● ●●● ●● ●● ●●●●

●

●● ●●● ●● ● ●● ●● ●●●● ●●●● ●●

●

●●● ●● ●●● ●● ●● ●●●● ●●●●●● ●●●● ●●●●●● ●●● ●

●

● ●● ●

●

● ●●● ●●● ●●● ● ●●● ●● ●●●● ●● ● ●● ●●●● ●●●● ●

●

●●● ●●●●●●● ●● ● ●●●●●● ●● ● ●● ●● ●●●●● ●●●● ●● ●●● ●●●●● ●●●●●●●●●● ●● ●● ●●● ●●● ●● ●●●●● ●● ●● ●●●●● ● ●● ●●●● ●● ●●● ●● ●●● ●●● ● ●●●● ●● ●● ●● ●●● ●● ●● ●● ●● ●● ●●●●●● ●●●●●● ●●● ●●●●●● ●●●● ●●● ●●● ●●● ● ●●●● ● ●●●●● ●●● ●●● ●● ●●● ●●●●●●●● ●●● ●● ●●●●●●● ●●● ● ●●● ●● ● ●● ●● ● ●●●● ●●●●● ●●● ●●● ●●●● ● ●●● ●●●● ● ●● ●●●● ●● ●●● ●● ●● ●● ●●● ●●●●● ●● ●●●● ● ●● ●●●● ●●●● ● ●● ● ●● ●●● ● ●●●●●● ●●●● ●● ●●● ●●● ●●●●●● ● ●●● ● ● ●●● ●●● ●●● ●●● ●●●●● ●● ●● ●●● ●● ●●● ●● ● ●●●● ●● ●●● ●●● ●● ●●●●●● ●● ● ●●●● ●●●● ●●● ●●● ●●●●● ● ●●●●● ●●● ●●● ●● ●●●●●● ●● ●●●●● ●●●●● ● ●

24 / 28

E�ect of cross-validation


0 % 25 % 50 % 75 % 100 %

Pse

udo−

obse

rved

eve

nt s

tatu

s

0 %

25 %

50 %

75 %

100 % 1000 bootstrap cross−validation stepsSame data used twice

25 / 28

Comparison of models

Relapse (t=36 months)Same data twice


0 % 25 % 50 % 75 % 100 %

Pse

udo−

obse

rved

eve

nt s

tatu

s

0 %

25 %

50 %

75 %

100 %Absolute risk regression

Cause−specific Cox

Fine−Gray

Bootstrap cross−validationB=1000


0 % 25 % 50 % 75 % 100 %P

seud

o−ob

serv

ed e

vent

sta

tus

0 %

25 %

50 %

75 %

100 %Absolute risk regressionCause−specific CoxFine−Gray

26 / 28

Summary of calibration: Brier score

BS(t, r) = E{N(t)− r(t|X )}2

Apparent performance (same data twice)

time Reference riskRegression CauseSpeci�cCox FGR timevar

3 5.2 4.9 4.8 4.9 4.712 12.1 10.5 10.4 10.3 10.336 15.2 13.2 13.2 13.2 13.1

Crossvalidation performance (B=1000)

time Reference riskRegression CauseSpeci�cCox FGR timevar

3 5.2 5 4.9 4.9 512 12.1 10.7 10.6 10.6 10.736 15.3 13.5 13.5 13.5 13.5

I The lower the betterI The null model ignores the covariatesI Conclusion: All models are better than reference, but otherwise comparable

27 / 28

Summary and discussion

I The transformation model with log-link yields absolute riskratios adjusted for confounders.

I A calibration plot is a graphical tool to investigate thereliability of a prediction model.

I It can be estimated in the presence of competing risks andright censored data based on

I external validation dataI cross-validation

I The scatterplot of pseudo-values indicates the distribution ofthe predicted risks and the level of censoring.

I Estimating a calibration plot is as hard as estimating a densityand the choice of independent bandwidth allows the user tomanipulate the calibration plot.

28 / 28

calibration plots for risk prediction models in the presence of...

Documents