a new exact test to globally assess a population pk and/or pd model c. laffont & d. concordet
DESCRIPTION
ECOLE NATIONALE VETERINAIRE T O U L O U S E. A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet UMR 181 Physiopathologie et Toxicologie Expérimentales INRA, ENVT. Classical metrics for global model evaluation. Weighted residuals WRES NONMEM - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/1.jpg)
A new exact test to globally assess a population PK and/or PD model
C. Laffont & D. Concordet
UMR 181 Physiopathologie et Toxicologie ExpérimentalesINRA, ENVT
ECOLENATIONALEVETERINAIRET O U L O U S E
![Page 2: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/2.jpg)
• Weighted residuals
– WRES NONMEM
– CWRES Hooker et al. (2007) Pharm Res 24:2187
– PWRES Monolix
Classical metrics for global model evaluation
![Page 3: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/3.jpg)
Weighted residuals: difficulty in interpretation
Simulated data analysed with the correct model !Simulated data analysed with the correct model !
CWRES WRES
Karlsson & Savic, 2007,Clin Pharmacol Ther 82:17
Diagnosing Model Diagnostics
![Page 4: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/4.jpg)
Weighted residuals (WR) calculation:
)()( 2/1iiii YEYYVarWR
• Calculated from the vector of observations Yi in subject i and denoted here WRi
• Simulation (PWRES)
• FO/FOCE approximation of the model (WRES,CWRES)
• Simulation (PWRES)
• FO/FOCE approximation of the model (WRES,CWRES)
Expected distribution if the model is correct ?
I)N(0,~WRi
For linear Gaussian models, Yi distribution is Gaussian so:
For nonlinear models, WRi distribution is unknown !!!
![Page 5: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/5.jpg)
Recently proposed metric: NPDE
• Simulation-based approach
– compares at each time j the distribution of WRij with their predictive distribution according to the model
Brendel et al. (2006) Pharm Res 23: 2036
Expected distribution if the model is correct ?
NPDE are assumed to be independent and Gaussian:
I)N(0,~NPDE
Independence issue rightly discussed by the authors:
when NPDE are dependent, they are jointly not Gaussian
![Page 6: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/6.jpg)
New metric : GUD (Global Uniform Distance)
The purpose of this work was to propose:
• an exact test for global model evaluation
• an easy diagnostic graph with no subjective interpretation
Reject model with 5% risk= black line (data) outside the ring = black line (data) inside the ring
Do not reject model
![Page 7: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/7.jpg)
GUD calculation & testing
• Step 1Step 1
– As for calculation of NPDE, we compute for each subject i the vector of WRi
)(ˆ)(ˆ 2/1iiii YEYYarVWR
M (=2000) simulations (unbiased)
2/1)( iYVar is the Cholesky decomposition of the full variance matrix of Yi
WRi are decorrelated within subject i
![Page 8: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/8.jpg)
Decorrelation does not imply independence !!!
Decorrelation= independence
Decorrelation= independence
Decorrelation independence
Decorrelation independence
• Only true for linear Gaussian models
iiiii tY 21
• Case of nonlinear models
1 compt i.v. model
![Page 9: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/9.jpg)
We project WRi vector on R random vectors eir taken from a uniform distribution on the unit sphere
WRi vector (observations)
Unit sphere
r = 1… R
• Step 2:Step 2:– To handle data dependency, we use a recent random
projection method (See Cuesta-Albertos et al. (2007) for an application)
vector ei1
ProjectionProjectioni1i1
GUD calculation & testing
![Page 10: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/10.jpg)
Unit sphere
WRi vector (observations)
vector ei2ProjectionProjectioni2i2
GUD calculation & testing
• Step 2:Step 2:– To handle data dependency, we use a recent random
projection method (See Cuesta-Albertos et al. (2007) for an application)
r = 1… R
We project WRi vector on R random vectors eir taken from a uniform distribution on the unit sphere
![Page 11: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/11.jpg)
Each subject i
)(, WRiiri feWRProjection on R (=100) random directions
...
)(WRif
Subject i
)(1 WRf
Subject 1
GUD calculation & testing
cdf cdf
)(2 WRf
Subject 2
Random pdf independent between subjectsRandom pdf independent between subjects
Mixture of projection distributions
![Page 12: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/12.jpg)
GUD calculation & testing
• Step 3:Step 3:
– Compare this global cdf obtained for the sample to its distribution under H0 (i.e. correct model)
cdf cdf
SampleSample Simulations under HSimulations under H00
95% prediction region
95% prediction region
![Page 13: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/13.jpg)
For each replicate, we compute the maximal absolute distance
from mean cdf curve (GUD)
GUDGUD
= Global Uniform Distance= Global Uniform Distance
00
K cdf curves
Calculation of 95% prediction region under H0
K =(5000) replicates of the study design
Simulations under H0 with tested model
Simulations under H0 with tested model
mean cdfmean cdf
95%5%
5% of curves that 5% of curves that are the most distant are the most distant
from mean cdffrom mean cdf
5% of curves that 5% of curves that are the most distant are the most distant
from mean cdffrom mean cdf
![Page 14: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/14.jpg)
Calculation of 95% prediction region under H0
mean cdfmean cdf
Uniform region containing 95% of curves
Uniform region containing 95% of curves
= Global Uniform Distance= Global Uniform Distance
95% prediction region
For each replicate, we compute the maximal absolute distance
from mean cdf curve (GUD)
K cdf curves
K =(5000) replicates of the study design
Simulations under H0 with tested model
Simulations under H0 with tested model
GUDGUD0
95%
![Page 15: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/15.jpg)
0
GUDGUD
GUD test for your sample
Sample cdf
Sample
P valueP value
Do not reject modelDo not reject modelSimulations under H0Simulations under H0 True modelTrue model
= Global Uniform Distance= Global Uniform Distance
5%
95% prediction region
![Page 16: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/16.jpg)
0
5%
P valueP value
Sample
Wrong modelWrong model
Sample cdf
GUD test for your sample
Reject modelReject model
GUDGUD
= Global Uniform Distance= Global Uniform Distance
95% prediction region
![Page 17: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/17.jpg)
Sample
QQ ring diagnostic plot
Reject modelReject model
Sample
Do not reject model
Do not reject model
![Page 18: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/18.jpg)
PK model PK/PD model Linear model
1 compt i.v. bolus Sigmoidal Emax model (γ=4)
Exponential IIV on CL, V
Proportional res. error
Exponential IIV on Emax, EC50
Additive res. error
Additive IIV on 1, 2
Additive res. error
Performances under H0: GUD vs. other metrics
iiiii tY 21
• Simulations under H0 to evaluate the level of the tests 100 subjects i.i.d with 4 obs./ subject 5000 replications of study design
![Page 19: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/19.jpg)
• Weighted residuals & NPDE calculated in C++ Kolmogorov-Smirnov (KS) test to test for a N(0,1)
PK model
PK/PD model
Linear model
KS testfor N(0,1)
WRES
CWRES
PWRES
NPDE
100
57
66
4.0
100
13
62
7.5
4.9
4.9
4.5
4.6
GUD test GUD 5.1 5.0 4.9
Level of the tests under H0
Type I error (%) - nominal level = 5%
![Page 20: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/20.jpg)
95% prediction interval for 5000 replicates (Dvoretzky–Kiefer–Wolfowitz)
p value
The p-value should follow a uniform distribution under H0 !
Exp
ecte
d v
alu
e fr
om
u
nif
orm
dis
trib
uti
on
05.0)05.0(:Ex valuepProb
PP plotPP plot
Level of the tests under H0
![Page 21: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/21.jpg)
GUD testGUD test
PK model PK/PD model
p value p value
Exp
ecte
d v
alu
e fr
om
u
nif
orm
dis
trib
uti
on
95% prediction interval for 5000 replicates (Dvoretzky–Kiefer–Wolfowitz)
Good whateverGood whatever the level !the level !
Good whateverGood whatever the level !the level !
Level of the tests under H0
![Page 22: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/22.jpg)
NPDE: KS test for N(0,1) NPDE: KS test for N(0,1)
PK model
95% prediction interval for 5000 replicates (Dvoretzky–Kiefer–Wolfowitz)
p value p value
Exp
ecte
d v
alu
e fr
om
u
nif
orm
dis
trib
uti
on
or or of of type I errortype I error or or of of type I errortype I error
PK/PD model
Level of the tests under H0
![Page 23: A new exact test to globally assess a population PK and/or PD model C. Laffont & D. Concordet](https://reader037.vdocuments.us/reader037/viewer/2022110213/5681435b550346895dafd981/html5/thumbnails/23.jpg)
Conclusion• Poor performances of weighed residuals
• NPDE show much better performances but do not deal with the issue of data dependency within subjects
– Possible increase or decrease of type I error depending on model
• New test and graph based on GUD metric
– Encouraging results
– More work needed to evaluate this test under more complex conditions (different sampling times per subject, real-case data…)