evaluation of process models with monte carlo...
TRANSCRIPT
![Page 1: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/1.jpg)
1
Evaluation of Process Models with Monte Carlo Methods
John KauffmanFDA\CDER Division of Pharmaceutical Analysis
Sharmista ChatterjeeFDA\CDER\ONDQA
Jean-Marie GeoffroyTakeda Global Research & Development Center, Inc
IFPAC 2012
The findings and conclusions in this presentation have not been formally disseminated by the Food and Drug Administration and should not be construed
to represent any Agency determination or policy.
![Page 2: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/2.jpg)
2
Outline1. Role of Monte Carlo Simulation in
estimating uncertainty
2. Discussion of Case Studies
3. Conclusion
![Page 3: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/3.jpg)
3
Process Models and Design Space
• Process models map process parameters and input variables onto the product attributes.– Design space may be defined in terms of a process model
• Process models are often based on a relatively small number of experiments– Designed experiments tend to be parsimonious.– Parsimonious experiments may have large variances.
![Page 4: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/4.jpg)
4
Process Models and Probabilities
• Process models focus on predicting response means.– In a QbD paradigm, “What is needed is a multivariate
predictive distribution for the quality responses.”– “If we can quantify the entire (multivariate) predictive
distribution of the process quality responses as a function of [input variables and process parameters], then we can compute the probability of a future batch meeting the quality specifications.”
– (Peterson, Pharmaceutical Manufacturing, June 25, 2010)
![Page 5: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/5.jpg)
5
Advantages of Monte Carlo Simulation1. We make no assumptions concerning sources
of uncertainty or variable covariance.
2. We see the distribution of output variable values, not just a standard deviation.
3. Sensitivity analysis allows us to prioritize high risk input variables and improve process control.
![Page 6: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/6.jpg)
6
Monte Carlo Methods• Develop a mathematical model.
– The Process Model.
• Add random variables.– Replace quantities of interest with random numbers
selected from appropriate distribution functions that are expected to describe the variables.
• Monitor selected output variables.– Output variables become distributions whose properties are
determined by the model and the distributions of the random variables.
![Page 7: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/7.jpg)
7
Implementation of Monte Carlo Simulation
• Monte Carlo simulation offers a simple tool to explore influence of random variation in input parameters on multivariate predictive distributions
• In this presentation, we will examine the application of Monte Carlo simulation to a process model for two case studies
![Page 8: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/8.jpg)
8
Case Study 1: Background Information
Adapted from ICH IWG Training, October, 2010
Design Space defined on the basis of Multivariate DOE results, RSM design, 20+3 center pointsDOE Inputs:
-API particle size (API), log(D90) microns- Magnesium Stearate specific surface area (MgSt), cm2/g- Lubrication time (LubT), min- Tablet hardness (Hard), N
DOE Response:-% Dissolved in 20 minutes (Diss)
Control Strategy:-Predictions from regression model derived from DOE data implemented
as asurrogate for traditional dissolution testing
Specification: 80% dissolved in 20 min
![Page 9: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/9.jpg)
9
Case Study 1: DOE DataExp No Run Order API MgSt LubT Hard Diss
1 1 0.5 3000 1 60 101.242 14 1.5 3000 1 60 87.993 22 0.5 12000 1 60 99.134 8 1.5 3000 10 60 86.035 18 0.5 12000 10 60 94.736 9 1.5 12000 10 60 83.047 15 0.5 3000 1 110 98.078 2 0.5 12000 1 110 97.689 6 1.5 12000 1 110 85.47
10 16 0.5 3000 10 110 95.8111 20 1.5 3000 10 110 84.3812 3 1.5 12000 10 110 8113 10 0.5 7500 5.5 85 96.8514 17 1.5 7500 5.5 85 85.1315 19 1 3000 5.5 85 91.8716 21 1 12000 5.5 85 90.7217 7 1 7500 1 85 91.9518 4 1 7500 10 85 88.919 5 1 7500 5.5 60 92.3720 11 1 7500 5.5 110 90.9521 12 1 7500 5.5 85 91.9522 13 1 7500 5.5 85 90.8623 23 1 7500 5.5 85 89
![Page 10: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/10.jpg)
10
Case Study 1: Problem DescriptionObjective: Evaluate the response variance in the predictive model
• Predictive model derived from analysis of DOE data• Standard Error for Diss = 0.72• Design space defined in terms of ranges evaluated during DOE
Design Space:API: 0.5 - 1.5MgSt: 3000 – 12000LubT: 1 – 10Hard: 60 – 110
![Page 11: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/11.jpg)
11
Case Study 1: Approach for Uncertainty Estimation
Method: Uncertainty estimated using Monte Carlo simulation, using @RISK
Following cases were studied:- Case A: Assumed measurement uncertainty for each of the inputs (i.e. common cause variation), modeled as a normal distribution, no uncertainty assumed for the model coefficients- Case B: Assumed measurement uncertainty for each of the inputs and also for the model coefficients, modeled as a normal distribution
![Page 12: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/12.jpg)
12
Case Study 1: Results for Case A
For the following combination of input variables ( API: 1.5, MgSt: 12000, LubT: 10, Hard: 110) there is a 5.7% probability of not meeting the dissolution specification
Thus we have ~94% ‘assurance of quality’, and, there is potential for dissolution failure at edge of design space
78.5
79.0
79.5
80.0
80.5
81.0
81.5
82.0
82.5
83.0
83.5
Minimum = 78.7
Maximum = 83.2
Mean = 81.0
St. Dev = 0.60
![Page 13: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/13.jpg)
13
Case Study 1: Results for Case B
For the same combination of input variables as in Case A, assuming uncertainty in model coefficients (s= 1% of mean value) there is a 22.4% probability of not meeting the dissolution specification
22.4% 72.6% 5.0%
80.00 83.06
76 77 78 79 80 81 82 83 84 85 86
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
12 / Mean Disso
12 / Mean Disso
Minimum 76.1386Maximum 85.8299Mean 80.9722Std Dev 1.2632Values 10000
Hence, there is an increase in potential for dissolution failure at edge of design space
Minimum = 76.1
Maximum = 85.8
Mean = 81.0
St. Dev = 1.3
![Page 14: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/14.jpg)
14
Case Study #2: Estimating Model Coefficients by Monte Carlo Simulation
• Previous Studies: Estimated model coefficient standard deviations do not predict the observed response uncertainty.
• Can we use Monte Carlo simulation to provide better estimates of model coefficient standard deviations?– Solve for the model coefficients using Monte Carlo
simulation.– Model coefficients are given as distributions.
![Page 15: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/15.jpg)
15
Estimating Variance of Process Model Regression Coefficients
Jth Model coefficient variance = Jth diagonal element of Cov(B)
Problems: 1.) We know that D (matrix of input variables) has uncertainty.2.) We suspect that uncertainties may be correlated.
Model coefficient Covariance matrix Cov(B)=[DTD]-1R
2
Response variance( p = # model coefficients
N = # experiments)R
2 = (Ri – Ri )2
N - p
^
i=1
N
Assumptions: Only R has uncertainty; Errors uncorrelated and constant
Cov(B)=D†[Cov(R)]D†T
![Page 16: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/16.jpg)
16
Least Squares Solution to a Process Model
Matrix Representation of Process Model: R = DB
Solving for the Model Coefficients: D†R = B
The pseudoinverse solution of a matrix equation gives the least squares best estimates of the B coefficients!
Define the pseudoinverse of D: D† = (DTD)-1DT
![Page 17: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/17.jpg)
17
Case Study 2: Modeling 45 minute dissolution (D45) of a tableting process
• 32 Factorial Experimental Design– Granulating Water (GS: 36-38 kg)– Granulating Power (P: 18.5-22.5 kW)
• Nested Compression Factors– Compression Force (CF: 11.5-17.5 kN)– Press Speed (S: 70-110 kTPH)
• Least Squares Predictive Model*
D45 = 68.35 – 1.34(GS) – 2.88(P) - 8.95(CF) + 2.43(GS)2
* Parameter values are mean-centered and range-scaled.Publication Reference Application of Quality by Design Knowledge (QbD) From Site Transfers to Commercial Operations Already in Progress,” J. PAT, Jan/Feb, pg. 8, 2006.
![Page 18: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/18.jpg)
18
Measurement Uncertainty and Prediction Uncertainty
Benchmark
Experiment Measured Mean Model Prediction Measured St. Dev.1 69.0 69.2 3.12 73.5 72.3 3.13 71.9 72.1 1.94 67.1 65.5 2.95 69.8 71.2 1.96 75.5 75.0 0.87 65.4 66.6 2.78 75.8 77.3 1.19 56.1 59.4 3.710 61.0 59.4 3.711 67.2 68.3 2.412 77.0 77.3 1.113 72.7 68.3 2.4
2.42.6
Standard Error of PredictionStandard Error (RMS Measurement Standard Deviation)
D45 (%)
![Page 19: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/19.jpg)
19
“Bias” Power (P) Water (GS) Force (CF) Water2
Ran
. Inp
ut Bias Coefficient
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
64 67 70 73
Power Coefficient
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
-8 -6 -4 -2 0 2 4
Water Coefficient
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
-8 -6 -4 -2 0 2 4 6
Water2 Coefficient
0
0.05
0.1
0.15
0.2
0.25
0.3
-4 -2 0 2 4 6 8 10
Force Coefficient
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
-16 -14 -12 -10 -8 -6 -4 -2
Increase in process parameter variance causes a shift in some model coefficients.
Increase in process parameter variance increases model coefficient variance.
Bias Coefficient
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
0.55
64 65 66 67 68 69 70 71 72 73
Power Coefficient
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
-3.5 -2.375 -1.25 -0.125 1
Water Coefficient
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
-7 -6 -5 -4 -3 -2 -1 0 1 2 3 4
Force Coefficient
0
0.2
0.4
0.6
0.8
1
1.2
-5 -4 -3 -2 -1
Water2 Coefficient
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
-4 -2 0 2 4 6 8 10
Reg
ress
ion
Case Study 2: How Do Variances in Process Parameters Influence Model Coefficients?
![Page 20: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/20.jpg)
20
Regression #1 #2 #3 #4P Std. Dev. - - 1 1
GS Std. Dev. - - 0.25 0.25 0.25CF Std. Dev. - - 1 1R Std. Dev. - measured - measured 2.5%
B0 68.35 ± 0.83 68.35 ± 0.93 68.41 ± 1.06 68.39 ± 1.44 68.39 ± 0.9B(GS) -1.34 ± 1.07 -1.34 ± 1.21 -1.23 ± 1.49 -1.22 ± 1.96 -1.24 ± 1.28B(P) -2.88 ± 0.96 -2.88 ± 1.04 -2.13 ± 1.18 -2.11 ± 1.53 -2.63 ± 1.06B(CF) -8.95 ± 1.17 -8.95 ± 1.38 -7.37 ± 1.35 -7.37 ± 1.87 -8.52 ± 1.2
B(GS^2) 2.44 ± 1.35 2.44 ± 1.52 2.13 ± 1.84 2.14 ± 2.43 2.15 ± 1.57Std. Error of Prediction 1.6% 1.8% 2.1% 2.8% 1.8%
Monte Carlo Simulation Parameters
Std. Error
Estimated Model Coefficient Uncertainties from Monte Carlo Simulation
0.5
0.5
![Page 21: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/21.jpg)
21
Regression versus Monte Carlo Distributions for D45
63.0
84
65.9
97
68.9
10
71.8
22
74.7
35
63.0
64.3
65.6
66.9
68.2
69.5
70.8
72.1
73.4
74.7
Regression Monte Carlo (random coeff.)
Minimum = 65.2
Maximum = 71.9
Mean = 68.3
St. Dev = 0.83
Minimum = 62.6
Maximum = 74.5
Mean = 68.4
St. Dev = 1.44
![Page 22: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/22.jpg)
22
Monte Carlo with and without measurement error
5.0% 90.0% 5.0%
64.51 72.29
58 60 62 64 66 68 70 72 74 76 78
0.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
0.16
0.18
Exp_1_1
Exp_1_1
Minimum 59.9748Maximum 77.7486Mean 68.3901Std Dev 2.3729Values 10000
63.0
64.3
65.6
66.9
68.2
69.5
70.8
72.1
73.4
74.7
Monte Carlo (random coeff.)
Minimum = 60.0
Maximum = 77.7
Mean = 68.4
St. Dev = 2.4
Minimum = 62.6
Maximum = 74.5
Mean = 68.4
St. Dev = 1.44
Monte Carlo (random coeff.plus response error)
![Page 23: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/23.jpg)
23
Conclusion
• Uncertainty analysis is a powerful tool to estimate robustness of a process model
• Monte Carlo simulation of process models facilitates implementation of risk mitigation techniques in the control strategy
![Page 24: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/24.jpg)
24
• Back up
![Page 25: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/25.jpg)
25
Experimentation and Process Modeling
D45exp 1
D45exp 2
D45exp 3
GSexp 1 Pexp 1 CFexp 1
GSexp 2 Pexp 2 CFexp 2
GSexp 3 Pexp 3 CFexp 3
![Page 26: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/26.jpg)
26
Propagation of uncertainty in process model predictions:
All Model Coefficient variances and Process Variable variances contribute to each predicted Response uncertainty in a model-dependent manner.
D45pred 1 = B0 + B1 ·GSexp 1 + B2 ·Pexp 1 + B3 ·CFexp 1 + B4 ·GS2exp 1
D45pred 2 = B0 + B1 ·GSexp 2 + B2 ·Pexp 2 + B3 ·CFexp 2 + B4 ·GS2exp 2
D45pred 3 = B0 + B1 ·GSexp 3 + B2 ·Pexp 3 + B3 ·CFexp 3 + B4 ·GS2exp 3
Experimentation and Process Modeling
![Page 27: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/27.jpg)
27
Case Study #2: Influence of Process Parameter Variation on Prediction
• Model Conditions– GS mean = 36 kg– P mean = 20 kW– CF mean = 14 kN– Input parameter standard deviations were varied.– Dissolution values were predicted.
![Page 28: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/28.jpg)
28
B0
B1
B2
B3
B4
A Process Model: Matrix Representation
R = DB
D45pred 1
D45pred 2
D45pred 3
1 GSexp 1 Pexp 1 CFexp 1 GS2exp 1
1 GSexp 2 Pexp 2 CFexp 2 GS2exp 2
1 GSexp 3 Pexp 3 CFexp 3 GS2exp 3
=
Response
matrix
Design
matrix
B
matrix
![Page 29: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/29.jpg)
29
Estimating Variance in Prediction: The Basis for Uncertainty in Design Space
Jth experimental variance = Jth diagonal element of Cov(R)
Response Covariance matrix Cov(R) = B[Cov(D)]BT
Assumptions: Only D has uncertainty.
Problems: 1.) We know that B has uncertainty.2.) We know that uncertainties in D will be correlated, but
we don’t know Cov(D)
![Page 30: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/30.jpg)
30
Example: Simulations 1-4
D45 S
imulation #1
0
0.02
0.04
0.06
0.08
0.1
0.12
6070
8090
100
GS Std. Dev.=0.25 kgP Std. Dev.=1 kWCF Std. Dev.= 1 kND45 Std. Dev.= 0%
D45 Simulation Mean = 74.6% Std. Dev. = 3.70%
D45 S
imulation #2
0
0.02
0.04
0.06
0.08
0.1
0.12
6070
8090
100
D45 S
imulation #3
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
6070
8090
100
D45 S
imulation #4
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
6070
8090
100
D45 Simulation Mean = 75.0% Std. Dev. = 4.59%
D45 Simulation Mean = 76.9% Std. Dev. = 7.89%
D45 Simulation Mean = 76.9% Std. Dev. = 8.26%
GS Std. Dev.=0.5 kgP Std. Dev.=1 kWCF Std. Dev.= 1 kND45 Std. Dev.= 0%
GS Std. Dev.=1 kgP Std. Dev.=1 kWCF Std. Dev.= 1 kND45 Std. Dev.= 0%
GS Std. Dev.=1 kgP Std. Dev.=2 kWCF Std. Dev.= 1 kND45 Std. Dev.= 0%
![Page 31: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/31.jpg)
31
Case Study 2: Influence of Dissolution Measurement Error
• Model Conditions– GS mean = 36 kg– P mean = 20 kW– CF mean = 14 kN– Input parameter standard deviations were varied.– Dissolution measurement error was added.– Dissolution values were predicted.
![Page 32: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/32.jpg)
32
Example: Simulations 5-7
GS Std. Dev=0.5 kgP Std. Dev=1 kWCF Std. Dev=0.5 kND45 Std. Dev=0%(Control)
D45 Simulation Mean = 75.0% Std. Dev. = 3.88%
D45 Simulation Mean = 75.0% Std. Dev. = 4.37%
D45 Simulation Mean = 75.0% Std. Dev. = 5.58%
D45 Simulation Mean = 75.0% Std. Dev. = 7.16%
Sim
ulation #5
0
0.02
0.04
0.06
0.08
0.1
0.12
5065
8095
Sim
ulaton 5-7 Control
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
5060
7080
90100
Sim
ulation #6
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
5060
7080
90100
Sim
ulation #7
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
5064
7892
GS Std. Dev=0.5 kgP Std. Dev=1 kWCF Std. Dev=0.5 kND45 Std. Dev=2%
GS Std. Dev=0.5 kgP Std. Dev=1 kWCF Std. Dev=0.5 kND45 Std. Dev=4%
GS Std. Dev=0.5 kgP Std. Dev=1 kWCF Std. Dev=0.5 kND45 Std. Dev=6%
![Page 33: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/33.jpg)
33
Propagation of Uncertainty in Process Modeling
1. Assign random variables to Dissolution values (R) and use Monte Carlo simulations to propagate error to the model coefficients (B).
Solving for the Model Coefficients: D†R=B
2. Assign random variables to Process Parameters (D) and use Monte Carlo simulations to propagate error to B.
B1 = D†11 ·RExp 1 + D†
12 ·RExp 2 + D†13 ·RExp 3 +…
B2 = D†21 ·RExp 1 + D†
22 ·RExp 2 + D†23 ·RExp 3 +…
The pseudoinverse of D: D†=(DTD)-1DT
![Page 34: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/34.jpg)
34
How Do Variances in Process Parameters Influence Model Coefficients?
• Simulation # 1 (1-0.25-1)– Measured D45 means and standard deviations.– P 19-23 kW ± 1 kW– GS 36-38 kg ± 0.25 kg– CF 12-18 ± 1 kN
• Compare to regression distributions– Model coefficient means– Model coefficient standard deviations
![Page 35: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/35.jpg)
35
Case Study 2: Simulation 1
Distribution for Water (kg)
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
35 35.5 36 36.5 37
Distribution for Power (kW)
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
15 17.5 20 22.5 25
Distribution for Force (kN)
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
9 12 15 18
D45 Simulation #1
0
0.02
0.04
0.06
0.08
0.1
0.12
60 70 80 90 100
GS Mean = 36 kg Std. Dev. = 0.25 kg
P Mean = 20 kW Std. Dev. = 1 kW
CF Mean = 14 kN Std. Dev. = 1 kN
D45 Simulation ResultMean = 74.6% Std. Dev. = 3.70%
D45 = 68.35 – 1.34(GS) – 2.88(P) - 8.95(CF) + 2.43(GS)2
![Page 36: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/36.jpg)
36
Case Study 2: Influence of Process Parameters Variation
• Increase in granulation water mass (GS) variance:– Increases predicted D45 variance.– Slightly shifts predicted D45 means.– Skews the predicted D45 distributions.
• Increase in granulator power (P) endpoint variance:– Increases predicted D45 variance.– Does not shift predicted D45 means.– Does not skew the predicted D45 distributions.
![Page 37: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/37.jpg)
37
Case Study 2: Influence of Dissolution Measurement Error
• Increase in D45 measurement variance:– does not shift predicted D45 means.– does not appear to skew predicted D45
distributions.– increases predicted D45 variance.
• Advantage #2, we get the distribution, not just the standard deviation.
• Advantage #3, sensitivity analysis allows us to prioritize process improvement.
![Page 38: Evaluation of Process Models with Monte Carlo Methodsqbdworks.com/wp-content/uploads/2014/06/Design... · Evaluation of Process Models with Monte Carlo Methods John Kauffman. FDA\CDER](https://reader034.vdocuments.us/reader034/viewer/2022042311/5ed9f3c128db2d5ca2492409/html5/thumbnails/38.jpg)
38
Benchmark
Experiment Measured Mean Model Prediction Measured St. Dev.1 69.0 69.2 3.12 73.5 72.3 3.13 71.9 72.1 1.94 67.1 65.5 2.95 69.8 71.2 1.96 75.5 75.0 0.87 65.4 66.6 2.78 75.8 77.3 1.19 56.1 59.4 3.710 61.0 59.4 3.711 67.2 68.3 2.412 77.0 77.3 1.113 72.7 68.3 2.4
2.42.6
Standard Error of PredictionStandard Error (RMS Measurement Standard Deviation)
D45 (%)
Measurement Uncertainty and Prediction Uncertainty