Aged sorption: Reliability of

estimated model parameters

Mechteld ter Horst, Jos Boesten (Alterra)Wendy van Beinum, Sabine Beulke (FERA)

Outline

Goal

Method

Results parameter estimation

Criteria for an acceptable fit

Conclusions

Goal

Identify for which cases parameter estimation is not possible

Problem with complex models: fit large number of model parameters

Therefore derivation of reliable parameters is challenging

Therefore important to test in which cases the model fitting doesn’t work (e.g. compounds with weak or strong sorption or with fast or slow degradation).

Give guidance on:

Parameter estimation (weight factor, no of sampling times)

Evaluate the results of the parameter estimation

Method: overview

Generate hypothetical datasets with Monte Carlo simulations using random errors as observed in FERA experiments

Parameter estimation (PEST) for each parameter combination using hypothetical datasets as measurements

Calculate for each parameter combination for fNE kd the CV_25,true-param

(coefficient of variation)

Identify for which cases parameter estimation is not possible (contour plots)

1

2

3

4

5Use parameter estimation output to find method on how to judge if the outcome of the parameter estimation is acceptable

Method: Generate 9450 datasets with Monte Carlo simulations using

random errors from FERA experiments

378 parameter combinations (see next slide)

25 hypothetical datasets per parameter combination containing per dataset:

measurements of mass and concentration on 9 sampling times

3 replicate measurements per sampling time

These datasets are

used as measurements

in the parameter

estimation

process.

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0 20 40 60 80 100 120

time (days)

co

nc

en

tra

tio

n (

ug

.mL

-1)

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

0 20 40 60 80 100 120

time (days)

co

nc

en

trati

on

(u

g.m

L-1

)

true values

replicate set 1

replicate set 2

replicate set 3

Example hypothetical dataset incl. random errors

Method: Generate 9450 datasets with Monte Carlo simulations using

random errors from FERA experiments

378 parameter combinations (6*7*3*3)

DegT50

(d)

10 20 40 60 80 100

KF,eq (L/kg)

0.1 0.32 1 3.2 5.5 7.5 10

kd

(d-1)

0.005 0.01 0.03

fNE

(-)

0.3 0.6 1.2

(6)

(7)

(3)

(3)

Freundlich exponent, N = 0.9 for all parameter combinations

Method: parameter estimation

PEST (Doherty, 2006): minimise the difference between simulated and measured values.

Sum of the squared residuals used as criterion function (object function Φ):

weighting of data needed: mass order of magnitude larger than concentration

weighting data mass and concentration got equal importance

2

1

m

i

ii rw

ri is the residual (difference between simulated and measured value corresponding to measurement i ),

wi is the weighting factor

m is the sum of numbers of measurements of mass and concentration

Method: parameter estimation

Estimated parameters: fNE, kd, Kom,eq, DegT50 and initial mass Pest runs for 8 sets of starting values

For each parameter combination (378), hypothetical dataset (25) and set of starting values (8) a PEST_PEARLNEQ run is performed (75600 runs).

Example PEST output

OPTIMISATION RESULTS

Parameters ----->

Parameter Estimated 95% percent confidence limits

value lower limit upper limit

fsne 0.211204 8.766462E-02 0.334744

crd 6.662777E-03 1.876465E-04 1.313791E-02

dt50 19.7934 19.0784 20.5084

masini 42.1954 40.4723 43.9185

komeq 280.040 263.763 296.317

Objective function ----->

Sum of squared weighted residuals (ie phi) = 0.3111

Method: calculation of CV_25,true-param (coefficient of variation)

Calculation of CV_25,true-param fNE and kd for each parameter combination (378)

1. Select from fits with 8 sets of starting values the best fit (lowest Φ)

2. Calculate from 25 fits (25 hypothetical datasets) the CV25,true-param according:

Note: kd as an example s = standard deviation

Small CVs indicate a small variability between the 25 fitted parameter values and a small deviation from the true value

25

...25

1

2

i

kdtruekdfitted.

skdtrue

sparamtrueCV

....25

Results : CV25,true-param of fNE on contour

0.005

0.01

0.03

fNE

kd 0.3 0.6 1.2

Results: CV25,true-param kd on contour

0.005

0.01

0.03

fNE

kd

0.3 0.6 1.2

Results parameter estimationResults visible from contour diagrams

i. Decreasing CV for increasing kd and fNE values

ii. Decreasing CV for increasing KF,EQ values, (except for DegT50 = 10 days and low values of kd and fNE)

iii. Increasing CV for increasing DegT50 values

For explanations see report

kd 0.005, fNE 0.3 large CV

kd 0.03, fNE

1.2 small CV

Larger KF,eq

smaller CV

Larger DegT50 larger CV

Criteria for an acceptable fit: Aim of analysis

Goal:

Test if the confidence interval of single fit can be used as an indication for an accurate parameter estimation (fitted parameter value close to true value).

Method:

compare confidence interval with difference between true and fitted value

Advantage of hypothetical dataset:

we know the true values and can therefore test how well the fitted values come close to the true values

Criteria for an acceptable fit: uncertainty of fitted parameter

Criterion 1: uncertainty measure: CV1,PEST conf.intv

large uncertainty large confidence interval

the 95% confidence interval (PEST output) of 1 fitted parameter of 1 fit (hypothetical dataset) is used to calculate the CV1,PEST conf.intv (RSE in report)

Large CV1,PEST conf.intv indicates large uncertainty

Uncertain parameters can be caused by:

parameter insensitivity and/or parameter correlation different parameter values may result in the same good visual fit

Bad (visual) fit fitted data (mass/conc.) does not correspond to measured data

Criteria for an acceptable fit: accuracy of fitted parameter

Criterion 2: accuracy measure: ratio true/fitted

Accurate fitted parameter fitted parameter value is close to the true parameter value.

The ratio was calculated so that the values were smaller or equal to 1

Values close to 1 indicate accurate fit

Possible to fit parameter value that is inaccurate and of which the uncertainty is large but also possible: accurate fitted parameter with large uncertainty

For each fitted value of fNE and kd uncertainty (CV1,PEST conf.intv) and accuracy (true/fitted) is determined

Note:

accuracy of 0.75 is

an arbitrary value:

we decided ourselves that

the fitted value should be

within 25% of the true value

CV1,PEST conf.intv ≤ 0.25 small uncertainty

Criteria for an acceptable fit: uncertainty and accuracy

0

0.2

0.4

0.6

0.8

1

0.01 0.1 1 10 100

Tru

e/f

itte

d

RSE model fit

Small uncertainty &

accurate

Large uncertainty but

accurate

Small uncertainty but inaccurate

Large uncertainty &

inaccurate

CV1,PEST conf.intv

0

0.2

0.4

0.6

0.8

1

0.01 0.1 1 10 100

Tru

e/f

itte

d

RSE model fit

Rightly accepted (small uncertainty

& accurate)

Wrongly rejected (small uncertainty

but accurate)

Wrongly accepted (small uncertainty but

inaccurate)

Rightly rejected (large uncertainty and inaccurate)

A

B

C

D

CV1,PEST conf.intv

Criteria for an acceptable fit: uncertainty and accuracy

Principle:

reliable fitted parameters should always be accurate whatever the degree of uncertainty

Accuracy cannot be used in practice as criterion because the true parameter value is unknown.

Uncertainty can be used as criterion in practice

Criteria for an acceptable fit: uncertainty as criterion

Suppose CV1,PEST conf.intv ≤ 0.25 is criterion for a reliable fiited parameter

How many fits would be rightly accepted and wrongly accepted?

42%

4%

17%

36%

CV 1,PEST conf.intv

AB

C

D

Conclusions

Contour plots of the CV25,true-param of fNE and kd shows that parameter estimation is more successful for larger sorption

The proposed CV1,PEST conf.intv ≤ 0.25 will result in only 4% of the cases in wrongly accepted fits.

Thank you for your attention !

© Wageningen UR