ensemble modelling & data uncertainty within glue · 2012. 7. 4. · 6 rainfall scenarios from...

Ensemble modelling & data uncertainty within GLUE Tobias Krueger, Jim Freer & John Quinton

20 years of GLUE 293 papers (that I found)

0

5

10

15

20

25

30

35

40

1992

1994

1996

1998

2000

2002

2004

2006

2008

2010

2012

Nu

mb

er

of

pa

pe

rs

all

Model hypothesis testing within GLUE Multiple model structures & data uncertainty

“Different sets of initial and boundary conditions may also be evaluated in this

way, indeed, in the general case, different model structures can be considered.” (p.

281)

“It is also worth noting, finally, that the GLUE procedure allows not only different

parameter sets, but also different model structures to be incorporated into the

uncertainty analysis, where by model structure we can subsume the use of different

discretizations or sets of boundary conditions as well as different models.” (p.

297)

“This type of measure [max. abs. residual] has been used, for example, by Keesman

and van Straten (1989, 1990) and van Straten and Keesman (1991) in modelling

lake eutrophication, with the additional requirement that residual lies within

measurement error bounds (else Lm = 0). […] They found that it was necessary to

make the ‘measurement error bounds’ unduly wide to obtain a reasonable number

of parameter sets that would satisfy this latter criterion at all time steps, […].” (p.

283f)

Beven & Binley, 1992

Outputs Inputs

ε

θ1, θ2, …, θn

Model 1

θ1, θ2, …, θn

Model 2

θ1, θ2, …, θn

Model n

…

L

ε

Model hypothesis testing within GLUE Multiple model structures & data uncertainty

Uncertainty in input data 43 papers

0

5

10

15

20

25

30

35

40

1992

1994

1996

1998

2000

2002

2004

2006

2008

2010

2012

Nu

mb

er

of

pa

pe

rs

all

explicit input data uncertainty, propagation

explicit input data uncertainty, calibration

Uncertainty in input data Groundwater modelling studies

Annual precipitation input range

Synthetic – Jensen, J. B. and K. Schaarup-Jensen (2002). Delineation of capture zones by an integrated

surface/subsurface model using the GLUE methodology. Calibration and Reliability in Groundwater Modelling: A

Few Steps Closer to Reality (Proceedings of ModelCARE'2002). IAHS Publ. No. 277: 478-488.

4 boundary condition ranges including daily recharge & ET rates

Synthetic – Rojas, R., L. Feyen and A. Dassargues (2008). "Conceptual model uncertainty in groundwater

modeling: Combining generalized likelihood uncertainty estimation and Bayesian model averaging." Water

Resources Research 44(12).

Synthetic – Rojas, R., L. Feyen and A. Dassargues (2009). "Sensitivity analysis of prior model probabilities and

the value of prior knowledge in the assessment of conceptual model uncertainty in groundwater modelling."

Hydrological Processes 23(8): 1131-1146.

Synthetic – Rojas, R., L. Feyen, O. Batelaan and A. Dassargues (2010). "On the value of conditioning data to

reduce conceptual model uncertainty in groundwater modeling." Water Resources Research 46.

Real – Rojas, R., O. Batelaan, L. Feyen and A. Dassargues (2010). "Assessment of conceptual model uncertainty

for the regional aquifer Pampa del Tamarugal - North Chile." Hydrology and Earth System Sciences 14(2): 171-

192.

3 recharge scenarios

Real – Rojas, R., S. Kahunde, L. Peeters, O. Batelaan, L. Feyen and A. Dassargues (2010). "Application of a

multimodel approach to account for conceptual model and scenario uncertainties in groundwater modelling."

Journal of Hydrology 394(3-4): 416-435.

Uncertainty in input data Groundwater modelling studies

Alternative recharge models

5 – Ye, M., K. F. Pohlmann, J. B. Chapman, G. M. Pohll

and D. M. Reeves (2009). "A Model-Averaging Method for

Assessing Groundwater Conceptual Model Uncertainty."

Ground Water 48(5): 716-728.

Reeves, D. M., K. F. Pohlmann, G. M. Pohll, M. Ye and J. B.

Chapman (2010). "Incorporation of conceptual and

parametric uncertainty into radionuclide flux estimates from a

fractured granite rock mass." Stochastic Environmental

Research and Risk Assessment 24(6): 899-915.“

• 2 – Rojas, R., O. Batelaan, L. Feyen and A. Dassargues

(2010). "Assessment of conceptual model uncertainty for the

regional aquifer Pampa del Tamarugal - North Chile."

Hydrology and Earth System Sciences 14(2): 171-192.

• 9 – Singh, A., S. Mishra and G. Ruskauff (2010). "Model

Averaging Techniques for Quantifying Conceptual Model

Uncertainty." Ground Water 48(5): 701-715.

Uncertainty in input data Rainfall-runoff modelling studies

Perturbation of area average rainfall input

Constant relative multiplier, 5 scenarios – Gourley, J. J. and B. E. Vieux (2003). The effects of radar-derived

rainfall uncertainties on forecasts from a distributed hydrological model. Weather Radar Information and

Distributed Hydrological Modelling: 130-137.

Constant relative multiplier, sampled as parameters – Ratto, M., P. C. Young, R. Romanowicz, F. Pappenberger,

A. Saltelli and A. Pagano (2007). "Uncertainty, sensitivity analysis and the role of data based mechanistic

modeling in hydrology." Hydrology and Earth System Sciences 11(4): 1249-1266.

Systematic (bias correction) & random (Gaussian), 100 scenarios – Lee, H., D. Balin, R. R. Shrestha and M.

Rode (2010). "Streamflow prediction with uncertainty analysis, Weida catchment, Germany." Ksce Journal of Civil

Engineering 14(3): 413-420.

Rainfall & snow input ranges

Blazkova, S. and K. Beven (2009). "A limits of acceptability approach to model evaluation and uncertainty

estimation in flood frequency estimation by continuous simulation: Skalka catchment, Czech Republic." Water

Resources Research 45.

6 rainfall scenarios from actual gauges

Krueger, T., J. Freer, J. N. Quinton, C. J. A. Macleod, G. S. Bilotta, R. E. Brazier, P. Butler and P. M. Haygarth

(2010). "Ensemble evaluation of hydrological model hypotheses." Water Resources Research 46.

Uncertainty in input data Rainfall-runoff modelling studies

Perturbation of spatial rainfall pattern

16 scenarios – Younger, P. M., J. E. Freer and K. J. Beven (2009). "Detecting the effects of spatial

variability of rainfall on hydrological modelling within an uncertainty analysis framework." Hydrological

Processes 23(14): 1988-2003.

Uncertainty in input data Water quality modelling studies

Flow, range, propagated by random sampling

• Page, T., K. J. Beven, J. Freer and A. Jenkins (2003). "Investigating the uncertainty in predicting responses to

atmospheric deposition using the model of acidification of groundwater in catchments (MAGIC) within a

generalised likelihood uncertainty estimation (GLUE) framework." Water Air and Soil Pollution 142(1-4): 71-94.

Flow, rectangular fuzzy, propagated by extension principle

• Krueger, T., J. N. Quinton, J. Freer, C. J. A. Macleod, G. S. Bilotta, R. E. Brazier, P. Butler and P. M. Haygarth

(2009). "Uncertainties in Data and Models to Describe Event Dynamics of Agricultural Sediment and Phosphorus

Transfer." Journal of Environmental Quality 38(3): 1137-1148.

Flow & suspended sediment concentration, rectangular & trapezoidal fuzzy, propagated

by extension principle

• Krueger, T., J. N. Quinton, J. Freer, C. J. A. Macleod, G. S. Bilotta, R. E. Brazier, J. M. B. Hawkins and P. M.

Haygarth (2012). "Comparing empirical models for sediment and phosphorus transfer from soils to water at field

and catchment scale under data uncertainty." European Journal of Soil Science 63(2): 211-223.

Uncertainty in input data Atmospheric modelling studies

Emission scenarios

25 – Page, T., J. D. Whyatt, K. J. Beven and S. E. Metcalfe

(2004). "Uncertainty in modelled estimates of acid deposition

across Wales: a GLUE approach." Atmospheric Environment

38(14): 2079-2090.

Page, T., K. J. Beven and D. Whyatt (2004). "Predictive

capability in estimating changes in water quality: Long-term

responses to atmospheric deposition." Water Air and Soil

Pollution 151(1-4): 215-244.

Heywood, E., J. D. Whyatt, J. Hall, R. Wadsworth and T.

Page (2006). "Presentation of the influence of deposition

uncertainties on acidity critical load exceedance across

Wales." Environmental Science & Policy 9(1): 32-45.

Whyatt, J. D., S. E. Metcalfe, J. Nicholson, R. G. Derwent, T.

Page and J. R. Stedman (2007). "Regional scale modelling

of particulate matter in the UK, source attribution and an

assessment of uncertainties." Atmospheric Environment

41(16): 3315-3327.

50 – Page, T., J. D. Whyatt, S. E. Metcalfe, R. G. Derwent and C. Curtis (2008). "Assessment of

uncertainties in a long range atmospheric transport model: Methodology, application and implications in

a UK context." Environmental Pollution 156(3): 997-1006.

Uncertainty in input data Flood inundation modelling studies

Flow input uncertainty via rating curve, 2 models & parameter ranges

Pappenberger, F., P. Matgen, K. J. Beven, J.-B. Henry, L. Pfister and P. de Fraipont (2006). "Influence of

uncertain boundary conditions and model structure on flood inundation predictions." Advances in Water

Resources 29(10): 1430-1449.

20 flow input scenarios

Pappenberger, F., K. Frodsham, K. Beven, R. Romanowicz and P. Matgen (2007). "Fuzzy set approach to

calibrating distributed flood inundation models using remote sensing observations." Hydrology and Earth System

Sciences 11(2): 739-752.

Pappenberger, F., K. Beven, K. Frodsham, R. Romanowicz and P. Matgen (2007). "Grasping the unavoidable

subjectivity in calibration of flood inundation models: A vulnerability weighted approach." Journal of Hydrology

333(2-4): 275-287.

Uncertainty in evaluation data 26 papers

0

5

10

15

20

25

30

35

40

1992

1994

1996

1998

2000

2002

2004

2006

2008

2010

2012

Nu

mb

er

of

pa

pe

rs

all

explicit evaluation data uncertainty, no LOA

explicit evaluation data uncertainty, LOA

Uncertainty in evaluation data 1st limits of acceptability (LOA)?

Total saturated area, custom fuzzy, 1 timestep

Franks, S. W., P. Gineste, K. J. Beven and P. Merot (1998). "On constraining the predictions of a

distributed moder: The incorporation of fuzzy estimates of saturated areas into the calibration process."

Water Resources Research 34(4): 787-797.

Uncertainty in evaluation data Fuzzy additive

Concentrations, trapezoidal – Page, T., K. J. Beven, J. Freer

and A. Jenkins (2003). "Investigating the uncertainty in

predicting responses to atmospheric deposition using the model

of acidification of groundwater in catchments (MAGIC) within a

generalised likelihood uncertainty estimation (GLUE)

framework." Water Air and Soil Pollution 142(1-4): 71-94.

Atmospheric depositions, various – Page, T., J. D. Whyatt, K. J.

Beven and S. E. Metcalfe (2004). "Uncertainty in modelled

estimates of acid deposition across Wales: a GLUE

approach." Atmospheric Environment 38(14): 2079-2090.

Page, T., K. J. Beven and D. Whyatt (2004). "Predictive

capability in estimating changes in water quality: Long-term

responses to atmospheric deposition." Water Air and Soil

Pollution 151(1-4): 215-244.

Heywood, E., J. D. Whyatt, J. Hall, R. Wadsworth and T. Page

(2006). "Presentation of the influence of deposition uncertainties

on acidity critical load exceedance across Wales."

Environmental Science & Policy 9(1): 32-45.

Whyatt, J. D., S. E. Metcalfe, J. Nicholson, R. G. Derwent, T.

Page and J. R. Stedman (2007). "Regional scale modelling of

particulate matter in the UK, source attribution and an

assessment of uncertainties." Atmospheric Environment 41(16):

3315-3327.


Atmospheric depositions, trapezoidal – Page, T., J. D. Whyatt, S. E. Metcalfe, R. G. Derwent and C. Curtis

(2008). "Assessment of uncertainties in a long range atmospheric transport model: Methodology, application and

implications in a UK context." Environmental Pollution 156(3): 997-1006.

Pixel inundation, quasi-trapezoidal – Pappenberger, F., K. Beven, M. Horritt and S. Blazkova (2005). "Uncertainty

in the calibration of effective roughness parameters in HEC-RAS using inundation and downstream level

observations." Journal of Hydrology 302(1-4): 46-69.

Pixel inundation, various – Pappenberger, F., P. Matgen, K. J. Beven, J.-B. Henry, L. Pfister and P. de Fraipont

(2006). "Influence of uncertain boundary conditions and model structure on flood inundation predictions."

Advances in Water Resources 29(10): 1430-1449.

Pixel inundation, various – Pappenberger, F., K. Frodsham, K. Beven, R. Romanowicz and P. Matgen (2007).

"Fuzzy set approach to calibrating distributed flood inundation models using remote sensing observations."




333(2-4): 275-287.

Flow, trapezoidal – Quinton, J. N., T. Krueger, J. Freer, G. S. Bilotta and R. E. Brazier (2010). A case study of

uncertainty: Applying GLUE to EUROSEM. Handbook of Erosion Modelling. R. P. C. Morgan and M. A. Nearing.

Chichester, Blackwell Publishing Ltd: 80-97.


Water table, trapezoidal

• Freer, J. E., H. McMillan, J. J. McDonnell and

K. J. Beven (2004). "Constraining dynamic

TOPMODEL responses for imprecise water

table information using fuzzy rule based

performance measures." Journal of

Hydrology 291(3-4): 254-277.

• Juston, J., J. Seibert and P.-O. Johansson

(2009). "Temporal sampling strategies and

uncertainty in calibrating a conceptual

hydrological model for a small boreal

catchment." Hydrological Processes 23(21):

3093-3109.

Uncertainty in evaluation data Uncertainty range as uncertain parameter

Flow, trapezoidal – Ratto, M., P. C. Young, R. Romanowicz, F. Pappenberger, A. Saltelli and A. Pagano

(2007). "Uncertainty, sensitivity analysis and the role of data based mechanistic modeling in hydrology."


Uncertainty in evaluation data Probabilistic forecasting skill metrics

Rainfall, combination of various fuzzy & probabilistic component errors – Pappenberger, F., A. Ghelli, R. Buizza

and K. Bodis (2009). "The Skill of Probabilistic Precipitation Forecasts under Observational Uncertainties within

the Generalized Likelihood Uncertainty Estimation Framework for Hydrological Applications." Journal of

Hydrometeorology 10(3): 807-819.

Uncertainty in evaluation data Limits of acceptability

Various fuzzy, subsequently relaxed – Blazkova, S. and K. Beven (2009). "A limits of acceptability approach to

model evaluation and uncertainty estimation in flood frequency estimation by continuous simulation: Skalka

catchment, Czech Republic." Water Resources Research 45.

Flow, trapezoidal fuzzy additive + LOA, flexible – Krueger, T., J. Freer, J. N. Quinton, C. J. A. Macleod, G. S.

Bilotta, R. E. Brazier, P. Butler and P. M. Haygarth (2010). "Ensemble evaluation of hydrological model

hypotheses." Water Resources Research 46.

Flow, trapezoidal fuzzy additive + LOA, flexible – Krueger, T., J. Freer, J. N. Quinton, C. J. A. Macleod, G. S.

Bilotta, R. E. Brazier, J. M. B. Hawkins and P. M. Haygarth (2010). Hydrological model hypothesis testing using

imprecise spatial flux measurements. Ninth International Symposium on Spatial Accuracy Assessment in Natural

Resources and Environmental Sciences, Leicester, University of Leicester.

Flow, triangular fuzzy additive + LOA, subsequently relaxed minimally so that 10% of timesteps violated

LOA – Liu, Y., J. Freer, K. Beven and P. Matgen (2009). "Towards a limits of acceptability approach to the

calibration of hydrological models: Extending observation error." Journal of Hydrology 367(1-2): 93-103.

Soil carbon stock, triangular fuzzy, subsequently relaxed allowing 1/9 of timesteps to violate LOA – Ortiz, C.,

E. Karltun, J. Stendahl, A. I. Gardenas and G. I. Agren (2011). "Modelling soil carbon development in Swedish

coniferous forest soils-An uncertainty analysis of parameters and model estimates using the GLUE method."

Ecological Modelling 222(17): 3020-3032.

Flow duration quantiles, triangular fuzzy additive + LOA, no relaxation but easier criterion – Westerberg, I. K.,

J. L. Guerrero, P. M. Younger, K. J. Beven, J. Seibert, S. Halldin, J. E. Freer and C. Y. Xu (2011). "Calibration of

hydrological models using flow-duration curves." Hydrology and Earth System Sciences 15(7): 2205-2227.

Uncertainty in evaluation data Limits of acceptability: fuzzy multiplicative

Cross section inundation, parabolic fuzzy support (physical limits)

Romanowicz, R. and K. Beven (2003). "Estimation of flood inundation probabilities as conditioned on

event inundation maps." Water Resources Research 39(3).

Uncertainty in evaluation data Limits of acceptability: fuzzy multiplicative

Various nitrogen cycle state variables, trapezoidal fuzzy (“soft data”)

Rankinen, K., T. Karvonen and D. Butterfield (2006). "An application of the GLUE methodology for estimating the

parameters of the INCA-N model." Science of the Total Environment 365(1-3): 123-139.

Suspended solids & total phosphorus concentrations, trapezoidal fuzzy

Krueger, T., J. N. Quinton, J. Freer, C. J. A. Macleod, G. S. Bilotta, R. E. Brazier, J. M. B. Hawkins and P. M.

Haygarth (2012). "Comparing empirical models for sediment and phosphorus transfer from soils to water

at field and catchment scale under data uncertainty." European Journal of Soil Science 63(2): 211-223.

Uncertainty in evaluation data Limits of acceptability: Gaussian conf. int.

Various flow related signatures, 95% confidence interval of Gaussian error model as

LOA

• Winsemius, H. C., B. Schaefli, A. Montanari and H. H. G. Savenije (2009). "On the calibration of hydrological

models in ungauged basins: A framework for integrating hard and soft hydrological information." Water Resources

Research 45.

Multiple model structures 19 papers

0

5

10

15

20

25

30

35

40

1992

1994

1996

1998

2000

2002

2004

2006

2008

2010

2012

Nu

mb

er

of

pa

pe

rs

all

multiple structures, boundary conditions

multiple structures, formal models

Multiple model structures Boundary conditions: geological models

2 – Jensen, J. B. and K. Schaarup-Jensen (2002). Delineation of capture zones by an integrated

surface/subsurface model using the GLUE methodology. Calibration and Reliability in Groundwater Modelling: A

Few Steps Closer to Reality (Proceedings of ModelCARE'2002). IAHS Publ. No. 277: 478-488.

7 – Rojas, R., L. Feyen and A. Dassargues (2008). "Conceptual model uncertainty in groundwater modeling:

Combining generalized likelihood uncertainty estimation and Bayesian model averaging." Water Resources

Research 44(12).

Rojas, R., L. Feyen and A. Dassargues (2009). "Sensitivity analysis of prior model probabilities and the value of

prior knowledge in the assessment of conceptual model uncertainty in groundwater modelling." Hydrological

Processes 23(8): 1131-1146.

Rojas, R., L. Feyen, O. Batelaan and A. Dassargues (2010). "On the value of conditioning data to reduce

conceptual model uncertainty in groundwater modeling." Water Resources Research 46.

4 – Rojas, R., O. Batelaan, L. Feyen and A. Dassargues (2010). "Assessment of conceptual model uncertainty for

the regional aquifer Pampa del Tamarugal - North Chile." Hydrology and Earth System Sciences 14(2): 171-192.

5 – Ye, M., K. F. Pohlmann, J. B. Chapman, G. M. Pohll and D. M. Reeves (2009). "A Model-Averaging Method

for Assessing Groundwater Conceptual Model Uncertainty." Ground Water 48(5): 716-728.

Reeves, D. M., K. F. Pohlmann, G. M. Pohll, M. Ye and J. B. Chapman (2010). "Incorporation of conceptual and

parametric uncertainty into radionuclide flux estimates from a fractured granite rock mass." Stochastic

Environmental Research and Risk Assessment 24(6): 899-915.

9 – Singh, A., S. Mishra and G. Ruskauff (2010). "Model Averaging Techniques for Quantifying Conceptual Model

Uncertainty." Ground Water 48(5): 701-715.

Multiple model structures Boundary conditions: geological models

3 – Rojas, R., S. Kahunde, L. Peeters, O. Batelaan, L. Feyen and A. Dassargues (2010). "Application of a

multimodel approach to account for conceptual model and scenario uncertainties in groundwater

modelling." Journal of Hydrology 394(3-4): 416-435.

Multiple model structures Boundary conditions: floodplain models

4 bridge implementations

• Pappenberger, F., P. Matgen, K. J. Beven, J.-B. Henry, L. Pfister and P. de Fraipont (2006). "Influence of

uncertain boundary conditions and model structure on flood inundation predictions." Advances in Water

Resources 29(10): 1430-1449.

Multiple model structures Boundary conditions: floodplain models

Cross section error (width & depth) via 3 parameters

• Pappenberger, F., K. Frodsham, K. Beven, R. Romanowicz and P. Matgen (2007). "Fuzzy set approach to

calibrating distributed flood inundation models using remote sensing observations." Hydrology and Earth System

Sciences 11(2): 739-752.



333(2-4): 275-287.

Multiple model structures Formal models

Artificial Neural Network structural uncertainty (input variable selection, lag-space

relationships, hidden node size)

• Asefa, T. (2009). "Ensemble Streamflow Forecast: A GLUE-Based Neural Network Approach." Journal of the

American Water Resources Association 45(5): 1155-1163.

72 rainfall-runoff models (flexible bucket structure)

• Krueger, T., J. Freer, J. N. Quinton, C. J. A. Macleod, G. S. Bilotta, R. E. Brazier, P. Butler and P. M.

Haygarth (2010). "Ensemble evaluation of hydrological model hypotheses." Water Resources Research

46.


4 rainfall-runoff models (TOPMODEL, NAM, HYMOD, TANK)

• Bastola, S., C. Murphy and J. Sweeney (2011). "The role of hydrological modelling uncertainties in

climate change impact assessments of Irish river catchments." Advances in Water Resources 34(5): 562-

576.

• Bastola, S., C. Murphy and J. Sweeney (2011). "The sensitivity of fluvial flood risk in Irish catchments to the range

of IPCC AR4 climate change scenarios." Science of the Total Environment 409(24): 5403-5415.

• Bastola, S., C. Murphy and R. Fealy (in press). "Generating probabilistic estimates of hydrological response for

Irish catchments using a weather generator and probabilistic climate change scenarios." Hydrological Processes:

n/a-n/a.

2 rainfall-runoff models (Catchmod, PDM)

• von Christierson, B., J.-P. Vidal and S. D. Wade (2012). "Using UKCP09 probabilistic climate information for UK

water resource planning." Journal of Hydrology 424: 48-67.


3 empirical flow-suspended solids models & 3 empirical suspended solids-total

phosphorus models

• Krueger, T., J. N. Quinton, J. Freer, C. J. A. Macleod, G. S. Bilotta, R. E. Brazier, J. M. B. Hawkins and P. M.

Haygarth (2012). "Comparing empirical models for sediment and phosphorus transfer from soils to water

at field and catchment scale under data uncertainty." European Journal of Soil Science 63(2): 211-223.

GLUE is more than Nash-Sutcliffe > 0.6 (or 0.3 …)

If both input and evaluation data uncertainties are comprehensively accounted for

then we can apply rigorous limits of acceptability to reject models

It is probably safer to estimate data uncertainties conservatively and risk Type I error

(falsely accepting models) as this can be corrected later … cost?

Then we only need to relax the limits for operational (and diagnostic) purposes

accounting for model structural error … and this decision is best made by those

using the predictions (Pappenberger et al., 2007)

Multiplicative uncertainty models, if bounded, act as limits of acceptability and

remove the need for other (ad hoc?) limits, although these can be additional criteria

Ensemble modelling: For rejecting model structures as a whole an optimisation step

might be effective (Krueger et al., 2010)

We need to work on incorporating correlations between model structures into priors

Model hypothesis testing within GLUE Some concluding thoughts

ensemble modelling & data uncertainty within glue · 2012. 7. 4. · 6 rainfall scenarios from...

Documents