![Page 1: Data Assimilation in Environmental Modellinghydrocast.dhigroup.com/publications/DataAssimilation_HOBEworkshop... · for data assimilation • OpenDA is an open interface standard](https://reader031.vdocuments.us/reader031/viewer/2022041416/5e1c06ff8c4073619f1506de/html5/thumbnails/1.jpg)
Henrik Madsen
Data Assimilation in Environmental Modelling
DHI
HOBE-TR32 Joint workshop, Sandbjerg Castle, 1-4 April 2012
![Page 2: Data Assimilation in Environmental Modellinghydrocast.dhigroup.com/publications/DataAssimilation_HOBEworkshop... · for data assimilation • OpenDA is an open interface standard](https://reader031.vdocuments.us/reader031/viewer/2022041416/5e1c06ff8c4073619f1506de/html5/thumbnails/2.jpg)
Data assimilation framework
state variables
parameters
Input Variables
1
Measured Variables
2 3 4
Forecast or
Output Variables
state variables
parameters
Input Variables
1 1
Measured Variables
2 2 3 3 4 4
Forecast or
Output Variables Numerical Model Data
assimilation
procedure
1. Correction of model forcing
2. State updating
3. Parameter estimation
4. Error forecasting
Refsgaard (1997)
![Page 3: Data Assimilation in Environmental Modellinghydrocast.dhigroup.com/publications/DataAssimilation_HOBEworkshop... · for data assimilation • OpenDA is an open interface standard](https://reader031.vdocuments.us/reader031/viewer/2022041416/5e1c06ff8c4073619f1506de/html5/thumbnails/3.jpg)
Filtering framework
),( 1 k
a
k
f
k uxx Model forecast Predictor
Corrector
(.) Model operator (one-step ahead predictor)
xk System state
uk Model forcing
zk Measurements
Ck Mapping of state space to measurement space
Gk Weighting matrix
k Time step
)( f
kkkxCz Measurements
))(( f
kkkk
f
k
a
kxCzGxx Update
![Page 4: Data Assimilation in Environmental Modellinghydrocast.dhigroup.com/publications/DataAssimilation_HOBEworkshop... · for data assimilation • OpenDA is an open interface standard](https://reader031.vdocuments.us/reader031/viewer/2022041416/5e1c06ff8c4073619f1506de/html5/thumbnails/4.jpg)
Kalman filter
kk
a
k
f
k uxx ),( 1Model forecast Predictor
Corrector
k
f
kkk xCz )(Measurements
))(( f
kkkk
f
k
a
kxCzGxx Update
• Stochastic interpretations of model and measurement equations
• Weighting matrix (Kalman gain) determined based on a least squares
minimisation of the expected error of the updated state estimate
• Estimation of model prediction uncertainty
![Page 5: Data Assimilation in Environmental Modellinghydrocast.dhigroup.com/publications/DataAssimilation_HOBEworkshop... · for data assimilation • OpenDA is an open interface standard](https://reader031.vdocuments.us/reader031/viewer/2022041416/5e1c06ff8c4073619f1506de/html5/thumbnails/5.jpg)
Kalman filter
kk
a
k
f
k uxx ),( 1Model forecast Predictor
Corrector
k
f
kkk xCz )(Measurements
))(( f
kkkk
f
k
a
kxCzGxx Update
Statistical properties:
• If the model is linear, and model and measurement errors are
independent white noise with known covariance matrices, the Kalman
filter is the best linear unbiased estimator (BLUE)
• If, in addition, model and measurement errors are Gaussian, the Kalman
filter is the maximum a posteriori (MAP) or maximum likelihood (ML)
estimator
![Page 6: Data Assimilation in Environmental Modellinghydrocast.dhigroup.com/publications/DataAssimilation_HOBEworkshop... · for data assimilation • OpenDA is an open interface standard](https://reader031.vdocuments.us/reader031/viewer/2022041416/5e1c06ff8c4073619f1506de/html5/thumbnails/6.jpg)
Application in environmental modelling
Challenges
• Non-linear model dynamics
• Non-Gaussian and possible biased model and measurement errors
• Unknown model and measurement error statistics
• High-dimensional systems
Solutions
• Steady-state Kalman filter
– Kalman gain calculated ”off-line”
– Assumes constant model and measurement error statistics
• Ensemble-based Kalman filters
– Covariance matrix represented by an ensemble of states (ensemble
size << dimension of state)
– Error propagation according to full non-linear model dynamics
– Sampling uncertainty decreases slowly with increasing ensemble size
![Page 7: Data Assimilation in Environmental Modellinghydrocast.dhigroup.com/publications/DataAssimilation_HOBEworkshop... · for data assimilation • OpenDA is an open interface standard](https://reader031.vdocuments.us/reader031/viewer/2022041416/5e1c06ff8c4073619f1506de/html5/thumbnails/7.jpg)
Joint state updating and model forcing
correction
• Model forcing main error source
• Temporal correlation model (e.g. AR(1) model) of model forcing included in filtering scheme using augmented state vector formulation
• Model forcing error updated along with the model state
state variables
parameters
Input Variables
1
Measured Variables
2 3 4
Forecast or
Output Variables
state variables
parameters
Input Variables
1 1
Measured Variables
2 2 3 3 4 4
Forecast or
Output Variables Numerical Model Data
assimilation
procedure
![Page 8: Data Assimilation in Environmental Modellinghydrocast.dhigroup.com/publications/DataAssimilation_HOBEworkshop... · for data assimilation • OpenDA is an open interface standard](https://reader031.vdocuments.us/reader031/viewer/2022041416/5e1c06ff8c4073619f1506de/html5/thumbnails/8.jpg)
Reference
False
Twin test experiment
• False run: Model forced with
erroneous boundary
conditions
• Update of false model using
water level measurements
at two locations (from
reference run)
Example:
MIKE FLOOD
![Page 9: Data Assimilation in Environmental Modellinghydrocast.dhigroup.com/publications/DataAssimilation_HOBEworkshop... · for data assimilation • OpenDA is an open interface standard](https://reader031.vdocuments.us/reader031/viewer/2022041416/5e1c06ff8c4073619f1506de/html5/thumbnails/9.jpg)
Reference False
Update
![Page 10: Data Assimilation in Environmental Modellinghydrocast.dhigroup.com/publications/DataAssimilation_HOBEworkshop... · for data assimilation • OpenDA is an open interface standard](https://reader031.vdocuments.us/reader031/viewer/2022041416/5e1c06ff8c4073619f1506de/html5/thumbnails/10.jpg)
Reference
False
Update
![Page 11: Data Assimilation in Environmental Modellinghydrocast.dhigroup.com/publications/DataAssimilation_HOBEworkshop... · for data assimilation • OpenDA is an open interface standard](https://reader031.vdocuments.us/reader031/viewer/2022041416/5e1c06ff8c4073619f1506de/html5/thumbnails/11.jpg)
Joint state updating and bias correction
• Bias aware Kalman filter
– Include bias using augmented state formulation
– Separate bias Kalman filter (Dual Kalman filter)
Drecourt et al. (2006)
![Page 12: Data Assimilation in Environmental Modellinghydrocast.dhigroup.com/publications/DataAssimilation_HOBEworkshop... · for data assimilation • OpenDA is an open interface standard](https://reader031.vdocuments.us/reader031/viewer/2022041416/5e1c06ff8c4073619f1506de/html5/thumbnails/12.jpg)
Twin-test experiment: Groundwater model
![Page 13: Data Assimilation in Environmental Modellinghydrocast.dhigroup.com/publications/DataAssimilation_HOBEworkshop... · for data assimilation • OpenDA is an open interface standard](https://reader031.vdocuments.us/reader031/viewer/2022041416/5e1c06ff8c4073619f1506de/html5/thumbnails/13.jpg)
state variables
parameters
Input Variables
1
Measured Variables
2 3 4
Forecast or
Output Variables
state variables
parameters
Input Variables
1 1
Measured Variables
2 2 3 3 4 4
Forecast or
Output Variables Numerical Model Data
assimilation
procedure
• Include model parameter estimation in Kalman filter
− Augmented state vector formulation
− Dual Kalman filter formulation
Joint state updating and parameter
estimation
![Page 14: Data Assimilation in Environmental Modellinghydrocast.dhigroup.com/publications/DataAssimilation_HOBEworkshop... · for data assimilation • OpenDA is an open interface standard](https://reader031.vdocuments.us/reader031/viewer/2022041416/5e1c06ff8c4073619f1506de/html5/thumbnails/14.jpg)
Jørn Rasmussen: Data assimilation in hydrological models – evaluation of filter
performances (Poster)
Twin-test experiment: Groundwater model
Perturbed EnKF Deterministic EnKF Ensemble transform
![Page 15: Data Assimilation in Environmental Modellinghydrocast.dhigroup.com/publications/DataAssimilation_HOBEworkshop... · for data assimilation • OpenDA is an open interface standard](https://reader031.vdocuments.us/reader031/viewer/2022041416/5e1c06ff8c4073619f1506de/html5/thumbnails/15.jpg)
state variables
parameters
Input Variables
1
Measured Variables
2 3 4
Forecast or
Output Variables
state variables
parameters
Input Variables
1 1
Measured Variables
2 2 3 3 4 4
Forecast or
Output Variables Numerical Model Data
assimilation
procedure
• In practice model innovations are not white noise
• Hybrid filtering and error forecasting includes forecast of model
innovation
Hybrid filtering and error forecasting
![Page 16: Data Assimilation in Environmental Modellinghydrocast.dhigroup.com/publications/DataAssimilation_HOBEworkshop... · for data assimilation • OpenDA is an open interface standard](https://reader031.vdocuments.us/reader031/viewer/2022041416/5e1c06ff8c4073619f1506de/html5/thumbnails/16.jpg)
Hybrid filtering and error forecasting
Innovation forecast
Time in forecasting period
State variable
Time in forecasting period
Model prediction
Updated
Hypothetical measurement
• Error forecast model applied to forecast innovation in measurement points -> virtual measurement
• Filtering using virtual measurements
Innovation forecast
Time in forecasting period
State variable
Time in forecasting period
Model prediction
Updated
Hypothetical measurement
Madsen and Skotner (2005)
![Page 17: Data Assimilation in Environmental Modellinghydrocast.dhigroup.com/publications/DataAssimilation_HOBEworkshop... · for data assimilation • OpenDA is an open interface standard](https://reader031.vdocuments.us/reader031/viewer/2022041416/5e1c06ff8c4073619f1506de/html5/thumbnails/17.jpg)
Regularisation
Challenges with ensemble based Kalman filtering methods
• Computationally expensive
• Introduces sampling uncertainties in covariance and Kalman gain
estimates (spurious correlations)
• Performance sensitive to description of model and measurement errors
(bias in error covariance estimation)
Include regularisation to provide more robust and computationally
efficient filtering schemes
![Page 18: Data Assimilation in Environmental Modellinghydrocast.dhigroup.com/publications/DataAssimilation_HOBEworkshop... · for data assimilation • OpenDA is an open interface standard](https://reader031.vdocuments.us/reader031/viewer/2022041416/5e1c06ff8c4073619f1506de/html5/thumbnails/18.jpg)
Regularisation methods
• Distance regularisation (localisation)
– Update state only in local region around measurement
– Reduces the effect of spurious correlation in data sparse regions
10,)1(1
k
smooth
k
smooth
kKKK
• Covariance or Kalman gain smoothing
– Temporal smoothing of covariance or Kalman gain
– Reduces the effect of sampling uncertainty
– But fade out high-frequency variability
= 0: Steady-state Kalman filter
= 1: Normal Kalman filter
Sørensen et al. (2004)
![Page 19: Data Assimilation in Environmental Modellinghydrocast.dhigroup.com/publications/DataAssimilation_HOBEworkshop... · for data assimilation • OpenDA is an open interface standard](https://reader031.vdocuments.us/reader031/viewer/2022041416/5e1c06ff8c4073619f1506de/html5/thumbnails/19.jpg)
Regularisation: Application example
With regularisation
Without regularisation
No DA Forecast skills
RMSE [m] as
function of lead
time
Assimilation
stations
Validation
stations
![Page 20: Data Assimilation in Environmental Modellinghydrocast.dhigroup.com/publications/DataAssimilation_HOBEworkshop... · for data assimilation • OpenDA is an open interface standard](https://reader031.vdocuments.us/reader031/viewer/2022041416/5e1c06ff8c4073619f1506de/html5/thumbnails/20.jpg)
Implementation
• OpenDA-OpenMI open framework
for data assimilation
• OpenDA is an open interface
standard for data assimilation and
offers a number of ensemble based
algorithms
• OpenMI is an open source
standard model interface
Marc-Etienne Ridler: Open Framework for
Data Assimilation (Poster)
![Page 21: Data Assimilation in Environmental Modellinghydrocast.dhigroup.com/publications/DataAssimilation_HOBEworkshop... · for data assimilation • OpenDA is an open interface standard](https://reader031.vdocuments.us/reader031/viewer/2022041416/5e1c06ff8c4073619f1506de/html5/thumbnails/21.jpg)
Concluding remarks
• A general filtering data assimilation framework
– Allows joint updating and estimation of model state, forcing,
parameters and bias
– Can be combined with error forecasting in measurement points
• Kalman filtering with regularisation efficient for real-time applications
• Hybrid filtering and error forecasting procedures can increase forecast
skills for longer lead times
• Challenges
– Representation of model and measurement uncertainty
– Computational efficiency
– Multi-variate data assimilation in integrated models using in-situ and
remote sensed data