hydroinformatics and some of its roles in the view of climate variability

1

AGUA 2009

Hydroinformaticsand some of its roles in the view

of climate variability

Dr. Dimitri P. SolomatineProfessor of Hydroinformatics

D.P. Solomatine. Hydroinformatics.2

Quick start: role of uncertaintyin flood management

0

10

20

30

40

50

60

70

80

1 11 21 31 41 51Ti me

Disc

harg

e

One est i mat eUpper boundLower bound

Alarm level

Prediction interval (uncertainty)

Deterministic forecast

Fore

cast

ed r

iver

dis

char

ge

So, issue a flood alarm or not?..


Climate is changing…

http://www.globalwarmingart.com/wiki/File:Holocene_Temperature_Variations_Rev_png


Global warming


Variability in annual temperatures locallySource: www.john-daly.com, based on data from NASA Goddard Institute (GISS), USA, and Climatic Research Unit (CRU) of the University of East Anglia, Norwich, UK


Climate is changing…

There are many factors leading to changes in the rate of climate change

Whatever the main reason is, the climate variations prompt fordeveloping the water management strategies that take climate uncertainties into account

the need for More observation systemsBetter predictive modelling toolsAnalytical methods to handle uncertaintyChanges in design and adaptive management practicesChanges in educational programmes at all levels

These issues are the current focus of Hydroinformatics


Encapsulation of knowledge related to water

Tacit (implicit) knowledge embedded within a personWords, texts, images

printedstored in electronic media

Mathematical modelsformulas, algorithmsalgorithms encapsulated in computer programs (software)

Integrated systems encapsulating all of above -Hydroinformatics systems


Hydroinformatics

modelling, information and communication technology, computer sciences

applied to problems of aquatic environment

with the purpose ofproper management

1991

2008


Flow of information in a Hydroinformatics system

Earth observation, monitoring

Numerical Weather Prediction Models

Data modelling, integration with hydrologic and hydraulic models

Access to modellingresults

Data Models Knowledge Decisions

Decision support

Map of flood probability


Where is data coming from?


Modellingis the heart of Hydroinformatics

02

=+−∂∂

+⎟⎟⎠

⎞⎜⎜⎝

⎛∂∂

+∂∂

fo gASgASx

hgA

A

Q

xt

Q


Modelling

Model is …a simplified description of realityan encapsulation of knowledge about a particular physical or social process in electronic form

Goals of modelling are:understand the studied system or domain (the past)predict the future use the results of modelling for making decisions (change the future)


Modelling is at heart of Hydroinformatics

Hydroinformatics deals with the technologies ensuring the whole information cycle, and integrates

data, models,

people


Main modelling paradigms

Physically-based model (process, simulation, numerical) is based on the understanding of the underlying processes

Data-driven model is based on the recorded values of variables characterising the system. They need less knowledge about the physical behaviour

Agent-based model consists of dynamically interacting relatively simple rule-based computational codes (agents)


Applications of models

River/urban flood forecasting and managementReservoir operationsSediment transport and morphologyEcology and water qualityStorm surges and coastal floodingDredging and reclamationUrban sewers and drainageWater distribution networksetc.


Example: a physically-based model of open channel flow: Saint Venant equations

The 1D continuity and momentum equations for open channel flow are also referred as Saint Venant equation

Form a pair of non-linear hyperbolic partial differential equations in Q (flow) and h (depth)

Analytically can not be solvedNumerically can be solved using

finite differences (explicit, implicit schemes), finite elements

Lqx

Q

t

A=

∂∂

+∂∂

02

=+−∂∂

+⎟⎟⎠

⎞⎜⎜⎝

⎛∂∂

+∂∂

fo gASgASx

hgA

A

Q

xt

Q

x=distance, t=time, A=cross-section, S0=bottom slope, Sf=energy grade line slope, B=width

Continuity equation

Momentum equation


Why 2D/3D modelling?Often 1D model is not enough

Horizontal velocity fields Vertical velocity fields


Some examples of using modellingin water-related issues


Warragamba Dam, Australia

Warragamba Dam - 65 km west of Sydney in the Burragorang Valley

provides the major water supply for SydneyWarragamba River flows through a 300-600 m wide gorge, about 100 m deep before opening out into a large valley. This allows a relatively short and high dam to impound a vast quantity of water.

A dam break of the WarragambaDam would be a major disaster. SOBEK (Delft Hydraulics) software was used for simulation


Warragamba Dam, AustraliaSimulation of the dam break with SOBEK by Deltares

The animation shows the simulation results. They may be used for disaster management, evacuation planning, flood damage assessment, urban planning


Models are indispensable in dealing with floods


Example: Hydroinformatics systems for flood warning – MIKE FloodWatch

MIKE Flood Watch (Danish Hydraulic Institute), a decision support system for real-time flood forecasting:

advanced time series data base MIKE 11, for hydrodynamic modelingMIKE 11 FF, real-time forecasting system, ArcView, Geographical Information System (GIS)


Hydroinformatics systems for flood warning: MIKE FloodWatch


Ecosystem Integrated Model:a Case Study for Sonso Lake, Colombia

Problem: 70% of the surface area of this shallow lake is covered by an invasive macrophite Water HyacinthCauses:

Nutrients pollution from agricultural use of landLack of sustainable management of the lake

Methodology:Ecological modelling of Water HyacinthIts integration with hydrodynamic modelAnalysis of Alternatives to Manage the Water Hyacinth Infestation


Ecosystem Integrated Model: a Case Study for Sonso Lake, Colombia

Ecosystem

Hydrodynamic

Water Quality

Sobek Rural1D2D

Sobek RuralDELWAQ

Water Hyacinth Model (coded using SOBEK RURAL Open

Process Library)

Water HyacinthNH4

NO3

Norg Porg

PO4

Organic Matter Settled

9 10

13

12

Solar Radiation

11

14

16

15

7

8

6

5Water Volume

2 3

1

4VelocityWater Depth

Flow

SEDIMENT

WATER SURFACE5

6

9

1. Input / Output2. Rainfall3. Evapotranspiration4. Advection/Dispersion

5. Input / Output6. Input / Output7. Sedimentation8. Resuspension

13. Photosynthesis14. Respiration15. Mortality16. Losses

9. Resuspension10. Hydrolysis11. Oxidation 12. Uptake/Growth

PROCESSES

Ref: MSc study by Carlos Velez (Colombia), UNESCO-IHE & Delft Hydraulics


Processes included:Growth and MortalityRespiration/PhotosynthesisTransportation by flow and windUptake/release of Nutrients from the waterMechanical, Biological and Chemical Control Options

Hydrodynamic Model 1D River and 2D Lake (Water Level)

Nutrients Model (Phosphate PO4)

Water Hyacinth Integrated Model(Plant Density)

27

Beyond “classical” modelling:current developments in Hydroinformatics

Machine learning in data-driven modelling

Multi-objective optimisation

Information theory

Predicting models’ uncertainty

Integration


Data-Driven Modelling

Uses (numerical) data (time series) describing some physical processEstablishes functions that link variables

outputs = F (inputs)Valuable when physical processes are unknownAlso useful as emulators of complex physically-based models (surrogate models)

Input dataModelled

(real)system

X

Actual (observed) output Y

Machinelearning

(data-driven)model Predicted output Y’

Learning is aimedat minimizing this

difference

Input dataModelled

(real)system

X

Actual (observed) output Y

Machinelearning

(data-driven)model Predicted output Y’

Learning is aimedat minimizing this

difference


Example of a data-driven model

observed data characterises the input-output relationship X Ymodel parameters are found by optimizationthe model then predicts output for the new input without actual knowledge of what drives Y

Linear regression model

Y = a0 + a1 X

X (e.g. rainfall)

Y (e.g., flow)

new input value

actual output value

model predicts new output value

Which model is “better”:green, red or blue?


Data-driven rainfall-runoff models: Case study Sieve (Italy)

mountaneous catchment in Southern Europearea of 822 sq. km


SIEVE: visualization of data

variables for building a decision tree model were selected on the basis of cross-correlation analysis and average mutual information:

inputs: rainfalls REt, REt-1, REt-2, REt-3, flows Qt, Qt-1

outputs: flows Qt+1 or Qt+3

FLOW1: effective rainfall and discharge data

0

100

200

300

400

500

600

700

800

0 500 1000 1500 2000 2500

Time [hrs]

Discharge [m3/s]

0

2

4

6

8

10

12

14

16

18

20

Discharge [m3/s]

Eff.rainfall [mm]

Effective rainfall [mm]


Using data-driven methods in rainfall-runoff modelling

Available data:rainfalls Rtrunoffs (flows) Qt

Inputs: lagged rainfalls Rt Rt-1 … Rt-LOutput to predict: Qt+T

Model: Qt+T = F (Rt Rt-1 … Rt-L … Qt Qt-1 Qt-A … Qtup Qt-1

up …)(past rainfall) (autocorrelation) (routing)

Questions: how to find the appropriate lags? how to build non-linear regression function F ?

Linear regression, neural network, support vector machine etc.

QQtt

QQttupup

RRtt


Artificial neural network: a universal function approximator (=non-linear regression model)

u F a a x

j= ,..., N

j oj ij ii

N

hid

inp

= +⎛

⎝⎜⎜

⎞

⎠⎟⎟

=∑

1

1

y F b b u

k= ,..., N

k ok jk ji

N

out

hid

= +⎛

⎝⎜⎜

⎞

⎠⎟⎟

=∑

1

1

There are (Ninp+1)Nhid + (Nhid+1)Nout parameters (weights) to be identified by optimisation process (training)

Hidden layer

a ij

Inputs

x 1x 2

x 3

x nOutputs

y1

y2

y3

ym

u 1x

u s

b jkweights weights

v

F(v)1

0 Non-linear sigmoid function: F(v) = 1/ (1 + e-v)


Neural network tool interface


ANN verification RMSE=11.353NRMSE=0.234COE=0.9452

MT verificationRMSE=12.548NRMSE=0.258COE=0.9331

SIEVE: Predicting Q(t+3) three hours ahead(ANN learned the relationship btw rainfall and flow)

Prediction of Qt+3 : Verification performance

0

50

100

150

200

250

300

350

0 20 40 60 80 100 120 140 160 180t [hrs]

Q [

m3/s

]

ObservedModelled (ANN)Modelled (MT)


Use of machine learning (data-driven) models in water resources

Hydrological modellingWater demand forecastingPrediction of ocean surgesModels of wind-wave interactionSedimentation modelling

Meta-models (emulating, fast models) of water systems –to replace complex physically-based models


MULTI-OBJECTIVE OPTIMIZATION

Finding variables’ values that bring the value of the “objective function” to a minimumIn water resources many problems require solving an optimization problem


Many optimization problems in water resources are multi-objective

there are several objectives that are to be optimized often they are in conflict, i.e. minimizing one does not mean minimizing another onea solution (the set of decision variables) is always a compromiseExamples:

multi-purpose reservoir operationelectricity generation vs. irrigation vs. navigability

models calibration (error minimization)models good "on average" vs. good for particular hydrologic conditions (floods)

pipe networks optimization (design and rehabilitation)costs vs. reduction of flood damage


Model-based optimization of urban drainage network

MOUSE modelling system (DHI Water and Environment)1D model of free-surface flow is used


Flo

od D

amag

e

Costs

Urban drainage system rehabilitation:use of multi-objective optimization

rehabilitation: changing pipes, creating additional storagesoptimization by multi-objective genetic algorithm: find a compromise btw. min. cost and min. damage due to flooding

Wastewater System Pipe Network Model (MOUSE)

Optimization Procedure (GLOBE, NSGA-II)

Data Processor Data Processor

Compromise optimal solutions


INFORMATION THEORY

Shannon entropy provides a mathematical framework to evaluate the amount of information contained in a data series

Average mutual information (AMI) is measure of information available from one set of data having knowledge of another set of dataAMI can be used to investigate dependencies and lag effects in time series data

2logH p p= −∑

2,

( , )AMI= ( , ) log

( ) ( )XY i j

XY i jX i Y ji j

P x yP x y

P x P y

⎡ ⎤⎢ ⎥⎢ ⎥⎣ ⎦

∑


Information theory and optimization for sensors locations for contaminant detection

in water distribution systemsThree criteria considered:

ConcentrationVolume of contaminated water delivered Time of detection

PhD research of Mr. Leonardo Alfonso, UNESCO-IHE. L. Alfonso , A. Jonoski , D.P. Solomatine. Multi-objective optimisation of operational responsesfor contaminant flushing in water distribution networks. ASCE J. Water Res. Plan.Manag., 2009.


Multi-objective optimization of sensors locations to detect contamination

20

30

40 50

60

70

80

90

100

110120

130

140

150

160

170

500

501

502

Source

Tank A

Tank B

Time of Detection

Contaminated Volume

Contaminant concentration

Locations found using different method

Location of 5 sensorsScenario: 2 sources of pollution


Average mutual information in optimizing the structure of a Neural Network model

Rainfall-runoff forecasting model:Qt+T = F (Rt Rt-1 … Rt-L … Qt Qt-1 Qt-A)

(past rainfall) (autocorrelation)

Finding optimal lags between Qt+T and rainfall Rt

0.0

0.2

0.4

0.6

0.8

1.0

0 5 10 15 20Time lags (hours)

Co

rr.

Co

ef.

0.00

0.05

0.10

0.15

0.20

0.25

0.30

AM

I

Cross-correlation Autocorrelation AMI

QQttRRtt


UNCERTAINTY

Uncertainties associated with climate change are very highDifferent IPCC scenarios lead to very different results of water modelsAny study exploring the impacts of CC needs powerful tools for analysing and predicting uncertainty


Uncertainty in flood management:evacuate?

0

10

20

30

40

50

60

70

80

1 11 21 31 41 51Ti me

Disc

harg

e



Point forecasts vs. Uncertainty bounds

900 920 940 960 980 1000 10200

500

1000

1500

2000

2500

3000

3500

4000

Time(days)

Dis

char

ge(m

3/s)


Sources of uncertainty in modelling

Inputs Model parameters Calibration data

Model

X(t) Q(t)p

y = M(x, s, θ) + εs + εθ + εx + εy


Monte Carlo simulation of parametric uncertaintyy = M(x, s, θ) + εs + εθ + εx + εy


Uncertainty analysis: issues

Most methods are aimed at analysing average model uncertainty, but not predicting it for the new inputsMost uncertainty analysis studies focus on the parametric uncertainty only. More has to be done to analyse and predict:

Input data uncertaintyResidual uncertainty (uncertainty associated with the deficiencies of the “optimal” model)

Model uncertainty is estimated. What next?:Should we combine in an ensemble several “good” models, instead of using one calibrated model?How can we predict model uncertainty for the future situations?How to communicate uncertainty to decision makers?

0

10

20

30

40

50

60

70

80

1 11 21 31 41 51Ti me

Disc

harg

e



UNEEC: Novel uncertainty prediction method

A calibrated model M of a water system is consideredM is run for the past hydrometeorological eventsIt is assumed that the errors of model M characterize the “residual” uncertainty in different situations (events)This data is used to train the machine learning model Uthat predicts the error (uncertainty) of model M, which is specific for a particular hydrometeorological event

UNEEC-M: parametric and input uncertainty is added as well

D.P. Solomatine, D.L. Shrestha. A novel method to estimate model uncertainty using machine learning techniques. Water Resources Res., 45, W00B11, doi:10.1029/ 2008WR006839, 2009.


UNEEC: fuzzy clustering and ANN in encapsulating the model uncertainty

New record. The trained f L and f U models will estimate the prediction interval

Error limits(or prediction intervals)

Flow Qt-1

Rainfall Rt-2

past records (examples in the space of inputs)

Error

Train regression (ANN) models:

PIL = fL (X)PIU = fU (X)

iμ∑

i

N

iμ∑

=1

i

N

iμ∑α

=12/

i

N

iμ∑α−

=1)2/1(

Prediction interval

Error distribution in cluster


750 775 800 825 8500

1000

2000

3000

4000

Time(day)

Ob

serv

ed

flo

w (

m3/s

)

90% prediction limits

Observed flow

Rainfall-Discharge plot

0

1000

2000

3000

4000

5000

6000

Jan-

88

May

-88

Sep

-88

Feb

-89

Jun-

89

Oct

-89

Mar

-90

Jul-

90

Nov

-90

Ap

r-91

Aug

-91

Jan-

92

May

-92

Sep

-92

Feb

-93

Jun-

93

Oct

-93

Mar

-94

Jul-

94

Dec

-94

Ap

r-95

Aug

-95

Time [days]

Run

off

[Cum

ec]

0

50

100

150

200

250

300

350

400

Pre

cip

itat

ion

[mm

]

Runoff [Cumec] Precipitation [mm]

Estimated prediction bounds: verification (Bagmati river basin, Nepal)

LZ

UZ

SM

RF

R

PERC

EA

Q=Q0+Q1Q1

Transformfunction

SP

Q0

SF

CFLUX

IN

SF – Snow

RF – Rain

EA – Evapotranspiration

SP – Snow cover

IN – Infiltration

R – Recharge

SM – Soil moisture

CFLUX – Capillary transport

UZ – Storage in upper reservoir

PERC – Percolation

LZ – Storage in lower reservoir

Qo – Fast runoff component

Q1 – Slow runoff component

Q – Total runoff

LZ

UZ

SM

RFRF

RR

PERCPERC

EAEA

Q=Q0+Q1Q1Q1

Transformfunction

SP

Q0Q0

SFSF

CFLUXCFLUX

ININ

SF – Snow

RF – Rain

EA – Evapotranspiration

SP – Snow cover

IN – Infiltration

R – Recharge

SM – Soil moisture

CFLUX – Capillary transport

UZ – Storage in upper reservoir

PERC – Percolation

LZ – Storage in lower reservoir

Qo – Fast runoff component

Q1 – Slow runoff component

Q – Total runoff


Hydroinformatics is aboutINTEGRATION

of data, models and people


HBV

Integration of atmospheric, hydro- and environmental models, data systems


Integration of models, communications and people

Internet – models on demand, distributed DSSMobile telephony – a channel for hazards warnings and advice systems

Ref: MSc by L. Alfonso (Colombia), UNESCO-IHE


Integration of Hydroinformatics systems and decision making

Multi-criteria, multi-stakeholder scenario analysisCommunication of model uncertainty to managers

Map of flood probability

0

10

20

30

40

50

60

70

80

1 11 21 31 41 51Ti me

Disc

harg

e



Education:Hydroinformatics at UNESCO-IHE,

Delft, The Netherlands


Postgraduate Education, Training Postgraduate Education, Training and Capacity Building and Capacity Building

in Water, Environment and Infrastructurein Water, Environment and Infrastructure


UNESCO-IHE: 14,000 Alumni

UNESCO-IHE Alumni Community

0 - 50 51-150 151-300 301-500 501-850 851-1200


Hydroinformatics Masters programme

Fundamentals, hydraulic, hydrologic and environmental processes Fundamentals, hydraulic, hydrologic and environmental processes

PhysicallyPhysically--based based simulation modelling simulation modelling

and toolsand tools

Information systems, GIS, communications, InternetInformation systems, GIS, communications, Internet

DataData--driven modelling driven modelling and computational and computational intelligence toolsintelligence tools

Integration of technologies, project managementIntegration of technologies, project management

Elective advanced topics

Systems analysis, Systems analysis, decision support, decision support,

optimizationoptimization

•• ArcGISArcGIS•• AccessAccess

•• SOBEKSOBEK•• RIBASIMRIBASIM•• Delft 3DDelft 3D•• SWATSWAT•• EPANETEPANET•• MOUSEMOUSE•• AquariusAquarius

•• MIKE 11MIKE 11•• HECHEC--RASRAS•• MIKE 21MIKE 21•• MIKE SHEMIKE SHE•• RIBASIMRIBASIM•• WEST++WEST++•• MODFLOWMODFLOW

•• LINGOLINGO•• GLOBEGLOBE•• BSCW BSCW •• AquaVoiceAquaVoice

•• NeuroSolutionsNeuroSolutions•• NeuralMachineNeuralMachine•• AFUZAFUZ•• WEKAWEKA

•• MatlabMatlab•• DelphiDelphiToolsTools

•• JAVA JAVA •• UltraDevUltraDev

with applications to:with applications to:-- River basin managementRiver basin management-- Flood managementFlood management-- Urban systemsUrban systems-- Coastal systemsCoastal systems-- Groundwater and Groundwater and catchment hydrologycatchment hydrology

-- Environmental systemsEnvironmental systems(options)(options)


Hydroinformatics Study Modules

Introduction to Water science and EngineeringApplied Hydraulics and hydrologyGeo-information systemsComputational Hydraulics and Information ManagementModelling theory and applicationsComputational Intelligence and Control SystemsRiver Basin ModellingFieldtrip to Florida, USASelective modelling subjects (2 modules each):

Flood risk managementUrban water systems modellingEnvironmental systems modelling

Hydroinformatics for Decision SupportGroupworkResearch proposal drafting and Special TopicsMSc research


Examples of MSc topics

Hydroinformatics for real time water quality management and operation of distribution networks, case study Villavicencio, ColombiaWater distribution modelling with intermittent supply: sensitivity analysis and performance evaluation for Bani-Suhila City, PalestineUrban Flood Warning System with wireless technology, case study of Dhaka City, BangladeshFlood modelling and forecasting for Awash river basin in EthiopiaHarmful Algal Bloom prediction, study of Western Xiamen Bay, ChinaApplication of Neural Networks to rainfall-runoff modelling in the upper reach of the Huai river basin, ChinaHeihe River Basin Water Resources Decision Support SystemDecision Support System for Irrigation Management in Vietnam1D-2D Coupling Urban Flooding Model using radar data in BangkokUsing chaos theory to predict ocean surge


A new programme is planned:International Masters in Hydroinformatics

UNESCO-IHE – UniValle-Cinara

jointly delivered byUNESCO-IHE Institute for Water Education,

Delft, The Netherlandsand

Universidad del Valle (UNIVALLE, Cinara), Cali, Colombia

and leading to the degree of Master of Science in Water Science and Engineering, specialisation in

Hydroinformatics,

accredited by the Dutch Ministry of Education

Planned to start in September 2010Fliers are available

Hidroinformáticamodelación y sistemas de información para la gestión del agua

Programa Internacional de Maestría en Ciencia


Programme structure

Period: Mid-October – mid-April. Public MSc defence and graduation –end of April

ECTS36

Location: Any of the core or the associated partners (at least the last month at UNESCO-IHE)

Block 4:MSc Thesis research

Period: Begin-September – Mid-October ECTS

10

Location: Any of the core partners (in the beginning UNESCO-IHE)

Block 3:MSc thesis proposal preparation + special topics

Thesis part

Period: Mid-January – end-August: 9 modules of the existing UNESCO-IHE WSE-HI programme (modules 4-12)

ECTS45

Location UNESCO-IHEBlock 2:Hydroinformatics theory and applications

Period: September-January

ECTS15

Location: UNIVALLE, CaliBlock 1: Fundamental subjects for hydroinformatics

Taught part


What Hydroinformatics alumni say...

the course has opened the new horizons in my professional life


Conclusion

Hydroinformatics is a unifying approach to water modelling and managementSpecialists in hydroinformatics play an integrating role linking various specialists and decision makersAccess to information by widening groups of stakeholders leads to democratisation of water servicesOne of the roles of Hydroinformatics is developing analytical methods to deal with climatic variability in modelling and management practiceFocus should be on education and training


…more data is needed…

hydroinformatics and some of its roles in the view of climate variability

Technology

hydroinformatics modelling

hydroinformatics systems7d

modelling model

modelling supporthydrologic

results of modelling

2d3d modelling

based model process

main modelling paradigms