ESoWC - ECMWF “Summer of Weather Code” 2019

A flood forecasting case study using

several machine learning models

Lukas Kugler & Sebastian Lehner

ESoWC - Mentors: Claudia Vitolo, Julia Wagemann, Stephan Siemen

2019-11-06

Modeling river discharge:

➢ possibly very large feature space

Involved parameters:

● precipitation

● snowdepth

● soil parameters

Time influence:

● easily extending to -180 days before

Spatial influence:

● catchment area, ~ 103 gridboxes of precipitation

The task: Rainfall-runoff modelling

2

©GLOWA Danube

Today‘s flood models: coupled dynamical models

with many meteorological & hydrological parameters and complex interactions

● Can statistical connections based on ML techniques with comparable skill

be established for extreme flooding events?

➔ Explorative comparison study: investigate flood forecasting

capability of ML techniques (with data from ECMWF/Copernicus)

➔ Code and documentation for interested peers

(open source and reproducible)

Motivation & goal

3

Data for training

● Predictor data from ERA5 reanalysis

via the Copernicus Climate Data Store

● Predictand data from GloFAS v2.0 reanalysis

Training used only reanalyses from physical models

→ small ‘forecast’ errors,

→ perfect-prog model

4

Climate Data Store API

Training1981 - 2005

Validation2006 - 2011

Test2012 - 2016

Methods

● Data resolution downsampled: daily sum and spatial mean

5

©GLOWA Danube

Methods

● Data resolution upsampled: daily sum and spatial mean

● Allow modeling the memory-effect by increasingly shifting & aggregating

6

Methods

● Data resolution upsampled:

daily sum and spatial mean

● Memory-effect: Shifting & aggregating

● River discharge: non-normal distribution

→ predict daily change in discharge

instead of absolute values

7

Reanalysis

day 1

ML-Model

day 2 day 3

Methods: Implications on verification

Predict daily changes in discharge

& Verify the cumulative sum of predictions

● Prediction starts at the „truth“

● Autocorrelation implies positive NSE

for constant zero predictions

→ Always compare to baseline-model

8

Reanalysis

day 1

ML-Model

day 2 day 3

Skill measures:

● RMS Error (m3/s)

● NSE = Variance reduction

compared to mean

Methods: Artificial neural net

(Time-Delay) Neural Net: Multilayer-Perceptron

Hyperparameter optimization:

● Number of hidden layers: 1 - 4

● Number of nodes per layer: [4, 8, 16, 32, 64]

● Dropout: 10% - 50%

one hidden layer, 64 nodes (1281 free parameters)

with batch size 90 days, tanh activation, 2 min training on 4 cores

9

Best setting:

x1

x2

x3

Model settings & Validation

Hyperparameter optimization (via GridSearch):

● OLS Linear Regression: no hyperparameters

● Support Vector Regression: kernel=poly, C=100, epsilon=0.01, degree=3

● Gradient Boosting Regression: n_estimators=200, learning_rate=0.1, max_depth=5

● Multilayer-Perceptron: one hidden layer, 64 nodes, batch size 90 days, tanh activation

10

Validation LR SVR GBR TDNN

RMSE [m3/s] 169 156 142 126

NSE 0.84 0.86 0.89 0.93

ML model setup // SupportVectorRegressor example

Hyperparameter optimization: via GridSearch

● kernel: rbf, poly

● C: [1, 10, 100]

● epsilon: [0.01, 0.1, 1]

● degree (if poly): [1, 2, 3, 4, 5]

11Notebook: 3.03_SupportVectorRegression.ipynb

Best setting:

Results // Model comparison in the full test period

14-day forecasts over the full test period: Best model => Time-delay neural net

12Notebook: 3.06_model_evaluation.ipynb

Results // Samples: rapid changes in discharge

● 14-day forecast samples

● Init times:

a. 30.06.2012

b. 22.10.2013

c. 03.08.2014

d. 23.08.2016

● Flooding event 2013 => more detailed case study

13Notebook: 3.06_model_evaluation.ipynb

a b c d

Results // Samples: rapid changes in discharge

metrics for all 4

sample periods together =>

14Notebook: 3.06_model_evaluation.ipynb

<= forecasts

for sample period a)

Results // Case study: 2013 European Flood

● Reruns are forecast-driven

● ML is reanalysis-driven

➔ Time-delay neural net &Support Vector Reg. perform best

➔ Grad. Boost. & Lin. Reg. Model show no significant increasein discharge (shape is quite fine, but amplitude not)

15Notebook: 3.06_model_evaluation.ipynb

Conclusion

● The ANN and SVR models

○ are able to predict an extreme flooding eventdespite their relatively simple setting,

○ may provide cheap additional information to assist forecasts.

● LR and GBR were not able to predict an adequate increase in discharge(although overall GBR performed almost as good as TD-NN and SVR)

● Additional tuning: Potential in neural nets by improving the architecture (LSTM, Embeddings)

● Open source (e.g. via Github, Jupyter Notebooks) approach helps efficient

sharing of notebooks and getting quick feedback

● small sample data set allows others to quickly reproduce results16

Open questions

● How can we ensure robust results wrt. input noise?

○ Introduce physical constraints?

● How do we test for robustness?

○ Generate test sets with the physical model,

& compare ML model to the physical one?

● Can we re-use the results from upstream river branches

or do we need to train a separate model for each gridpoint?○ Partly solved: Catchment similarity via Embeddings (Kratzert et al 2019*)

17

Github: github.com/esowc/ml_flood Nbviewer: link*Kratzert, F., Klotz, D., Shalev, G., Klambauer, G., Hochreiter, S., & Nearing, G. (2019). Benchmarking a Catchment-

Aware Long Short-Term Memory Network (LSTM) for Large-Scale Hydrological Modeling. arXiv preprint

arXiv:1907.08456.

18

AcknowledgementsBig thanks go to

● Our mentors:

Claudia Vitolo, Julia Wagemann and Stephan Siemen

for always providing helpful comments

● Esperanza Cuartero,

for communicating the results

● The department at the University of Vienna

for office and server access

● ECMWF-Copernicus,

for setting up ESoWC and providing such an opportunity!

Github: github.com/esowc/ml_flood ESoWC Twitter: @esowc_ecmwf

19

Appendix

Data // Preprocessing

CDO (Climate data operator) used for processing raw (hourly) data

Generate:

● daily averages

● daily accumulated precipitation

● extract data from bounding box (spatial region)

● merge into one .nc file

20

Setup and doc (mpimet): link

Python implementation: https://pypi.org/project/cdo/

Notebook: 1.04_preprocessing_with_cdo.ipynb

Feature selection // Heatmap

● Filter method: Correlation analysis

21Notebook: 2.01_feature_selection.ipynb

Feature selection // Relevant parameters

● Filter method: Correlation analysis (Heatmap)

○ precipitation (convective as well as large-scale)

○ time-lagged precipitation

○ runoff

○ soil water content layer 1

★ Total column water vapor high correlation, BUT collinearity with

precipitation!

★ Time-lagged runoff high corr, BUT collinearity to itself (non time-lagged)!

22Notebook: 2.01_feature_selection.ipynb

Feature selection // Engineering

● Only consider data in the given catchment

● Average features

● Shift time (time-lagged features)

● Specially treated: Precipitation (large-scale)

○ Aggregation over time large time periods

○ captures long memory effects

○ up to 180 days into the past

23Notebook: 2.05_prepare_data_for_the_models.ipynb

ML model setup // Overview

Models:

● LinearRegressionModel via scikit learn

● SupportVectorRegressor via scikit learn

● GradientBoostingRegressor via scikit learn

● Time-Delay Neural Net via Keras

24

Training1981 - 2005

Validation2006 - 2011

Test2012 - 2016

ML model setup // LinearRegressionModel

Hyperparameter optimization:

● Standard OLS Linear Regression does not have any hyperparameters ✓

25Notebook: 3.02_LinearRegression.ipynb

RMSE NSE

169 m3/s 0.84

ML model setup // SupportVectorRegressor

Hyperparameter optimization: via GridSearch

● kernel: rbf, poly

● C: [1, 10, 100]

● epsilon: [0.01, 0.1, 1]

● degree (if poly): [1, 2, 3, 4, 5]

26Notebook: 3.03_SupportVectorRegression.ipynb

Best setting:

kernel C epsilon degree RMSE NSE

poly 100 0.01 3 156 m3/s 0.86

ML model setup // SupportVectorRegressor

Hyperparameter optimization: via GridSearch

● kernel: rbf, poly

● C: [1, 10, 100]

● epsilon: [0.01, 0.1, 1]

● degree (if poly): [1, 2, 3, 4, 5]

27Notebook: 3.03_SupportVectorRegression.ipynb

Best setting:

kernel C epsilon degree RMSE NSE

poly 100 0.01 3 156 m3/s 0.86

ML model setup // GradientBoostingRegressor

Hyperparameter optimization: via GridSearch

● n_estimators: [10, 50, 100, 200, 300]

● learning_rate: [0.05, 0.1, 0.2]

● max_depth: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15]

28Notebook: 3.04_GradientBoost.ipynb

n_estimators learning_rate max_depth RMSE NSE

200 0.1 5 142 m3/s 0.89

Best setting:

ML model setup // GradientBoostingRegressor

Hyperparameter optimization: via GridSearch

● n_estimators: [10, 50, 100, 200, 300]

● learning_rate: [0.05, 0.1, 0.2]

● max_depth: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15]

29Notebook: 3.04_GradientBoost.ipynb

Best setting:

n_estimators learining_rate max_depth RMSE NSE

200 0.1 5 133.82 0.91

ML model setup // Time-Delay Neural Net

Hyperparameter optimization:

● Number of hidden layers: 1 - 4

● Number of nodes per layer: [4, 8, 16, 32]

● Dropout: 10% - 50%

one hidden layer, 64 nodes (1281 free parameters)

with batch size 90 days, tanh activation, 2 min training on 4 cores

30Notebook: 3.05_Time-delay_neural-net.ipynb

RMSE NSE

126 m3/s 0.93

Best setting:

x1

x2

x3

Outlook

● Incorporate stationary information

○ e.g. catchment specific variables

in a CNN or an LSTM as Embedding

(see e.g. Kratzert et al 2019*)

● Explore different kinds of catchments

● Compare more extreme events with

forecast reruns and analyse them

● Coupled model: Concept idea

31

Github: github.com/esowc/ml_flood Nbviewer: link*Kratzert, F., Klotz, D., Shalev, G., Klambauer, G., Hochreiter, S., & Nearing, G. (2019). Benchmarking a Catchment-

Aware Long Short-Term Memory Network (LSTM) for Large-Scale Hydrological Modeling. arXiv preprint

arXiv:1907.08456.