bridging the gap in operational hydrologic data …rto) transition plans for the hydrologic ensemble...

1 Data Assimilation Workshop, Delft Nov 1-3, 2010

Bridging the gap in operational

hydrologic data assimilation -

Challenges and a way forward

D.-J. Seo1, Yuqiong Liu2,3, Haksu Lee2,4, Victor Koren2, Jiarui

Dong5,6, Michael Ek5, Pedro Restrepo2

1The University of Texas at Arlington, Arlington, TX, USA

2NWS/Office of Hydrologic Development, Silver Spring, MD, USA 3Riverside Technology, Inc., Fort Collins, CO, USA

4University Corporation for Atmospheric Research, Boulder, CO, USA 5NWS/National Centers for Environmental Prediction, Camp Springs, MD, USA

6I. M. Systems Group, Inc., Rockville, MD, USA

Acknowledgments APRFC

And many collaborators in the research and operational communities

In this presentation

• Context of this presentation

• DA in operational hydrology in NWS

– Past, present

• Lessons learned

• Needs

• Way forward

– Strategy, opportunities

Community Hydrologic Prediction

System (CHPS)

• Modular software to enhance collaboration and accelerate R2O

• Extension of the Flood Early Warning System (FEWS) architecture:

– Incorporates NWS models with models from FEWS, U.S. Army

Corps of Engineers (ACE), and academia

Flexible, open modeling architecture linking program elements

Implementation Status: AWIPS-II compatible prototype

hardware and software for all RFCs

Conducting parallel operations at 4

RFCs, remaining by early 2011

Retire legacy system in early 2012 FEWS

Models

From Carter (2010)

CHPS and Hydrologic Data Assimilation (DA)

From http://www.weather.gov/oh/hrl/chps/index.html

DA planning with the RFCs (2008)

• Develop Phase 2 R&D and research-to-operations

(RTO) transition plans for the Hydrologic Ensemble

Forecast System (HEFS)

• Identify ensemble and DA capabilities that may address

new and existing service needs (particularly in light of

recent experiences and emerging issues)

– Implementable in CHPS

– Steer enhancement/evolution of FEWS

– Strengthen collaborations with the external research

community

(Extra) Motivation

• One can only forecast based on all available information

– There are/will be more sources of information

– There are/will be significant additions to the forecast process

• They will require additional attention and intervention, and

expertise and judgment of human forecasters

• Making dynamically and statistically coherent/consistent sense

out of them is and will be a large challenge

• It is (past) time to rebalance what human forecasters (should) do and

what computers (should) do

– Forecaster role: Master analyzer, interpreter, judge and translator of

all available hydrologic, hydrometeorological and

hydroclimatological information

• In CHPS, we have a “miss-it-or-lose-it” window of opportunity to do this

rebalancing toward realizing the integrated water science and services

vision

A brief history of DA in NWS

Blending (Adjust-Q)

• Practiced ever since river forecasting began

• Merges the observed hydrograph with the simulated

• Does not correct the underlying cause of the discrepancy

• Is objective and easy to implement

• Cannot correct for certain common sources of errors

such as erroneous precipitation reports or timing errors

CHAT (Computed Hydrograph

Adjustment Technique)

• Sittner, W. T. and Krouse, K. M. (1979) Improvement of hydrologic

simulation by utilizing observed discharge as an indirect input

(Computed Hydrograph Adjustment Technique-CHAT). NOAA Tech.

Memo. NWS HYDRO-38, US Dept of Commerce, Silver Spring.

• Applicable to runoff events not involving snow and to headwater basin

outflow hydrographs

• Adjusts the precipitation input and modifies the shape of the unit

hydrograph until the simulation is in satisfactory agreement with the

discharge observation

• Utilizes a unique objective function that models, to some degree, the

thought processes which a human forecaster uses in judging the

seriousness of a disagreement between the rising limb of a simulated

hydrograph and the discharge observations

– Unlike conventional optimizing techniques, does not seek to

minimize the objective function, but rather, attempts to reduce it to

an acceptable value

Assimilation of snow course data

• Carroll, T. R. (1979) A procedure to incorporate snow course data

into the National Weather Service River Forecast System.

Proceedings, Modeling of Snow Cover Runoff (AGU, AMS, Corps of

Engineers and NWS, Hanover, New Hampshire, September 1978),

pp. 351-358.

• Incorporates snow course data into the NWSRFS

• Weights the simulated and observed water equivalent values

• Intended primarily to reduce monthly volume errors

• Is of particular value where the precipitation data do not accurately

represent the snow accumulation at higher elevations

– Basins where precipitation stations are located primarily at

elevations lower than the snow courses

State updating via Kalman filter

• Kitanidis, P. K. and Bras, R. L. (1978) Real time forecasting of river flows.

Ralph M. Parsons Laboratory for Water Resources and Hydrodynamics,

Dept of Civil Engineering, MIT, TR235.

• Uses SAC (re-)written in continuous state-space form with certain

modifications

– Linearized using various techniques and then integrated to obtain the

state transition matrix for a discrete time Kalman filter

• An automatic procedure to estimate the model noise covariance matrix and

input noise covariance

• A generalized likelihood ratio test to detect input errors

• Limitations

– Considered only SAC with a crude procedure to time-distribute channel

inflow and route to the gauging station

– Considered the model and input noise covariance to be constant

– The procedure used to identify input errors did not consider the pure

timing error

SS-SAC (State-Space Sacramento)

• NWSRFS Operation No. 22

• Georgakakos, K. P., and J. A. Sperflage, 1995: Hydrologic Forecast

System - HFS: A user’s manual. HRC Tech. Note 1, Hydrologic

Research Center, San Diego, CA, 17 pp.

• Performs real time updating of SAC and channel-routing model

storage elements from observations of discharge, and provides

discharge forecast-error variance

• Uses user-supplied degree-of-belief estimates for the variance of

SAC parameters and of forecast evapotranspiration demand and

precipitation or rain-plus-melt input to SAC

• The channel model component is a conceptual kinematic channel

routing model consisting of a series of nonlinear or linear reservoirs

– In the latter case, the unit hydrograph coordinates may be used

to estimate the channel-routing model parameters

ASSIM (Assimilator Operation)

• NWSRFS Operation No. ?

• Koren, V. and Schaake, J. (1993). Nile Technical Note #147:

Updating Algorithm and Program.

• A soil moisture updating technique which modifies current soil

moisture states

• Uses the iterative Rosenbrock Optimization Technique

(Rosenbrock, 1960) to estimate the current 'optimal' soil moisture

states

• Initial soil moisture states and input precipitation are varied over the

run period and an objective function that includes the difference

between observed and simulated flows, precipitation adjustments

and state adjustments, is minimized

SEUS (Snow Estimation & Updating System)

• McManamon, A., R. K. Hartman, and R. Hills, "Implementation of the

Snow Estimation and Updating System (SEUS) in the Clearwater

River Basin, Idaho", Proceedings: 63rd Annual Western Snow

Conference, pp. 56-65, Sparks, NV., April 1995.

• Consists of three components – calibration, operational, updating

– The updating component

• Modifies the existing snow water equivalent states of the

conceptual snow model within NWSRFS based on the

weighted contributions of the simulated model snow states

and the estimates of the snow states developed using the

snow observations

• The weighting for each estimate reflects the relative

uncertainty of the estimate due to such factors as model bias

and measurement error

Snow Updating System

• Snow updating user manual, Version 2.00.02 Updated 2003-09-26,

Riverside Technology, Inc.

• An extension to NWSRFS and provides tools to update the snow conditions

• Consists of a graphical user interface (GUI) and batch programs, which are

based on a Principal Components Analysis (PCA) program

– The batch system can process all or a subset of the basins in a system

– The GUI simplifies running a subset of the system and also provides

displays to view the data and results

• Initially consisted only of the batch snow update software, which was

implemented by Riverside Technology, inc. for the Bonneville Power

Administration. The GUI was subsequently added, using BPA support and

technologies developed by RTi

• The PCA software was developed by Dave Garen of the USDA

Recent prototype development

• 2DVAR

– State updating for SAC-UHG

– Seo et al. (2003)

– Implemented in the Site Specific Hydrologic Prediction (SSHP)

system, used at a number of WFOs and RFCs

• 4DVAR

– State updating of gridded SAC and kinematic-wave routing in

Research Distributed Hydrologic Model (RDHM)

– Lee et al. (submitted to AWR)

• 1DVAR

– Real-time updating of 3-parameter Muskingum routing

– Under integration in CHPS via OpenDA

2DVAR - Comparative evaluation

FA VAR-aided SAC-UHG forecast

FC MODa-aided SAC-UHG forecast

aRun-time modification by human forecaster

Seo, D.-J., L. Cajina, R. Corby and T. Howieson, 2009:

Automatic State Updating for Operational Streamflow

Forecasting via Variational Data Assimilation, 367,

Journal of Hydrology, 255-275.

From “When NWSRFS, SEUS, and Snow

Intersect.” Vernon C. Bissell, Western Snow

Conference, Bend, Oregon, 1996.

From Snow updating user manual (2003)

Operation #RFCs

ADD/SUB 13

ADJUST-H 1

ADJUST-Q 13

ADJUST-T 2

API-CONT 1

API-HFD 1

API-MKC 1

BASEFLOW 4

CHANGE-T 13

CHANLOSS 11

CLEAR-TS 13

CONS_USE 3

DELTA-TS 3

DWOPER 3

FFG 10

FLDWAV 6

GLACIER 1

LAG/K 13

LAY-COEF 1

LIST-FTW 2

LOOKUP 11

LOOKUP3 6

MEAN-Q 13

MERGE-TS 12

MULT/DIV 4

MUSKROUT 3

NOMSNG 9

PLOT-TS 8

PLOT-TUL 13

RES-SNGL 11

RSNWELEV 4

SAC-SMA 12

SARROUTE 1

RES-J 7

SET-TS 9

SNOW-17 10

SSARRESV 1

SS-SAC 1

STAGE-Q 13

STAGEREV 1

TATUM 2

TIDEREV 2

UNIT-HG 13

WEIGH-TS 10

Reality NWSRFS Operations in use (as of October 2007)

• Snow Updating System used at NWRFC

• 2DVAR, implemented in the Site-Specific Prediction System (SSHP), is used by various WFOs and RFCs

2DVAR - Lessons learned

• Use the same, operational models (soil moisture accounting, snow, routing, etc.)

– Model physics and parameters must be the same and completely transferable

• Allow forecaster control/intervention

– To reflect any prior or additional information that the forecaster may have

– Restart (warm or cold) may be necessary if the model deviates from the real world

• Provide, and effectively present, model-dynamical information that explains the DA results

• Clearly demonstrate the value of DA through objective comparative verification

– In the context of the end-to-end forecast process

– Relative to the current operational practice

• Training

From MBRFC’s presentation in XEFS Phase 2 R&D and RTO Planning (2008)

• Levee failures

Accounting for timing errors

due to spatially non-uniform

distribution of rainfall and/or

varying rainfall intensity

From MBRFC’s presentation in XEFS Phase 2 R&D and RTO Planning (2008)

For forecaster control

• For forecaster control of the end-to-end DA process, versatile,

informatics-based graphical user interface would be

necessary that allows, e.g., displays of with- and without-DA

results over multiple time periods for pattern identification.

• It is possible that, at times, the forecaster may have to restart

the DA process, going back to some user-specified time and

negating any changes thereafter. The DA tools should be

flexible enough to allow such forecaster-controlled warm

restarts.

From NWS/OHD Strategic Science Plan (2010)

Science and technology

challenges • End-to-end DA

– A suite of integrated DA tools for forcings,

hydrologic models and hydraulics models

– Decomposition, control

• Synergistically linking land DA and DA for

operational hydrologic forecasting

Closing thoughts

• Development, implementation and Institutionalization of forecaster-

supervised automatic DA in operational hydrology will require:

• Collective efforts from both the research and the operational

communities

• Cross-cutting community planning and action to capitalize on recent

advances and to seize the window of opportunity

• We, the cross-cutting community, need:

• A science roadmap to

− Lay out clear and scientifically-sound pathways to operationally-

viable practical solutions

− Identify and articulate specific scientific and technological challenges

therein

• Community-wide tools to

− Make R&D and RTO transition feasible

− Improve cost-effectiveness

− Foster cross-cutting collaborations

Adapted from Integrated Water Science Plan (NWS 2004)

What kind of tools?

Airborne

Satellite

In-Situ

physiographic

observing

systems

NASA USGS

Operational data feeds

baseline developmental stream

Once validated, new data are

fed into operational stream

Operational Data

Assimilation

OTHERS

Experimental Data

Assimilation

DA Testbed

Parameters

States

CHPS-OpenDA

Model Adapter OpenDA

MODELS: SNOW-17, SACSMA UNITHG, Lag/K, …

CHPS/FEWS

DATABASE: Forecast Observation

What kind of tools? (cont.)

Community Contribution

Models DA Tools

VAR EnKF, EnKS MLEF, PF

Thank you

For more information on the

presentation, contact:

djseo@uta.edu

Yuqiong.Liu@noaa.gov

Hydrologic Ensemble Prediction System

Ensemble Pre-Processor

Parametric Uncertainty Processor

Data Assimilator

Ensemble Post-Processor

Hydrology & Water Resources Ensemble Product Generator

Hydrology & Water Resources Models

Hydrologic Ensemble Processor

QPF, QTFQPE, QTE, Soil Moisture

Streamflow

Improved

accuracy,

Reliable

uncertainty

estimates,

Benefit-cost

effectiveness

maximized

Hydrologic Ensemble Forecast System (HEFS)

Vision for Ensemble and Data Assimilation in Hydrologic Forecast Operations

Modeling & DA

From Snow updating user manual (2003)

Snow models

Soil moisture accounting

models

Hydrologic routing

models

Hydraulic routing models

reservoir, etc., models

In-situ snow water

equivalent (SWE)

In-situ soil moisture

Streamflow or stage

Snowmelt

MODIS-derived snow cover

MODIS-derived cloud coverPrecipitation

Potential evap. (PE)

Runoff

River flow or stage

Atmospheric forcing

AMSR-derived SM1

AMSR-derived SWE1

MODIS-derived surface

temperature

1 pending assessment

SNODAS SWE

Satellite altimetry

NWS Operational DA Strategy

From Sorenson (2002)

The Red River Flood of 1997

• East Grand Forks city engineer Gary Sanders reasoned that predicting a crest at 49.0 ft in 1997 was putting the flood in the same class as the 1979 flood, which crested at 48.8 ft with a peak flow of 82,000 cfs

• The peak flow in 1997 reached 137,000 cfs

– "Missing by five ft may not sound like much, but when you talk about flow, the National Weather Service missed by almost 100 percent.“ (Sanders)

DA strategy for operational

hydrologic forecasting • Decompose

into smaller ones such that:

− The suboptimal solutions from the decomposed problems are

close to the optimal solution from the full-blown problem

− the resulting DA process is forecaster-controllable

nmnmnn

Multi-Model Soil Moisture Percentile for May 2008

D4 D3 D2 D1

Drought D0 < 20%

(a): http://www.cpc.ncep.noaa.gov/products/Drought/Drought_index/Drought_index.shtml

(a) NARR

(b) & (c): http://ldas.gsfc.nasa.gov/drought

(b) Noah

(c) Mosaic

D4 D3 D2 D1

Commonly dry regions: Southeast, Southern Plains, North Dakota, California

Snow Water Equivalent (a) 2008-06-01

http://www.nohrsc.nws.gov/nsa

(b) 2008-05-01

Snow melt providing water

over west mountain region.

Pacific Northwest: heavy

streamflow might due to

snow melt

STREAMFLOW FORECAST EXAMPLE 12-DAY LEAD-TIME (APRIL 1, 2006)

Ensemble mean

similar to analysis

– Error ~10% of flow

Ensemble spread

comparable to error

in ensemble mean

Analysis (NLDAS) Ensemble Mean

Error=Ens. Mean - analysis Ensemble Spread

From Hou et al.

Strategy for bridging/coupling land data assimilation and hydrologic assimilation

NWS’s Integrated Water Science Vision

Coupled ocean/atmospheric/

land-surface model (global)

Coupled atmospheric/land-surface

model (regional)

Interface with Estuary Models,

Water Quality Models

WEATHER &

CLIMATE FOCUS

WATER FOCUS

Observing Systems

Common Hydrologic

Land-Surface Modeling System

(Uncoupled)

Research Labs

Land Surface Hydrology & Water Resources Modeling

From Integrated Water Science Plan (NWS 2004)

stageriverV

moistsoilV

evappotenV

precipV

moistsoilX

stageriverZ

moistsoilZ

evappotenZ

precipZ

stageriverV

moistsoilV

evappotenV

precipV

moistsoilX

stageriverZ

moistsoilZ

evappotenZ

precipZ

1_____

moisturesoilV

moisturesoilVmoistsoilX

moistsoilX

stageriverV

moistsoilV

evappotenV

precipV

moistsoilX

stageriverZ

moistsoilZ

evappotenZ

precipZ

Strategy for bridging/coupling land data assimilation and hydrologic assimilation (cont.)

Predicting Floods to Droughts

In Your Neighborhood

River channel

Developed

Shrubs/Grass

Agriculture

Wetlands

Forecast Point

Forecast Basin

River Services (600 miles per forecast point)

Water Resource Services (6 square mile forecast basins)

River Conditions Soil Conditions From Carter (2006)

Feb 3, 2010

Distributed Modeling System Simulated water balance model and channel routing states over the Arkansas River basin

before and after a storm; cumulated rainfall are shown at the top

From Koren

et al. 2004

Water Issues: Too Much, Too Little, Poor Quality

• New strategic plans for NOAA and NWS recognize decision-makers in all water management sectors need:

– Higher resolution information in space and time

– Quantification of uncertainty to manage risk

• Water resource challenges are significant and getting bigger:

– Population growth and economic development are stressing water supplies and increasing vulnerability

– A changing climate is impacting water availability and quality

– Socio-economic risks of floods and droughts are escalating

• Growing needs for water resource forecasts:

– Soil moisture for agriculture and forest management

– Low flow for maintaining water supply

– Water temperature and salinity forecasts for fisheries management and healthy ecosystems

From Carter (2010)

bridging the gap in operational hydrologic data …rto) transition plans for the hydrologic ensemble...

Documents

rainfall-river forecasting: overview of noaa’s role...

seminar e: basic theory of the hefs hydrologic ensemble...

the hydrologic ensemble prediction experiment (hepex) ·...

probabilistic hydrologic forecasts for decision support...

advanced hydrologic prediction service quarterly report ·...

3rd hepex workshop - jrc publications repository:...

hydrologic ensemble prediction

hydrologic ensemble forecasting service (hefs)...• 1x...

fr hefs ad ids - cooking matters

ensemble verification system (evs) manualdeveloped by the...

mmefs operational ensemble hydrologic forecasting at … ·...

a hydrologic ensemble seasonal forecast system over the...

ensemble river stage forecasts from the site specific...

1 advancing hydrologic ensemble forecasting using...

hydrologic ensemble forecasting service (hefs)enspost, and...

hydrologic ensemble prediction for risk-based...

hydrologic model output statistics (hmos) …...hydrologic...

national weather service application of cfs forecasts in nws...

hydrologic ensemble forecasting service (hefs) · 2015. 3....

roadmap for hydrologic ensemble forecast services