a services oriented architecture for water resources data

A Services Oriented Architecture for Water Resources Data

David R. MaidmentCenter for Research in Water Resources

University of Texas at Austin

A Services Oriented Architecture for Water Resources Data

• CUAHSI and WATERS• WaterML and WaterOneFlow • Hydrologic Information Server• Modeling services

A Services Oriented Architecture for Water Resources Data

• CUAHSI and WATERS• WaterML and WaterOneFlow • Hydrologic Information Server• Modeling services

Ocean Sciences

What is CUAHSI?

• CUAHSI – Consortium of Universities for the Advancement of Hydrologic Science, Inc

• Formed in 2001 as a legal entity

• Program office in Washington (5 staff)

• NSF supports CUAHSI to develop infrastructure and services to advance hydrologic science in US universities

Earth Sciences

AtmosphericSciences

UCAR

CUAHSI

Unidata

HISNational Science Foundation

Geosciences Directorate

CUAHSI Member Institutions

112 US Universities as of September 2007

HISTeam

WATERSTestbed

WATERS Network Information System

NSF has funded work at 11 testbed sites, each with its own science agenda. HIS supplies the

common information system

Minnehaha Creek

SuperComputerCenters:NCSA,TACC

Domain Sciences:

Unidata, NCARLTER, CZEN

GEON

Government:USGS, EPA,

NCDC

Industry:ESRI, Kisters,

MicrosoftCUAHSI HIS

HIS Team and its Cyberinfrastructure Partners

HIS Team:Texas, SDSC,Utah, Drexel,

Duke

SupercomputerCenters:NCSA,TACC

Domain Sciences:

Unidata, NCARLTER, GEON

Government:USGS, EPA,

NCDC

Industry:ESRI, Kisters,

Microsoft

HISTeam

WATERSTestbed

WATERS Network Information System

CUAHSI HIS

HIS, WATERS and the CUAHSI Community

SupercomputerCenters:NCSA,TACC

Domain Sciences:

Unidata, NCARLTER, GEON

Government:USGS, EPA,

NCDC, USDA

Industry:ESRI, Kisters,

OpenMI

HISTeam

WATERS Testbed

WATERS Network Information System

CUAHSI HIS

International Partners

CSIRO and Bureau of MeteorologyAustralian Water Resources Information System

European CommissionWater database design and model integration (OpenMI)

HIS Goals• Hydrologic Data Access – providing better

access to a large volume of high quality hydrologic data across the nation

• Support for Observatories – integrating local observations by academic investigators with hydrologic data for a region

• Advancement of Hydrologic Science – modeling and analysis of “hydrology in a dynamic earth”

• Hydrologic Education – bringing more data into the classroom

A Services Oriented Architecture for Water Resources Data

• CUAHSI and WATERS• WaterML and WaterOneFlow • Hydrologic Information Server• Modeling services

DefinitionThe CUAHSI Hydrologic Information System (HIS) is a geographically distributed network of data sources and functions that are integrated using web services so that they operate as a connected whole.

Services Oriented Architecture• Service-oriented Architecture (SOA) is an

architectural design pattern that concerns itself with defining loosely-coupled relationships between producers and consumers.

• A major focus of Web services is to make functional building blocks accessible over standard Internet protocols that are independent from platforms and programming languages.

• The Web Services Description Language (WSDL, pronounced 'wiz-dəl' or spelled out, 'W-S-D-L') is an XML-based language that provides a model for describing Web services.

(from Wikipedia)

Defined by the World Wide Web Consortium (W3C)

Web Pages and Web Serviceshttp://www.safl.umn.edu/ http://his.safl.umn.edu/SAFLMC/cuahsi_1_0.asmx?

Uses Hypertext Markup Language (HTML)Uses WaterML

(an eXtended Markup Language for water data)

Rainfall & SnowWater quantity

and quality

Remote sensing

Water Data

Modeling Meteorology

Soil water

Water Data Web Sites

NWISWeb site output# agency_cd Agency Code# site_no USGS station number# dv_dt date of daily mean streamflow# dv_va daily mean streamflow value, in cubic-feet per-second# dv_cd daily mean streamflow value qualification code## Sites in this file include:# USGS 02087500 NEUSE RIVER NEAR CLAYTON, NC#agency_cd site_no dv_dt dv_va dv_cdUSGS 02087500 2003-09-01 1190USGS 02087500 2003-09-02 649USGS 02087500 2003-09-03 525USGS 02087500 2003-09-04 486USGS 02087500 2003-09-05 733USGS 02087500 2003-09-06 585USGS 02087500 2003-09-07 485USGS 02087500 2003-09-08 463USGS 02087500 2003-09-09 673USGS 02087500 2003-09-10 517USGS 02087500 2003-09-11 454

Time series of streamflow at a gaging station

USGS has committedto supporting CUAHSI’sGetValues function

Observation Stations

Ameriflux Towers (NASA & DOE) NOAA Automated Surface Observing System

USGS National Water Information System NOAA Climate Reference Network

Map for the US

Water Quality Measurement Sites in EPA Storet

Substantial variation in data availability from states

Data from Bora Beran, Drexel University

Water Quality Measurement Sites from Texas Commission for Environmental Quality (TCEQ)

Geographic Integration of Storet and TCEQ Data in HIS

Observations CatalogSpecifies what variables are measured at each site, over what time interval,

and how many observations of each variable are available

Point Observations Information ModelData Source

Network

Sites

Variables

Values{Value, Time, Qualifier, Offset}

USGS

Streamflow gages

Neuse River near Clayton, NC

Discharge, stage (Daily or instantaneous)

206 cfs, 13 August 2006• A data source operates an observation network• A network is a set of observation sites• A site is a point location where one or more variables are measured• A variable is a property describing the flow or quality of water• A value is an observation of a variable at a particular time• A qualifier is a symbol that provides additional information about the value• An offset allows specification of measurements at various depths in water

http://www.cuahsi.org/his/webservices.html

GetSites

GetSiteInfo

GetVariables

GetVariableInfo

GetValues

Locations

Variable Codes

Date Ranges

WaterML and WaterOneFlow

GetSiteInfoGetVariableInfoGetValues

WaterOneFlowWeb Service

Client

STORET

NAMNWIS

DataRepositories

Data

DataData

EXTRACTTRANSFORMLOAD

WaterML

WaterML is an XML language for communicating water dataWaterOneFlow is a set of web services based on WaterML

WaterOneFlow• Set of query functions • Returns data in WaterML

NWISNWIS

ArcGISArcGIS

ExcelExcel

NCARNCAR

UnidataUnidata

NASANASAStoretStoret

NCDCNCDC

AmerifluxAmeriflux

MatlabMatlabAccessAccess JavaJava

FortranFortran

Visual BasicVisual Basic

C/C++C/C++

Some operational services

CUAHSI Web ServicesCUAHSI Web Services

Data SourcesData Sources

ApplicationsApplications

Extract

Transform

Load

http://www.cuahsi.org/his/

GetValues for Dissolved Oxygenat Minnehaha Creek

http://his.safl.umn.edu/SAFLMC/cuahsi_1_0.asmx?WSDL

A Services Oriented Architecture for Water Resources Data

• CUAHSI and WATERS• WaterML and WaterOneFlow • Hydrologic Information Server• Modeling services

Hydrologic Information Server• Supports data discovery,

delivery and publication– Data discovery – how do I

find the data I want?• Map interface and

observations catalogs• Metadata based Search

– Data delivery – how do I acquire the data I want?

• Use web services or retrieve from local database

– Data Publication – how do I publish my observation data?

• Use Observations Data Model

Water Resource Regions and HUC’s

National Hydrography Dataset

NHDPlus for Region 17E

NHDPlus Reach Catchments ~ 3km2

About 1000 reach catchments in each 8-digit HUC

Average reach length = 2km 2.3 million reaches for continental US

Reach Attributes

• Slope• Elevation• Mean annual flow

– Corresponding velocity• Drainage area• % of upstream

drainage area in different land uses

• Stream order

Observations CatalogSpecifies what variables are measured at each site, over what time interval,

and how many observations of each variable are available

Hydrologic Information Server

Microsoft SQLServer Relational Database

Observations Data Geospatial Data

GetSites

GetSiteInfo

GetVariables

GetVariableInfo

GetValues

DASH – data access system for hydrologyWaterOneFlow services

ArcGIS Server

Minnehaha Creek Experimental Sitehttp://his.safl.umn.edu/DASH/

Water Temperature (°C)

Dissolved Oxygen (mg/L) Depth (m)

Dissolved Oxygen (% saturation)

Data Heterogeneity

• Syntactic mediation– Heterogeneity of

format– Use WaterML to get

data into the same format

• Semantic mediation– Heterogeneity of meaning– Each water data source

uses its own vocabulary– Match these up with a

common controlled vocabulary

– Make standard scientific data queries and have these automatically translated into specific queries on each data source

• Search multiple heterogeneous data sources simultaneously regardless of semantic or structural differences between them

Objective

NWIS

NARR

NAWQANAM-12

request

request

request

request

request

requestrequest

request

request

return

return

return

return

return

returnreturn

return

return

What we are doing now …..

Michael PiaseckiDrexel University

Semantic MediatorWhat we would like to do …..

NWIS

NAWQA

NARR

generic request

GetValues

GetValues

GetValues

GetValues

GetValues

GetValuesGetValues

GetValues

GetValues HODM

Michael PiaseckiDrexel University

Hydroseekhttp://www.hydroseek.org

Supports search by location and type of data across multiple observation networks including NWIS and Storet

A Services Oriented Architecture for Water Resources Data

• CUAHSI and WATERS• WaterML and WaterOneFlow • Hydrologic Information Server• Modeling services

• Project sponsored by the European Commission to promote integration of water models within the Water Framework Directive

• Software standards for model linking• Uses model core as an “engine”• http://www.openMI.org

OpenMI Conceptual Framework

VALUES

All values are referenced in a what-where-when framework, allowing different data resources or models to communicate data

Space, L

Time, T

Variables, V

D

An application of the data cube to integrate simulation modelsJon Goodall, Duke University

Typical model architectureApplication

User interface + engineEngine

Simulates a process – flow in a channelAccepts inputProvides output

ModelAn engine set up to represent a particular location e.g. a reach of the Thames

Engine

Output data

Input data

Model application

Run

Write

Write

Read

User interface

Accepts Provides

Rainfall(mm)

Runoff(m3/s)

Temperature(Deg C)

Evaporation(mm)

Accepts Provides

Upstream Inflow(m3/s)

Outflow(m3/s)

Lateral inflow(m3/s)

Abstractions(m3/s)

Discharges(m3/s)

River Model

Linking modelled quantities

Rainfall Runoff Model

Data transfer at run time

Rainfall runoff

Output data

Input data

User interface

River

Output data

Input data

User interface

GetValues(..)

Models for the processes

River(InfoWorks RS)

Rainfall(database)

Sewer(Mouse)

RR(Sobek-Rainfall

-Runoff)

Data exchange3 Rainfall.GetValues

River(InfoWorks-RS)

Rainfall(database)

Sewer(Mouse)

2 RR.GetValues

7 RR.GetValues

RR(Sobek-Rainfall

-Runoff)

1 Trigger.GetValues

6 Sewer.GetValues

call

data

4

5 8

9

Coupling the HIS with Models using OpenMI

ODM

ObservationsData Model

WaterOneFlow Web Services

Water Markup Language

WOF WaterML

MODFLOW HEC-RAS Others

OpenMI

SWATHSFP“academic models”

Linking the HIS with HEC-RAS via OpenMI

HIS WaterOneFlow Web Service forNWIS Realtime

Streamflow

A HEC-RAS Model

Workflow “trigger”

Using real-time streamflow from the HIS as a boundary condition for a HEC-RAS simulation.

SOAP-based Web Service for Models

HEC-RAS

USGSNWIS

WSDL

WSDLSimulation Model

Database

OpenMI Workflow

In an effort to build cyberinfrastructure for the hydrologic sciences, we are extending OpenMI to utilize models as web services.

Extending OpenMI for Distributed Computing

Connects to remote database via web

services

Connects to remote model via web services

Goal: To allow a modeler to create a workflow from OpenMI components that wrap web services.

Model linkage designed on client machine

Conclusion

The CUAHSI Hydrologic Information System (HIS) is a geographically distributed network of hydrologic data sources and functions that are integrated using web services so that they function as a connected whole.

For more information: http://www.cuahsi.org/his.html