implementation of model data interoperability for ioos: successes and lessons learned
DESCRIPTION
Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned. Rich Signell USGS Woods Hole, MA / NOAA Silver Spring USA. Model Data Interoperability is Low-Hanging Fruit!. US Integrated Ocean Observing System (IOOS ® ). IOOS ® Plan defines: Global Component - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/1.jpg)
Implementation of Model Data Interoperability for IOOS:
Successes and Lessons Learned
Rich Signell USGS Woods Hole, MA / NOAA Silver Spring USA
Model Data Interoperability
is Low-Hanging Fruit!
![Page 2: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/2.jpg)
US Integrated Ocean Observing System (IOOS®)
IOOS® Plan defines:• Global Component• Coastal Component
17 Federal Agencies 11 Regional Associations
![Page 3: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/3.jpg)
HF radar
DMAC approved services for Gridded Data and Model Outputs
Table ofContents
Metadata
Data Values
DataProvider
CF Metadata
QualityControl
satellite
griddedmodeloutputs
OGC WCS(Web
CoverageService)
OPeNDAP/CF
![Page 4: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/4.jpg)
The Problem: Stovepiped Model Access
UCLA/JPL ROMS SCB Model Yi Chao, JPL
ChesROMS Model Wen Long, UMaryland
![Page 5: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/5.jpg)
Result: Inability to compare and assess model results & underutilization of model products
The GoMOOS Nowcast/Forecast Circulation Model (University of Maine)
![Page 6: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/6.jpg)
Issue: Ocean grids are not regularly spaced!
Stretched surface and terrain following vertical coordinates
Curvilinear orthogonal horizontal coordinates
![Page 7: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/7.jpg)
Climate and Forecast (CF) Metadata Conventions
Groups adopting CF:GO-ESSP: Global Organization for Earth System Science Portal IOOS-DMAC: Integrated Ocean Observing SystemESMF: Earth System Modeling FrameworkOGC: Open Geospatial Consortium (via WCS netcdf encoding)
![Page 8: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/8.jpg)
CF-Compliant Ocean Model Files_rho:standard_name = "ocean_s_coordinate" ;s_rho:formula_terms = "s: s_rho eta: zeta depth: h a: theta_s b: theta_b depth_c: hcs_rho:field = "s_rho, scalar" ;
double s_w(s_w) ;s_w:long_name = "S-coordinate at W-points" ;s_w:valid_min = -1. ;s_w:valid_max = 0. ;s_w:standard_name = "ocean_s_coordinate" ;s_w:formula_terms = "s: s_w eta: zeta depth: h a: theta_s b: theta_b depth_c: hc" ;s_w:field = "s_w, scalar" ;
float salt(ocean_time, s_rho, eta_rho, xi_rho) ;salt:long_name = "averaged salinity" ;salt:units = "PSU" ;salt:time = "ocean_time" ;salt:coordinates = "lat_rho lon_rho" ;salt:field = "salinity, scalar, series" ;
float temp(ocean_time, s_rho, eta_rho, xi_rho) ;temp:long_name = "averaged potential temperature" ;temp:units = "Celsius" ;temp:time = "ocean_time" ;temp:coordinates = "lat_rho lon_rho" ;temp:field = "temperature, scalar, series" ;
float u(ocean_time, s_rho, eta_u, xi_u) ;u:long_name = "averaged u-momentum component" ;u:units = "meter second-1" ;
![Page 9: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/9.jpg)
NetCDF Java Library
![Page 10: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/10.jpg)
NcML XML markup language for NetCDF
<aggregation dimName="time" type="joinExisting"> <netcdf location="http://rocky.umeoce.maine.edu/cdfs/gomoos.20050509.cdf" /> <netcdf location="http://rocky.umeoce.maine.edu/cdfs/gomoos.20050510.cdf" /> </aggregation> <variable name="elev"> <attribute name="coordinates" value="x y" /> </variable> <variable name="depth"> <attribute name="coordinates" value="x y" /> </variable> <variable name="temp"> <attribute name="coordinates" value="x y zpos" /> </variable> <variable name="time"> <attribute name="units" value="days since 2002-01-01 00:00 UTC" /> </variable> <variable name="zpos"> <attribute name="standard_name" value="ocean_sigma_coordinate" /> <attribute name="formula_terms" value="sigma: zpos eta: elev depth: depth" /> <attribute name="positive" value="up" /> <attribute name="units" value="1" /> <attribute name="axis" value="Z" /> <values> 0 -0.002 -0.004 -0.008 -0.016 -0.024 -0.032 -0.04 -0.06 -0.08 -0.10 -0.12 -0.14 -0. </variable> <attribute name="Conventions" value="CF-1.0" />
![Page 11: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/11.jpg)
Data Interoperability Model
![Page 12: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/12.jpg)
NJ Toolkit for Matlab http://njtbx.sourceforge.net
• Objective: Make it simple to access CF data• Example function:• [t, geo]=nj_tslice(URI,’temp’,1);• t = 22x120x180 single• geo =
– lat: [120x180 single] – lon: [120x180 single]– z: [22x120x180 double]– time: 733582 (matlab datenum)
• nj_tslice works identically for ROMS, POM, ECOM, WRF, Wavewatch3
• URI can be: local NetCDF, remote NetCDF, NcML, OpenDAP Data URL
• Coding by Sachin Kumar Bhate MSSTATE
![Page 13: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/13.jpg)
Comparing Models with Data in Matlab
Model 1: UMASS-ECOM Model 2: UMAINE-POM Data: SST 2008-Sep-08 07:32
![Page 14: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/14.jpg)
Buoy_comp.m: Model/Data comparison of Wave Height
![Page 15: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/15.jpg)
Browser-based Mapping with ncWMS/Godiva2
![Page 16: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/16.jpg)
3D Visualization with the Integrated Data Viewer (IDV)
![Page 17: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/17.jpg)
Single Point of Access Catalog:GEO-IDE => IOOS Regions
http://ferret.pmel.noaa.gov/thredds/geoideCatalog.html
![Page 18: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/18.jpg)
Working with Data in ArcGIS using the NOAA/ASA Environmental Data Connector
![Page 19: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/19.jpg)
14 Different Ocean Forecast Models Spanning CONUS Waters in IDV
![Page 20: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/20.jpg)
OOI-CI OSSE (Sep-Oct, 2009)
![Page 21: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/21.jpg)
Models being used for BP Spill
![Page 22: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/22.jpg)
Successes
• Existing Unidata technology allowed for standards-based delivery of aggregated geospatial data from native model grids that put little or no effort on the data provider
• Demonstrated interoperability with 17 THREDDS servers in 11 IOOS Regions
• What did we get? – Single point of access for regional model results– Interoperable software– More eyes on the model results– More usage of model results– Faster feedback to modelers– Improved models– Increased community support for standards-based access
![Page 23: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/23.jpg)
What Next?
• Continue work on a high-level scientific toolbox for standardized access to model data. (Matlab toolbox based on NetCDF-Java)
• Develop unstructured grid support: Common Data Model implemented in NetCDF-Java with methods comparable to structured grid
• Applications need to be modified to read standardized data via OPeNDAP instead of home-grown files
• Work on discovery and documentation
![Page 24: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/24.jpg)
Lessons Learned
• Honor the existing practices• Users are best served by developing
infrastructure for Developers• Focus on success: find something that works,
clone it, and build off it
![Page 25: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/25.jpg)
Questions, Feedback?
Dr. Richard P. Signell+1 (508) 457-2229
![Page 26: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/26.jpg)
Role of Regions
• Develop customized THREDDS catalogs for regional users
• Report status of services (e.g. NetCheck)• Assess and help with gridded data standards
compliance following UAF-developed techniques
• Develop end-user applications that access gridded data, using CF-compliant OPeNDAP
• Help educate regional scientists, developers, consultants that standardized services, libraries and tools are available
![Page 27: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/27.jpg)
Late Breaking News
![Page 28: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/28.jpg)
THREDDS
• When people say “THREDDS”, they can mean a catalog or a data server– THREDDS Catalogs are XML files that define
metadata, datasets and services– THREDDS Data Server is a Java data server
application developed by Unidata, a simple and flexible way for delivering CF-compliant data via OPeNDAP
![Page 29: Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned](https://reader035.vdocuments.us/reader035/viewer/2022062501/568164c9550346895dd6e4e7/html5/thumbnails/29.jpg)
OPeNDAP
• When people say “OPeNDAP”, they can mean protocol, a server, or an organization– OPeNDAP (or just DAP) protocol: a remote data
access service protocol with multiple responses• DDS – describes the format of the data• DAS – describes associated attributes• DDX – XML response containing the DDS & DAS• DODS – the actual data
– OPeNDAP server: a server capable of serving the DAP protocol (e.g. THREDDS Data Server, Hyrax, Pydap, GrADS Data Server)
– OPeNDAP organization: Defines and maintains the DAP protocol, develops servers and clients, provide custom OPeNDAP solutions