epa continuous monitoring data strategy: synopsis ioos dmac 29 may 2015
TRANSCRIPT
![Page 1: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/1.jpg)
EPA Continuous Monitoring Data Strategy: Synopsis
IOOS DMAC
29 May 2015
![Page 2: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/2.jpg)
TCEQ HIGHLIGHTSInterviews- New Jersey Department of Environmental Protection
![Page 3: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/3.jpg)
Texas CEQ Highlights
• TDS early warning for irrigators• Guiding and assessment of remediation
activities; assessment of WQ attainment• Emergency preparedness, recreational
activities (flows)• Assessment of sediment NPS loads• Public engagement
![Page 4: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/4.jpg)
Texas CEQ Highlights
• Moving from LEADS (air quality!) and MANVAL to AQUARIUS
• Provisional data published within 30-60 minutes, manually validated data with 150 days.
• Pre- and post-deployment QC are necessary, not sufficient.
• Sensors deployed by partners (GBRA, USGS)
![Page 5: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/5.jpg)
NJDEP HIGHLIGHTSInterviews- New Jersey Department of Environmental Protection
![Page 6: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/6.jpg)
New Jersey DEP Highlights
• Buoys and airborne sensor to support shellfish management
• Glider deployed to improve marine assessments of attainment
• Continuous data in streams and rivers used for criteria and TMDL development, source water protection, long-term trends
• QAPPs required for assessment data
![Page 7: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/7.jpg)
New Jersey DEP Highlights
• Some data available near-realtime with lower QC, all available in 3-9 months with full QC.
• Marine data currently available through web portal, freshwater data will also be added.
• Other:– “Continuous Monitoring could use EPA’s
continuous support.– Get USGS to put more emphasis on WQ.
![Page 8: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/8.jpg)
IOOS HIGHLIGHTSInterviews – Integrated Ocean Observing System
![Page 9: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/9.jpg)
IOOS Highlights
• Coordinates ocean data from buoys, gliders, satellites, models for monitoring, assessment and management of HABs, acidification, coral reefs, beach closures, biodiversity
• Most data are realtime with automated QC, some climate-related data published on a delayed basis– Certain data published worldwide through GTS
![Page 10: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/10.jpg)
IOOS Highlights
• Hybrid data management framework includes both centralized and distributed elements.– How much should Data Management and
Communications reflect governance structure?• IOOS is more operationally oriented,
consistent with its mission.– Cost per sample is lower for sensor data– Currently implementing automated QC: Quality
Assurance of Real-Time Ocean Data
![Page 11: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/11.jpg)
USGS HIGHLIGHTSInterviews – Integrated Ocean Observing System
![Page 12: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/12.jpg)
USGS Highlights
• Nutrients– Great Lakes tributaries: nutrient surrogates– Iowa: source water protection– Mississippi River: nitrate photometers
• Fisheries: temperature monitoring to improve understanding of fish movements
• Other applications– Road salt, lake turnover, manure spreading
![Page 13: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/13.jpg)
IOOS Highlights
• Generally homogenous hardware– Campbell Scientific CR1000, CR800, CR10– Some level loggers, OnSet Hobos
• Extensive QA, QC and management toolset– Continuous Record Plan (CRP) and Techniques and
Methods (T&M) documents– ADAPS, GRSat, DCT, RDT, NWIS RA, QW,
AQUARIUS, CHIMP, SITES, …
![Page 14: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/14.jpg)
LATENCY AND QCSmall Thoughts
![Page 15: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/15.jpg)
Latency, Quality and UseLower QC Higher QC
Low Latency
Available within minutes or hours
“Operational”
•Automated QC•Useful for trends•Exceedance of an action limit may not be a violation
Unrealistic?
High Latency
Available weeks or months after
Useless?
“Assessment”
•Manual QC•Can be compared to WQ standards
![Page 16: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/16.jpg)
DESIRABLE ATTRIBUTESSmall Thoughts
![Page 17: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/17.jpg)
Desirable Attributes
• Discoverability, transparency and interoperability are data management virtues.– Development and publication of metadata in
accessible catalogs. – Use of appropriate detail and a controlled
vocabulary in metadata and datasets. – Publication of data using open standards.
• Provenance, scalability and sustainability too.
![Page 18: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/18.jpg)
DEFINING “CONTINUOUS”Small Thoughts
![Page 19: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/19.jpg)
What we’ve heard …
• Hourly or better; minimal human involvement/automated (TWDB)
• Demonstrates serial correlation; reflects/captures diurnal cycles (TCEQ)
• Usually no less than hourly frequency; deployed for at least 72 hours (NJDEP)
• Unattended/autonomous; metadata less variable than data; considers Nyquist limit (IOOS)
• In-situ time series in-situ, cost-effective (USGS)
![Page 20: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/20.jpg)
Working definition?
• A time series of observations taken at regular intervals with an automated sensor:– Sensor may be fixed or moving (glider/AUV)– Data may be delivered in real time or logged for
later retrieval– Metadata are invariant for a deployment (?)– QC information may be observation-specific (?)
• What about automated samplers?
![Page 21: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/21.jpg)
SOME EXISTING ALTERNATIVES
![Page 22: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/22.jpg)
Some Existing Alternatives
• Status Quo: WQX, STORET, Water Quality Portal
• USGS/NWIS• USEPA/AirNow• CUAHSI/HIS• NOAA/IOOS
![Page 23: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/23.jpg)
Status Quo
Data User
Sensor
Organization WQX STORET
WQ Portal
metadatasummaryblob
met
adat
a
sum
mar
y
data
![Page 24: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/24.jpg)
Status Quo
Pro• Familiar• Existing support and
sustainability
Con• Clumsy to enter• Inefficient storage• Difficult to discover and
access time series data
![Page 25: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/25.jpg)
USGS NWIS (AQUARIUS Buildout)
Data User
Sensor
Organization NWIS
Water Data for the Nation
metadatam
etad
ata
data
data
![Page 26: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/26.jpg)
USGS NWIS
Pro• Centralized• Standardized• Well-established
Con• Rigorous QA and QC
requirements preclude publication of “just anybody’s” data
• In the midst of transition
![Page 27: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/27.jpg)
AirNow
Data User
Sensor
Organization AirNow DMS
AirNow
metadatam
etad
ata
data
data
data
AirNow-Tech (Decision Support
System)
Analysis tools
EnviroFlash (public alerting system)data
data
AirNow API (standardized output and data feeds)
data
![Page 28: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/28.jpg)
AirNow
Pro• Established system already
integrating sensors across the US and the world.
• Designed to support extension to water, other media
• Adaptable to new parameters • Handles real-time and bulk
upload data processing• Existing homogeneous and
heterogeneous QA/QC routines• Web-based
Con• Does not currently support full
range of desired OGC services (SOS/WaterML 2, CS-W)
![Page 29: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/29.jpg)
CUAHSI-HIS
Sensor
Organization
data
metadata
data
![Page 30: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/30.jpg)
CUAHSI-HIS
Pro• Built on OGC standards to
support discoverability and interoperability
• Working on inexpensive cloud-based appliances for serving data.
• Supports heterogeneous problem domain by standardizing data exchange
Con• Scalability uncertain• Current issue with
connection between sensors and cloud appliances.
![Page 31: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/31.jpg)
SOS ServerSOS Server
NOAA/IOOS
Data User
Sensor
Organization SOS Server
Catalog
metadata
data
metadata
data
data
![Page 32: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/32.jpg)
NOAA/IOOS: Detail for GLOS
![Page 33: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/33.jpg)
NOAA/IOOS
Pro• Like CUAHSI, heavily OGC-
compliant to promote interoperability and discoverability.
• Involved in development of standardized QC for real-time data.
• Tools like Scalability Experiment demonstrate value of approach.
Con• Data standards are well-
defined, but many different technology stacks exist for implementation.
• Focus is on oceanographic variables, which include only some WQ parameters.
![Page 34: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/34.jpg)
![Page 35: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/35.jpg)
HYBRID APPROACH?
![Page 36: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/36.jpg)
Hybrid Approach: A Trial Balloon
Principles:•Data (observations) available from distributed servers.•Metadata (site/sensor/deployment) harvested and made available from centralized server.•Interoperable with WQX and Water Quality Portal
![Page 37: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/37.jpg)
Hybrid Approach
User Activities•Submit CWQS site to WQX•Submit sensor/deployment QC metadata to ?•Connect (near-realtime) or upload (batch) observations and QC data to appliance•Search for sensors in discovery tool•Download metadata and data through discovery tool
![Page 38: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/38.jpg)
Hybrid Approach
Required Elements•Servers for data, ideally “owned” by submitter– Commercial software (AQUARIUS, KiWIS, …) or– Data appliance in cloud (CUAHSI HIS server?)
•Repository and server for metadata– Extend WQX with tagging of sites as continuous
•Discovery tool– WQ Portal or IOOS Scalability Experiment
![Page 39: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/39.jpg)
Status Quo
Data User
Sensor
Organization WQX STORET
WQ Portal
metadatasummaryblob
met
adat
a
sum
mar
y
data
![Page 40: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/40.jpg)
Status Quo (worst case)
Sensor
Organization
data
![Page 41: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/41.jpg)
Data ApplianceData Appliance
Hybrid Approach
Data User
Sensor
Organization WQXmetadata
data
Discovery ToolData Appliance
data
CWQSD Attributes
CS-W Service
![Page 42: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/42.jpg)
Hybrid Approach
Concerns (so far)•Handling of QC metadata– Serve with site (WQX) or with data (appliances)?
•Disambiguation/deduplication of data– Integrating NWIS, IOOS, other data streams
•Sustainability: archiving of data– Design to support optional transition to
centralized data? System of record?
![Page 43: EPA Continuous Monitoring Data Strategy: Synopsis IOOS DMAC 29 May 2015](https://reader033.vdocuments.us/reader033/viewer/2022051401/56649e6f5503460f94b6c659/html5/thumbnails/43.jpg)
Data ApplianceData Appliance
Hybrid Approach (2)
Data User
Sensor
Organization WQXmetadata
data
Discovery ToolData Appliance
data
; Q
C?
CWQSD Attributes
CS-W Service
STORET
WQ Portal
data
CWQSD Attributes
WQX-C Archive
data
met
adat
a
met
adat
a
data; QC?
CWQSD Awareness
metadata
metadata
NWIS, IOOS, …
data
CS-W Service