LRGS & OPENDCSNRT Data Acquisition Systems

Enterprise Data and Information Management Section Informatics Branch, Cooperate Services Division

Alberta Environment and Sustainable Resource Development

March 11, 2015 WISKI Conference, Edmonton

Enterprise Data & Information Management Staff

Bob Werner

Vacant

Vacant

Vacant

NRT Data Acquisition Team

Bob

Werner

Near-Real-Time Monitoring

Hydrometric/Lakes/Ice• Water level

• Stream Flow

• Storage

Meteorological• Precipitation

• Air temperature

• Snow depth/water

• Wind speed/direction

• Humidity

Water Quality• Conductivity

• Dissolved oxygen

• pH

• Turbidity

Ground Water• Water level

• Temperature

Data Acquisition Group Responsibilities

• Manage the real-time data collection systems (LRGS, OPENDCS)

• Provide data support for Field Staff (GOES, problems log)

• Provide 24x7 NRT support during major events (ice breakup, floods)

• Provide access to hydrometric, meteorological, groundwater, & water quality data

• Archive historical ESRD data

WISKI: ESRD Enterprise System

• WISKI serves many users

& functions within ESRD

• Primary Users: River Forecast &

Engineering

Dam Operations

• WISKI allows: Modeling

Graphing

Export & Analysis

NRT & Archival

Continuing to expand…

Data Collection Working Together

Office

Results External Partners

Monitoring

Data ManagementSDOWISKI Support

RiverEngineering

River Forecasting

Dam Operations

Irrigation Operations Public

Field

Data Acquisition (LRGS/OPENDCS)

• Decoding• Data validity checks• Sensors verification• Unit conversions• Time conversions• Time adjustments

• Create stations

• Insert/update metadata

• Create and assign timeseries

• Create aggregations and processing time series

• Input monitoring visits and apply corrections

• Verify data accuracy

Data Management Activities

WISKI Database

Data Acquisition Alerts and Tools

Our system requires numerous alerts and tools to pre-emptively discover and resolve issues

• Broken decodes

• Lack of platform connection and/or data currency

• System alerts

• Change decoding script as required

• Test connection and troubleshoot a solution

• Dashboard to identify overall health of system

Data Sources

Realtime Data Communications

Satellite DCPs (450)

SCADA Server (44)

Phone Loggers (115)

WISKISFTP Servers (15)

Webpage Data Mining (36)

FIRES Meteorological (250)

IP Connect (88)

Bulk Historical Data Products

RAW MODEM OUTPUT21,2013,279,1200,1393,-1.265,-.527,0,.316,1.265,1.265,1.265,1.265,1.265,1.265,1.265,1.265,1.265,1.16,.633,.105,0,0,0,0,0,0,0,-.105,-.105,-.843,-.843,-.738,-1.16,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-1.265,-2.003,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.53,-2.636,-3.584,-3.69

19,2013,279,1300,1393,23.52,100,12.91,12.86,5.649,73.8,7.4,9.26,73.8,5.64,59.46,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,1.775,11.44,92.7,13.1,334.5,334.5,334.5,334.5,0,2.759,0,2.385,0,1.704,0,1.602,0,334.4,334.5,334.5,334.5,5.435,5.366,5.292,5.354,0,-.009,5.626,5.52,7.36,7.28,9.12,9.21,5.581,5.461,333.5,331,333.5,331,333.5,331,333.5,331,333.4,330.8,333.4,330.8,333.4,330.8,333.4,330.8,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,6998,2193,0,358,0,358,.088,0,358,0,1200,358,0,358,0,358

GOES BINARY4341007413339151034G34+0NN016WXW00082bB1EB?CB?BB?CB~}B~~B~}B~yB~xB~yB~uB~sB~sB~rB~nB~lB~iB~mB~iB~gB~eB~eB~eB~aB~aL@]@H 4346414A13339154034?30-0HF016WFF00087`B1ECAjCAeCAdCAdCA`Caas0F5qa?>+rq9V)n]1j;ik$}.5:=Kf5RKPKFTD#H3d(i**+{*m&

GOES 5 MINUTE DATA – HUMAN READABLE FORMAT4555E4EE13339001711G35+1NN167EXE00315 :HG 2 #5 34.153 34.153 34.156 34.155 34.156 34.153 34.152 34.157 34.157 34.156 34.154 34.156 34.156 34.157 34.157 34.157 34.157 34.157 34.157 34.153 34.152 34.153 34.153 34.153 34.153 34.153 34.153 34.157 34.157 34.153 34.154 34.153 34.153 34.152 34.153 34.153 :VB 17 #60 13.6 14.2 14.9 :YP 17 #60 1068 1082 1093

TAU INCOMING TEXT FILEBighorn ,02-Dec-2013 ,10:00:00 ,4307.18 ,3832.68 ,0 ,800.302 ,0 ,-14.5695

Brazeau ,02-Dec-2013 ,10:00:00 ,3159.93 ,1316.21 ,0 ,976.17 ,0 ,-16.3949

Nakiska ,02-Dec-2013 ,10:00:00 ,7.78512 ,0 ,0 ,0 ,0 ,10.4279

Rose_Creek ,02-Dec-2013 ,10:00:00 ,0 ,0 ,0 ,0 ,0 , 0

Canmore ,02-Dec-2013 ,10:00:00 ,0 ,0 ,0 ,0 ,0 ,-9.80865

Spray ,02-Dec-2013 ,11:00:00 ,5564.43 ,379.208 ,0 ,123.722 ,0 ,-9.52422

WEB PAGE DATA MINE FROM INTERNET<HTML><HEAD><TITLE>Hourly Hydromet Data for Lake Sherburne, Swift Current Creek near Babb, MT</TITLE></HEAD><BODY><PRE>

SHER |13DEC03| FB | AF | OBG | OBT | RB | RD | QR | QS | Q |

| 00:00 | 4761.59 |28735.00 | 33.94 | 561.61+| 0.80 | 0.73 | 0.00 | 0.00 | 0.00 |

| 00:15 | 4761.63 |28780.00 | 33.96 | 561.02+| 0.79 | 0.73 | 0.00 | 0.00 | 0.00 |

INTERNET TRANSFERATE/TIME (yyyy/mm/dd) AirTemp Precip Bat vol

(standard time) (C.) (mm) (volts)

2013/11/30 00:24:00 266 13.1

2013/11/30 01:18:00 2.5

2013/11/30 01:24:00 266 13.1

2013/11/30 02:18:00 2.4

ENVIRONMENT CANADA STANDARD NATIONAL OUTPUT FOR MODEM01+0011. 02+2013. 03+0340. 04+1600. 05+1230. 06+21.19 07+100.0 08+12.93 09+12.77 10-32.66 11+66.25 12-32.78 13-32.13 14+68.15 15-35.48 16+62.03

17+0.034 18+50.03 19+0.468 20+58.40 21+13.66 22+1.123 23+60.78 24+1.056 25+1500. 26+089.7 27+0.039 28+36.35 29+0.034 30+55.28 31+0.000 32+34.65

33+146.4 34-28.12 35+317.0 36+317.0 37+317.0 38+317.0 39+0.000 40+0.434 41+0.000 42+0.610 43+0.000 44+0.065 45+0.000 46+0.000 47+0.000 48+317.1

49+317.1 50+317.1 51+317.1 52+27.40 53+27.36 54+27.24 55+27.29 56+0.000 57-0.008 58+6998. 59+6998. 60+6998. 61+6998. 62+6998. 63+6998. 64+6998.

65+6998. 66+6998. 67+6998. 68+6998. 69+6998. 70+6998. 71+6998. 72+6998. 73+6998. 74+6998. 75+6998. 76+6998. 77+6998. 78+6998. 79+6998. 80+6998.

81+6998. 82+6998. 83+6998. 84+6998. 85+6998. 86+6998. 87+6998. 88+6998. 89+6998. 90+6998. 91+6998. 92+6998. 93+6998. 94+6998. 95+1158.

Incoming Data Formats

System History – 1984 to 2015

Rafi Morgantini and Lenard Cheng Project Managers, Alberta Environment, Systems and Computing

STREAM/COLLECT system on VAX, 12th Floor Oxbridge

Late 1980’s: Moved to RFC 11th floor and managed by Thai Nguyen and Ian Barnaby after the dissolution of Systems and Computing

1980’s STREAM/COLLECT System on VAX

1990’s DACQ/COLLECT System Data General

DACQ Database System & COLLECT Acquisition

COLLECT Acquisition acquired GOES via dialup Telnet, manual data inputs via dialup Telemark and automated tone encoder Telemark2 for ~ 45 stations and stored on DACQ system.

2000 NewLeaf and COLLECT Red Hat UNIX

DACQ decommissioned prior to Y2K and COLLECT now independent of the new database application called NewLeaf (Ingres). Eventually Data General was decommissioned and COLLECT moved to Red Hat UNIX server (Allister Chisholm).

System History – 1984 to 2011

This period constitutes a major growth of our NRT network via GOES, MSC dialup modems, BCHydro, TransAlta, Merchant, Scada, SnoTel, AutoPoll, Hydras3, and LoggerNet. This involved the deployment of VM technology that provided a scheduled poll of each data steam as required.

2001 - 2010 COLLECT and Acquisition Modules

2001 Local Readout Ground Station (LRGS)Dialup via Telnet was eventually discontinued and the JAVA open-source software provided by USGS was deployed to interface with COLLECT.

2005 LRGS (DELL) and DCS-Toolkit (XML)The USGS consultant, Michael Maloney (ILEX Engineering) was hired to install a server and several licensed copies of the DCS-Toolkit. This provided DM with the tools necessary for writing decoding scripts for dozens of new types of messages, including pseudo-binary.

2011 LRGS & DCS-Toolkit Upgrade (Oracle)ILEX Engineering was hired to convert XML to Oracle and provide clients to desktops.

System Timeline – 2014 to 2015

All hardcoded decoding scripts embedded into COLLECT replaced by DECODES. This is essential, as decoding scripts on UNIX are no longer supported to provide enhancements and/or the creation of new decodes based on new platforms or firmware upgrades.

2014 COLLECT Replacement – Phase 1

2015 COLLECT Replacement – Phase 2LRGS is upgraded to handle EDL (electronic data logger) and other non-GOES data inputs. Also, the dialup modem polling is rewritten to utilize DIGI ConnectPort device.

April, 2015 LRGS/OPENDCS Acceptance TestingBoth Phase 1 and 2 are currently being UAT tested till the end of the contract period. Once complete, we will be running LRGS version 9 and OPENDCS RC 4.0.

May, 2015 LRGS/OPENDCS PROD LaunchAfter the ICE season and before the spring runoff, a decision will be made to go live. If so, COLLECT will be engaged as a backup system.

January, 2015 LRIT Satellite Project

Legacy System ConfigurationPhase 1 and 2 Identified

NRT Data AcquisitionLRGS/OPENDCS Final State

What is LRGS?

Local Readout Ground Station is a server that can be run on any operating system, as it is written in JAVA. UNIX is our preferred operating system, however this can be changed depending on our system support.

Non-GOES Messages

OPENDCS

DOMSAT

Local Readout Ground Station is a server that can be run on any operating system, as it is written in JAVA. UNIX is our preferred operating system, however this can be changed depending on our system support.

Non-GOES Messages

OPENDCS

DOMSAT ReceiverDecommissioning service by 2015/16

Domestic Communications satellite is a single high-speed channel that NOAA rebroadcasts to, allowing the entire DCS (Data Collection System) GOES messages for consumption by users. ERSD owns a dish a primary receiver on top of Oxbridge tower and a backup site, at EMA (Emergency Management Alberta) in Edmonton. This service will be decommissioned by the end of 2015.

DOMSAT ReceiverDecommissioning service by 2015/16

DCP’s transmit to either GOES W or GOES E and captured at NOAA, which in turn are broadcast to DOMSAT (AMC-4) to our DOMSAT receiver. These messages are then captured on our LRGS server.

LRGS

LRIT ReceiverInstalled on January, 2015, Oxbridge Place

Our first LRIT satellite service was installed on January, 2015 and will become our primary source of GOES messages once DOMSAT is decommissioned. Messages are rebroadcast to both GOES E and GOES W. A secondary site is planned for the year end by our Provincial Telecom Coordinator, Forestry Telecommunications Shop, ESRD.

Non-GOES Messages

OPENDCS

LRIT ReceiverInstalled on January, 2015, Oxbridge Place

The GOES messages are received at NOAA and rebroadcast back to GOES East and GOES West. Messages are then relayed back to an LRIT receiver which we now own on Oxbridge Place.

Satellite Antennas @ Oxbridge

Currently, our newly installed LRIT antenna is pointing at GOES East (SE direction).

DCS via Internet

As a backup, should messages fail via our satellite receivers, a redundancy is built into the system by the internet. We current use the internet feed from Wallops Command and Data Acquisition Station (WCDAS) near Chincoteague, Virginia, and Emergency Data Distribution Network (EDDN) Sioux Falls, South Dakota.

Non-GOES Messages

OPENDCS

Wallops Command Centre

WCDAS Statistics

Annual Funding at Wallops (NOAA Appropriations): Approximately $11 million

Workforce: 91 (mixture of Government and contractors)

Number of buildings and square footage: Seven buildings with 48,920 square feet (not including antenna shelters)

Number of antennas: 16

Data rate throughput: Currently 41.7 megabits/second

Date NOAA occupied site: Original permit signed on February 2, 1965 and dedicated on: June 22, 1966

Non-GOES MessagesFTP, HTTP, SQL, Dial-up Modem

Since the completing of the COLLECT replacement project, phase 2, we now have the capacity to acquire all non-GOES messages, except for stations currently using Campbell Scientific LoggerNet software. Additional work is required to integrate LoggerNet API into LRGS (2015/16).

Non-GOES Messages

OPENDCS

Non-GOES MessagesFTP, HTTP, SQL, Dial-up Modem

LoggerNet

Campbell Scientific SoftwareIP Protocol and PakBus platforms require this software. Phase 3 (2015/16) will integrate the LoggerNet API into LRGS, thereby completing the total integration of non-GOES messages.

Phase 2

Non-GOES MessagesDial-up Modem

One major improvement allows for LRGS to function with modems. We are retiring our Sangoma cards that plugged directly into our UNIX server for a device that sits on the Ethernet.

16 US Robotics modems

Sangoma Card (legacy) – prevented us from upgrading our OS and server.

DIGI LTS ConnectPort 32 allows for all 16 modems to have a telnet port on the internet available to the LRGS server.

Why still dialup modems?

We currently still maintain 115 sites with dialup modems that are decreasing at a rate of ~ 7-10 per year now for 3 years. Our peak was 145 stations. We predict that within the next 6 years, this number will drop of significantly. The cost of providing copper connection to remote stations is becoming increasingly more expensive and maintenance of existing lines, more probiotic. Providers are increasingly switching to either cellular or GOES.

OPENDCS

We started with the raw messages that are acquired by LRGS (GOES and non-GOES) and will end this presentation with the second part, that of the OPENDCS (red box). The other outputs that we own and operate are, Web DCP Monitor, Message Browser (integrated into OPENDCS) and the DCS Toolkit currently used in production (blue boxes).

Non-GOES Messages

OPENDCS

OPENDCS

The OPENDCS structure is similar to DCS Toolkit currently being used in production. The reasons that make this software valuable are:

Designed in JAVA, which allows for cross platform use

Utilizes DECODES format structure to decode messages

Configuration XML easily shared with other NRT acquisition members

OPEN means, open source and therefore will not require a separate licence for each installation

RC03 denotes the release is Canadian (RC), specific to ESRD requirements

All station configurations formerly contained on several flat files in COLLECT are now stored in Oracle and more easily backed up and restored when required

Software is easily upgraded on workstations Decoding changes are easily applied Platform terminal scripts are more easily edited

OPENDCS – LRGS Status

LRGS Version 9.0 – This inference displays the current good, parity and dropped messages from DOMSAT, LRIT and also, electronic data logger (EDL). It also displays the internet backup from WALLOPS and EDDN.

OPENDCS – DCP Messages

This familiar component is used to quickly retrieve messages as well as apply the decoding scripts. In this example, pseudo-binary messages are decoded to the desired parameters.

OPENDCS – Decodes Editor

This is where all the DECODES are added/edited. Tabs include, Platforms, Sites, Configs, Equipment, Presentation, Routing, Sources, Network Lists and Schedule Entry. The Configs tab is open, displaying numerous DECODES applied to different platforms.

OPENDCS – Decodes EditorExample of a decoding script

Each script has a series of format statements that apply to the incoming messages. In this case, pseudo-binary messages are decoded to the desired parameters. This latest edition of DECODES is now color coded.

OPENDCS – Decodes EditorCommon DECODES Format Operations

NRT Data AcquisitionIn the middle of the process.

• Raw Data transmitted or

EDL

• Acquired & Decoded

(LRGS/OPENDCS)

• WISKI

Acronyms

API Application Program InterfaceBNF Backus-Naur Form. Grammar used for specifying protocols.

DADDS DCS Automated Data Dissemination System – this is the primary computer at NOAA’s DCS facility in Wallops, VA.

DAIDatabase Access Interface – An abstract Java interface defining how some type of object is stored in, and retrieved from the database. This allows an application to read and write an object without caring about the way the object is stored in a database.

DAODatabase Access Object – A concrete implementation of a DAI for a particular database. In DECODES, there are several objects for which DAOs are written for both XML and SQL databases.

DCPData Collection Platform – a collection of equipment including battery, solar panel, data logger, antenna, transmitter, and a set of hydro-meteorological sensors.

DCSData Collection System. The “GOES DCS” encompasses DCPs, satellites, and ground systems necessary to collect environmental data and deliver it to client organizations who need to use it.

DECODES DEvice COnversion and DElivery System – software used to decode raw DCP data and convert it into standard formats like ZXRP.DOMSAT DOMestic SATellite. In the DCS, DOMSAT refers to NOAA service in which DCP data is rebroadcast over a commercial satellite carrier.

DRGSDirect Readout Ground System. As opposed to rebroadcast services such as LRIT, NOAAPORT, and DOMSAT, a DRGS reads the primary data feed directly from GOES.

GOESGeostationary Operational Environmental Satellite. NOAA operates two GOES satellites, designated GOES East and West. AESRD primarily uses GOES West for its DCPs.

HADSHydrologic Automated Data System, run by the U.S. National Weather Service to process DCP data deemed to be of interest to that agency and its cooperators.

HDLC High-Level Data Link Control. On DOMSAT, data is encoded into HDLC blocks.HRIT High Rate Information Transfer. Starting with GOES R, this will replace LRIT.LNB Low Noise Block Down Converter – Installed at the focus of a parabolic satellite dish. Down-converts RF to IF over a coaxial cable.kbps Kilobits per second

LRITLow Rate Information Transfer – one of the services available on GOES. This service is a rebroadcast of all DCS data mixed with various imagery and forecast data from NWS.

NESDIS National Environmental Satellite Data Information Service – a division of NOAA.NFS Network File System – a high-performance Unix/Linux based mechanism for sharing folders over a network.

NOAA National Oceanic and Atmospheric Administration, a division of the Commerce Department of the U.S. Federal Government.NOAAPORT A rebroadcast service from NWS. It contains raw and decoded data deemed to be of interest to the NWS.

NOC Network Operations Center – The central hub for satellite based Internet. All traffi c is first routed to the NOC and then to the destination.NWS National Weather Service – a division of NOAA.ODBC Open Database Connectivity – an application program interface (API) for programmatically accessing database systems.RFP Request for Proposal

SCADA Supervisory control and Data AcquisitionSHEF Standard Hydro-Meteorologic Exchange Format – A standard for file format and data type codes published by the National Weather Service.

SMBServer Message Block, also called “Samba”. This is a protocol that allows Unix/Linux systems to access shared drives and other resources on a Windows system. It is not as effi cient or secure as NFS.

SQL Structured Query Language – that language used by relational database systems such as Oracle, PostgreSQL, or Microsoft SQL ServerSTIWG Satellite Telemetry Interagency Working Group – a consortium of organizations that use the DCS. STIWG meets twice per year.USACE United States Army Corps of Engineers – responsible for maintaining many U.S. waterways for navigation and flood control.

USBRUnited States Bureau of Reclamations – A division of the Interior Department of the U.S. Federal Government. USBR manages water resources in the arid western regions.

USGS United States Geological Survey – A division of the Interior Department of the U.S. Federal Government.VM Virtual Machine

WMO World Meteorological OrganizationZRXP Kisters WISKI software input data format.

Credits

Questions??