WESCML: A Data Standard for Exchanging Water and Energy Supply and Consumption Data… to support Big Data!
CSIRO LAND AND WATER
Bruce A. Simons, Jonathan Yu*, Benjamin Leighton Environmental Informatics – water informatics, information platforms and data ecosystems
22 August 2016
Jonathan Yu | A Water and Energy Supply and Consumption Data Hub for Australia
AURIN & WESC Data Hub Project$24 AUD million e-infrastructure project
• Secure online portal • Urban data & tools • Access: researchers and
government staff in research and policy analysis
https://aurin.org.au
Clockwise - Felix Lipkin, Magnus Moglia, Ben Leighton, Fareed Mirza, Jonathan
Yu, Matthew Inman, Bruce Simons, Ben Caradoc-Davies, Ramneek Singh,
Richard Goh
Liveable-Resilient Cities + Environmental Informatics
AURIN Team: Chris Pettit, Emma Joughin, Martin Tomko, Luca
Morandini, Serryn Eagleson, Rachel Lerm, Phil Delaney, Jack Barton, Emma Williams, Stewart Wallace, others…
2 |
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
Outline1. Background and data challenges
2. WESCML SF-0/SF-1 data standard
3. Application of WESCML for Geospatial Big Data and Cloud Computing
3 |
Towards linked data conventions for delivery of environmental data using netCDF | Jonathan Yu
We’re not data poor“90% of the world’s data has been produced over the last two years”1
Where is it? What is it? Is it relevant?How do I use it?
4 |
1 http://www-01.ibm.com/software/data/bigdata/what-is-big-data.html
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!5 |
(Big) data challenges
Dealing with data silos
Unconnected, unknown, unmanaged Improving data access
and connectivity
Making sense of the data
Understanding and using data in context.
Integrating it with other datasets.
http://www.wired.com/2013/03/big-data-2/
https://www.hc1.com/laboratory-informatics-and-data-silos/
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
Urban policy, sustainability goals, city indicators rely on better, integrated data… and at scale
6 |
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
Urban policy, sustainability goals, city indicators rely on better, integrated data… and at scale
7 |
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
Urban policy, sustainability goals, city indicators rely on better, integrated data… and at scale
waterenergy
8 |
Jonathan Yu | A Water and Energy Supply and Consumption Data Hub for Australia
Specific domain challenge1. Enable consistency between water/energy datasets
2. Enable improved data integration across WESC & other domains
3. “Catch ‘em all!” i.e. public/private org, NFP… nationally (~50-300 orgs)9 |
Jonathan Yu | A Water and Energy Supply and Consumption Data Hub for Australia
Specific domain challenge1. Enable consistency between water/energy datasets
2. Enable improved data integration across WESC & other domains
3. “Catch ‘em all!” i.e. public/private org, NFP… nationally (~50-300 orgs)10 |
Easy! Piece of cake! No worries!
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!11 |
Researchers
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!12 |
Researchers
Data sources
Data infrastructure
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!13 |
Researchers
Data sources
Data infrastructureLots of work in the tooling side
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!14 |
Reality
Need to fix the supply chain
Jonathan Yu | A Water and Energy Supply and Consumption Data Hub for Australia
Current data collection, analysis, reporting
Govt. Agencies
Data Custodians & Providers
Utilities
poor linkages
ad-hoc, inefficient
not well discovered
value of data not maximised
X
15 |
Policy makers
End users
Generalpublic
Researchers
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!16 |
Know data is available
Most cases data needs to join up
How do we enable access to do big data
analysis?
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!17 |
vs
Traditional database/warehouse Data ecosystem like the internetPrefer this – plug in data nodes and grow out
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!18 |
Know data is available
Most cases data needs to join up
How do we enable access to do big data
analysis?
Tools to build data connections
Design ‘data pipe network’ for its content
Build it!
Standardise ‘data pipes’
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
WESCML SF1, SF0 Data Standard
http://wescml.org
19 |
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
Aggregated water/energy consumption
Utility Supply Data
Water Supply
Electricity Supply
WESCML Conceptual models Vocabulary def’s Enabling access to data
Standardised language and terminology
(easy for data to make sense)
Rapid deployment of datainfrastructure on the cloud… and at scale
(enable easy connectivity)
Connecting people with data
Data providers
Researcher/Policy Analyst
Community building
(enable access to datasets)
20 |
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!21 |
Minutely Hourly Daily Weekly Month Season Annual
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
Aggregated Consumption model (SF-0 model) - CombinedMeterReading
Postcode/ Locality/ABSGeographies
22 |
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
Supply Model – Water and electricity
23 |
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
WESC Controlled vocabularies
http://wescml.org/vocab http://demo.sissvoc.info/uwdc/vocab-viz/
24 |
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
WESC SF-1 ModelWESC info model:
• Application Schema reusing generic ISO 19000 series information models (Observations and measurements)
• Accommodates specialisation of recent OGC TimeseriesML standard
25 |
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
WESC SF-1 Model – modularisation of agents, commodities, monitoring locations (meters)
26 |
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
WESC SF-1 Model – modularisation of agents, commodities, monitoring locations (meters)
27 |
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
WESCML SF-0 and SF-1
28 |
Standardised language and terminology for encoding and delivery of water and energy consumption and supply data.
Easier to make sense of the data.
Gives standard data types and attributes to slice and dice data.
https://github.com/CSIRO-enviro-informatics/wescmlhttp://doi.org/10.4225/08/574D1DEEA50DD
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
Application of WESCML for Geospatial Big Data and Cloud Computing
29 |
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
wescml.org
Individual WESC Data Service infrastructure
WESCData serviceWESC XML
Schema
WESC Conformance
Tests
WESC Controlled
Vocabularies
WESC Info Model
AURIN
AURIN Portal
PostGISDB
WESC Data Service
Researcher/Policy Analyst
Postcode/ Locality/ABSGeographies
Water/Elec.Data
30 |
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
Rapid deployment of data infrastructure on the cloud… and at scale
31 |
Rapidly spin up more
containers and connect
up data
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
WESC Data Hub
• Discover• View• Analyse• Visualise• Download
AURIN
Researcher/Policy Analyst
Via the WESC data
protocol
32 |
Jonathan Yu | A Water and Energy Supply and Consumption Data Hub for Australia
Melb – Res water consumption by postcode
Source:
Yarra Valley WaterCity West WaterSouth East Water
2011Q1
33 |
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
Melb – Res electricity consumption vs Res water consumption by postcode
Source:
Yarra Valley WaterCity West WaterSouth East Water
2011Q1
34 |
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
Melb - Electricity consumption vs # separate houses
Source:
Electricity consumption (ABS household electricity 2011)
Number of Separate houses (ABS Dwelling structure from 2011 survey)
35 |
Jonathan Yu | A Water and Energy Supply and Consumption Data Hub for Australia
National scale datasets
WESC Data contributors/collaborators
36 |
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
CSIRO-AURIN WESC Data Hub – building bridges
Policy makers
End users
Govt. Agencies
Data Providers
Utilities
37 |
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
Takeaway #1
39 |
http://www.wired.com/2013/03/big-data-2/
Challenged by making sense of big data – volume and variety?
Particularly water/energy consumption and supply?
Aggregated water/energy consumption
Utility Supply Data
Water Supply
Electricity Supply
Conceptual models Vocabulary def’s
Standardised approach enables dealing with making sense of big data simply
WESCML
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
Takeaway #2
DataAnalysis
Data DataSensor
ReportProduct/ServiceProduct/Service
Do you have data silos?
Data infrastructure tools to enable
easy connectivity
40 |
Data infrastructure ecosystem
Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!
SummaryIf we want to be able to harness big data, we must fix the ‘pipes’!
Standardise and systematize each part of the data supply chain so we can deploy and connect easily and rapidly.
Socio-technical challenge.
WESCML and its application is an example of doing this for water/energy supply and consumption
41 |
LAND AND WATER
Land and WaterJonathan YuResearch data architectt +61 3 9545 2457e jonathan.yu [at] csiro.au
http://wescml.org http://aurin.org.au
Thanks and acknowledgements