Rainer Sternfeld, CE September 2014

Indexing the real world

March, 2015

André Karpištšenko Research Lead, Planet OS

2 March 2015

Planet OS Presence

Tallinn

Rio de Janeiro

Washington DC

HoustonLos Angeles

Sunnyvale

3 March 2015

Sensor Data Discovery EngineOcean Data Management

From a small buoy to Big Data

Data BuoysMarket: $2 billion

Competitors: 100+ producers Scalability: poor to limited

2008 2012 2014

Market: $5 billion Competitors: 25+

Scalability: good but slow

Market: $100+ billion Competitors: 15+

Scalability: very scalable and fast

4 March 2015

Data

Time

Trends in Sensor Data

“By 2020, 40% of all data ever collected by human kind will be generated by sensors.”

Hewlett Packard:

5 March 2015

Map of all devices on the Internet

August 2, 2014

6 March 2015

Consolidate dataflows, organize and make sense of your data

Access your data with 3rd party tools and systems

One interface to search and discover your local and remote data

Securely exchange, acquire or sell datasets

Build advanced domain specific solutions without hassle

Industrial IoT Platform for Real-World Sensor Networks designed for ocean, land, air and space data

7 January 2015

Planet OS is a Data Discovery Engine designed for sensor and machine data

January 20158

/

9 March 2015

Case #1: NOAA Real-time weather and climate data + forecasts

Image Credit: NASA

They are working hard on a challenge presented by:

• Tens of thousands of devices deployed in the ocean, on land, and space

• Tens of terabytes coming in every day

• 700 scattered “web services” (FTPs, flat files, Thredds servers, no APIs)

• Breaking connections (all.the.time.)

• 25,000 employees

• Critical data for the government, the industry, and citizens

What if all this data would be easily discoverable and machine-readable?

10 March 2015

Example to case #1: Marinexplore.org 40,000+ data streams from 33 organizations of open data

Advanced Data Discovery Raster data / heat map overlays Access with third party applications

Raster data / quiver plots Graph Monitor Build custom datasets

11 January 2015

3D situational awareness

12 March 2015

Professional Services Experience: Data IntegrationImage Credit: NASA

• 42,000+ data streams from 33 organizations • Work with up to 100 years of historic data • Buy single datasets or subscribe to vendor deliveries

• Ask Planet OS about: • More data sources for your needs • Custom APIs to data • On-demand alerts if something you need is found

KEY BENEFITS

Devices Data Products

• 21,134 devices like the WMO Global Telecommunications System (GTS) buoys, discus buoys, weather buoys,

profiling buoys, drifting buoys, moored buoys etc.

• 5,041 research vessels measuring temperature, salinity, CO2, analyzing important hydrographic and biological

parameters, collecting ADCP profiles and other parameters in the ocean.

• 3,842 operational Argo floats collecting CTD datasets.

• 969 common buoys, for example buoys supporting marine operations.

• 931 fixed stations, for example stations related to US Meteorological Aerodrome Report.

• 126 Teledyne SLOCUM gliders.

• 21 tagged animals by University of St. Andrews.

• 13 station streams from U.S. HF-Radar Network (CODAR SeaSonde)

• 11 Ships of Opportunity, for example the Baltic Operational Oceanographic System related ships.

• 7 Liquid Robotics Wave Gliders, configurable platforms designed to support a wide variety of sensor payloads.

• 2 satellites EUMETSAT ASCAT microwave scatterometer and Aquarius/SAC-D.

• NOAA NCEP WAVEWATCH III, GFS and Global CFSR Reanalysis Hindcasts.

• ETOPO1 Global Relief Model, including both bedrock and ice surface, a 1 arc-minute global relief model.

• The Group for High-Resolution Sea Surface Temperature (GHRSST) Blended Global 1-km Resolution SST.

• NOAA International Comprehensive Ocean-Atmosphere Data Set (ICOADS) SST, Sea Level Pressure and Air Temp.

• NOAA Ocean Surface Current Analyses - Realtime (OSCAR) Third Degree Sea Surface Velocity.

• Daily near-real-time sea height anomaly (SHA) data from TOPEX, ERS-2 and GFO altimeters.

• Hybrid Coordinate Ocean Model (HYCOM) + NCODA Southeast United States and Global Analysis

• KLIWAS North Sea Climatology - Hydrographic Climatology.

• Baltic Sea Physics Reanalysis from DMI and SMIH (1990-2009), Baltic Sea - Sea Ice concentration and Thickness

• Ocean Colour Sea Surface Level-3 Chlorophyll-A Concentration

• Global Ocean Biogeochemistry Analysis: nitrate, phosphate, dissolved oxygen, chlorophyll, phytoplankton

• Mediterranean Sea High Resolution and Ultra High Resolution Sea Surface Temperature Analysis.

• Ocean Optics Products for daily observations and monthly averages reprocessed for the 1997-2010 time period.

• Global Ocean OSTIA SST and Sea Ice (Re)Analysis

• Ifremer Sea Surface Temperature L3 observations

• Sea surface winds from scatterometer, namely descending and ascending swaths.

• NOAA/NCDC Blended daily 0.25-degree Sea Surface Winds.

13 March 2015

Example to Case #2: Bravante Helping to deliver offshore data reports 80% faster

Problems and challenges:

• How to disseminate information to 5 end-customers onshore in parallel?

• How to engage specialists and mangers with the same tool?

• How not to change anything in the equipment on the vessel?

• How to improve speed and quality of the environmental baseline

surveys?

What if all this data would be available in the cloud?

14 March 2015

R/V Jean Charcot

Interactive reporting

Case #2: Bravante Helping to deliver offshore data reports 80% faster

15 March 2015

Robotic ocean-borne sensor platforms

increase productivity

LIQUID ROBOTICS WAVE GLIDER

16 March 2015

A decade of growth in marine acoustics sensor data

summer year-round

2004 2014

2K 400Kn < 10 n > 10002D 4D6 weeks 6 months

sample rate# of sensorsdimensionstime spanactivity

PGS SURVEY VESSEL, 12KM STREAMERS

17 March 2015

Satellites are getting smaller and cheaper.

150 launched since 2011 (3x of the market estimate)

SPIRE, A SAN FRANCISCO STARTUP BUILDING NON-IMAGING LOW-ORBIT NANOSATELLITES USING RF SENSORS

18 March 2015

Unmanned vehicles are estimated to grow 10x in 10 years

Image Credit: Northrop Grumman

19 March 2015

Supported data types and formats

19

VIDEO SONARS (ADCP)

SPATIO-TEMPORAL INDEXING

TIME-SERIES

DATA TYPES

VECTORS

ARRAYS

RASTERS

INSTRUMENTS

SATELLITES

IN-SITU DEVICES

MODELS

HF RADARS

SEISMIC

M

M

M Metadata-level

High-Level Platform Architecture

UIUI

Private and Public Data

Access Securely Anywhere

Connect Storages

UI Integrated Tools

Data Services

Secure Roles & Projects

Exchange Data

ETL

Diverse Formats

PUBLIC

Run Analysis

Vertica, Postgres, S3, EBS Python/Cython, numpy/scipy GDAL, C++ OPeNDAP, NetCDF4 Web GIS, Leaflet, d3, Django

20 January 2015

21 January 2015

StreamFlow is a Dataflow Consolidation Engine

Transformation

Aggregation

Indexing

Data Segments

Incoming Datasets Configuration Channels

Data fetching

Computations

Input Streams

Data Streams

Collection

Plugins

Derived streams

Original Data Formats

Plugins

22 March 2015

Traditional Data Value Chain is Missing Data Exchange

SATELLITES DRONES CARS BUOYS OIL PLATFORMS SHIPS TRACTORS LIGHTING

PHYSICAL CHEMICAL BIOLOGICAL OPTICAL NON-OPTICAL (RF)

TIME-SERIES RASTER ARRAYS VECTORS SEISMIC VIDEO ACOUSTIC

QA/QA OUTLIER DETECTION MODELING DATA FUSION DATA LOGISTICS VISUALIZATION ENHANCEMENT DERIVATIVES

HARDWARE SENSORS RAW DATA SOFTWARE ANALYTICS

CUSTOMER APPLICATIONS

ENERGY WEATHER AGRICULTURE TRANSPORTATION INSURANCE TELECOMMS LIGHTING HEALTHCARE

DATA EXCHANGE LIVES HERE

Rainer Sternfeld, CE September 2014

www.planetos.com andre@planetos.com