big data: geoevent & geoanalytics, - esri · big data store geoanalytics server. big data file...

Big Data: GeoEvent & GeoAnalytics, Leveraging the Spatiotemporal Big Data Store

Adam MollenkopfReal-Time & Big Data GIS Capability Lead, Esri

amollenkopf@esri.com

@amollenkopf

Ricardo TrujilloReal-Time & Big Data GIS Developer, Esri

rtrujillo@esri.com

@rtrujill007

1

2

3

4

5

6

Agenda

why a new type of data store?

visualizing observation data

writing observation data from GeoEvent Server

on-the-fly aggregations

writing analytic results from GeoAnalytics Server

deeper dive

R&D projects sneak-peek7

why a new type of data store?1

Web GISwith real-time capabilities

Web GIS

ingest visualize

analyze store

real-time & big data

thousands3K e/s

3K e/s 200 e/s

BYO-RDBMS

X

X

200 e/s

GeoEvent

Server

Web GISwith real-time capabilities

Web GIS

ingest visualize

analyze store

real-time & big data

GeoEvent

Server

thousands3K e/s

3K e/s 200 e/s

relational

data store

X

X

200 e/s

Web GISwith real-time & big data capabilities

Web GIS

spatiotemporal

big data store

ingest visualize

analyze store

real-time & big data

GeoEvent

Server

millions4K e/s

4K e/s 10Ks e/s

4K e/s

Web GISwith real-time & big data capabilities

Web GIS

spatiotemporal

big data store

ingest visualize

analyze store

real-time & big data

millions4-12K e/s

4-12K e/s 10Ks e/s

GeoEvent

Server

4K e/s

4K e/s

4K e/s

bring-your ownmessage broker

ArcGIS Enterprisewith real-time & big data capabilities

ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

Big DataIoT

ingest visualize

analyze store

real-time & big data

millions4K e/s

4K e/s 10Ks e/s

GeoAnalytics

Server

ArcGIS Enterprisewith real-time & big data capabilities

ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

Big DataIoT

GeoAnalytics

Server

1

2 43

MINIMUM environment

functional servers & spatiotemporal big data store

SHOULD BE on ISOLATED machines!!!

ArcGIS Enterprisewith real-time & big data capabilities

ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

Big DataIoT

GeoAnalytics

Server

1

2 3 64 75 8

RECOMMENDED environment

functional servers & spatiotemporal big data store

SHOULD BE on ISOLATED machines!!!

ArcGIS Enterprisewith real-time & big data capabilities

DesktopWeb Device

visualization

live & historic

aggregates & features

map & feature service

• Ingest high velocity real-time

data into ArcGIS

• Perform continuous analytics

on events as they are received

• Store observations in a

spatiotemporal big data store

• Run batch analytics on

stored observations

• Visualize high velocity &

volume data:

- as an aggregation

- as discrete features

• Notify those who need to

know about patterns of

interest

stream service

live features

ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

storage

ingestion

analytics

GeoAnalytics

Server

analytics

visualizing observation data2

Observation datatypes of observation data

movingsomething

that moves

• planes

• vehicles

• animals

• satellites

• storms

stationarystands still but

attributes change

• water gauges

• weather stations

• traffic sensors

• air quality

discretesomething that

“just happens”

• crimes

• lightning

• accidents

Observation datadefined

• An observation is a recording of a feature’s attribute values and location at a specific

moment in time.

- Observations are immutable, they happen and are typically not edited.

- Observations can be replayed over space & time.

- Moving observations are identifiable by a unique attribute, known as a TRACK_ID.

space & time

AM

Observations can be

shown at a specific time

PM

Moving observations can

be identified by a unique attribute

tracks

1

1

1

2

22

3

3

3

the tracks of moving observations

Moving observation illustration for two tracks

over space (X, Y) and time (T)

can be reconstructed

T

On-the-fly aggregationsof observations

• The spatiotemporal big data store enables you to aggregate data on-the-fly

- aggregating on-the-fly accommodates for real-time data

- you can perform exploratory analysis via any combination of space, time and attribute queries

- toggle from aggregation to raw feature view

- access feature level attributes in aggregate or feature level view

Demo

map & feature service

visualizing observation data

writing observation data

from GeoEvent Server3

Writing observation datafrom GeoEvent Server

ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

IoT

map & feature service

data source

input output

GeoEvent Service

Demo

into a spatiotemporal big data store

from GeoEvent Server

Writing observation data

on-the-fly aggregations4

Preparing data for on-the-fly aggregationsspatial indexing: geohash

• as data is written to a data source in the spatiotemporal big data store:

- a spatial index for geohash aggregation is continuously updated

geohash aggregation (based on a geohash index)

Preparing data for on-the-fly aggregationsspatial indexing: geohash & square

• as data is written to a data source in the spatiotemporal big data store:

- a spatial index for geohash aggregation is continuously updated

- a spatial index for square aggregation is continuously updated

square aggregation (based on a square index)geohash aggregation (based on a geohash index)

Preparing data for on-the-fly aggregationsspatial indexing: triangle

• as data is written to a data source in the spatiotemporal big data store:

- a spatial index for ‘pointy’ triangle aggregation is continuously updated

- a spatial index for ‘flat’ triangle aggregation is continuously updated

flat triangle aggregation (based on a flat triangle index)pointy triangle aggregation (based on a pointy triangle index)

Preparing data for on-the-fly aggregationsspatial indexing: hexagon (same as triangle)

• as data is written to a data source in the spatiotemporal big data store:

- a spatial index for ‘pointy’ hexagon (pointy triangle) aggregation is continuously updated

- a spatial index for ‘flat’ hexagon (flat triangle) aggregation is continuously updated

flat hexagon aggregation (based on a flat triangle index)pointy hexagon aggregation (based on a pointy triangle index)

Preparing data for on-the-fly aggregationsspatial indexing

• as data is written to a data source in the spatiotemporal big data store:

- up to four types of spatial indices are supported: geohash, square, pointy & flat hexagon/triangle

- this is in addition to a temporal index on the time field

- and an inverted index on each of the attribute fields

geohash

square

trianglepointy

triangleflat

hexagonflat

hexagonpointy

Visualizing on-the-fly aggregationsgeohash

geohash aggregation response geohash aggregation response

• map & feature services can query a data source in the spatiotemporal big data store with results

aggregated on-the-fly.

map servicefeature service

lodType=geohash&lod=2

feature servicelodType=geohash&lod=2&returnGeometry=false

geohash aggregation response

Visualizing on-the-fly aggregationssquare

map servicefeature service

lodType=square&lod=2

square aggregation response square aggregation response

• map & feature services can query a data source in the spatiotemporal big data store with results

aggregated on-the-fly.

Visualizing on-the-fly aggregationshexagon aggregation

map service

flat hexagon aggregation response

feature servicelodType=flatHexagon&lod=2

flat hexagon aggregation response

• map & feature services can query a data source in the spatiotemporal big data store with results

aggregated on-the-fly.

Visualizing on-the-fly aggregationsspatial projection

• some of the spatial indices support projection(s):

- the geohash spatial index only supports GCS WGS 1984, however you can project on-the-fly

- square, pointy & flat hexagon/triangle spatial indices are defined with a spatial reference

geohash aggregation in world mollweide projection (wkid = 54009) square aggregation in world mollweide projection (wkid = 54009)

Visualizing on-the-fly aggregationsdiscover which aggregations are enabled on a layer (data source)

Demo

geohash, square & hexagon

spatial projection

Visualizing on-the-fly aggregations

http://github.com/esri/aggregation-viewer-server-map-service http://github.com/esri/aggregation-viewer-client-feature-layer

writing analytic results

from GeoAnalytics Server5

ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

Big Data

GeoAnalytics

Server

Writing analytic resultsfrom GeoAnalytics Server

IoT

GeoAnalytics Serveranalytic capabilities

Summarize DataAggregate PointsSummarize NearbySummarize WithinReconstruct TracksCreate Panel

Find LocationsFind Similar Locations

Analyze PatternsCalculate DensityFind Hot SpotsCreate Space Time Cube

Use ProximityCreate Buffers

Manage DataCopy to Data StoreJoin Features

* time aware tools

ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

IoT

GeoAnalytics

Server

Writing analytic resultsfrom GeoAnalytics Server

• GeoAnalytics Server has the ability to:

- perform analytics against data sources in the spatiotemporal big data store

- and write the analytic results to the spatiotemporal big data store

Web GIS layers

new Web GIS layers

use your Web GIS layers through Pro, Portal,Python notebooks or the REST API

ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

GeoAnalytics

Server

big datafile shares

featureservice

.shp

shapefiles

.csv

text files

Hadoopfile system

Hive

Writing analytic resultsfrom GeoAnalytics Server

• GeoAnalytics Server has the ability to:

- perform analytics against files in an external big data file share

- and write the analytic results to the spatiotemporal big data store

new Web GIS layers

Web GIS layers

Loading datavia ‘Copy to Data Store’ GeoAnalytics tool

GeoAnalytics tool interface

output features

big datafile shares

.shp

shapefiles

.csv

text files

Hadoopfile system

Hive

featureservice

relationaldata store

spatiotemporalbig data store

Writing analytic resultsvia ‘Reconstruct Tracks’ GeoAnalytics tool

GeoAnalytics tool interface

big datafile shares

.shp

shapefiles

.csv

text files

Hadoopfile system

Hive

featureservice

relationaldata store

spatiotemporalbig data store

Writing analytic resultsvia ‘Reconstruct Tracks’ GeoAnalytics tool

output features

input features GeoAnalytics tool interface

R&D projects sneak-peek7

Polly Q*bert

project Pollyspatial indexing: polygon

• as data is written to the spatiotemporal big data store:

- a spatial index for polygon aggregation is continuously updated

polygonstate

polygoncounty

polygon aggregation (based on a county polygon index)polygon aggregation (based on a state polygon index)

projecton-the-fly & interactive space-time cubes

• leverage existing geohash, square & hexagon aggregation capabilities of

the spatiotemporal big data store to construct on-the-fly space-time cubes.

- enables exploratory & interactive analysis of real-time & historic observation data

ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

Big DataIoT

GeoAnalytics

Server

ArcGIS API for

JavaScript

& WebGL

client-side rendering ofspace-time cubes via WebGL

lodType=geohash&lod=2&time=1474675200000,1475279999000&timeInterval=1&timeUnit=hour

on-the-fly & interactive space-time cubes feature service

space-time geohash (based on a geohash aggregation) response

project

feature servicelodType=geohash&lod=2

geohash aggregation response

lodType=geohash&lod=2&time=1474675200000,1475279999000&timeInterval=1&timeUnit=hour&f=json

also considering:f=protobuf

an array of time-slices of aggregated features

&returnGeometry=false &returnGeometry=false

Demo

on-the-fly & interactive space-time cubes

project

Q*bert

Summary

http://links.esri.com/geoevent-sbds

spatiotemporal big data store

• The spatiotemporal big data store enables:

- GeoEvent Server to write high velocity & volume observation data

- On-the-fly aggregations to be visualized using Map & Feature Services

- GeoAnalytics Server to read & write high volume analytic results

• To learn more:

- See the ‘Spatiotemporal Big Data Store’ tutorial

Real-Time & Big Data GIS

• Building Android Location Awareness with GeoEvent Server Thu, 6:00-6:30pm, Mesquite C

• GeoEvent Server: Making 3D Scenes Come Alive Fri, 8:30-9:30am, Primrose C-D

• GeoEvent Server: Internet of Things (IoT) Fri, 1:00-2:00pm, Primrose

other sessions

Questions / Feedback?

http://links.esri.com/geoevent

http://links.esri.com/geoevent-forum

To learn more:

Adam MollenkopfReal-Time & Big Data GIS Capability Lead, Esri

amollenkopf@esri.com

@amollenkopf

Ricardo TrujilloReal-Time & Big Data GIS Developer, Esri

rtrujillo@esri.com

@rtrujill007