arcgis geoevent server: leveraging stream services · leveraging stream services this is a...

Real-Time GIS:Leveraging Stream Services

Mark Bramer

Senior Technical Analyst

Esri Professional Services

Vienna, VA

mbramer@esri.com

Josh Joyner

Product Engineer

Real-Time GIS

Product Team

jjoyner@esri.com

Kevin Armstrong

Solutions Engineer

Esri, Charlotte, on behalf of

AccuWeather

karmstrong@esri.com

In this presentationLeveraging Stream Services

• Two fundamental topics:

• Stream services

• Stream layers

• We expect that you are familiar with:

- Web maps and feature layers

- The ArcGIS GeoEvent Server (“GeoEvent”)

• This presentation will demonstrate and discuss:

- Publishing stream services using the GeoEvent Manager

- Discovering stream services in the ArcGIS REST Services Directory

- Advantages stream services provide vs. traditional feature services

- Support for stream services available in the 10.5 product release

- Analysis with stream services

- Viewing stream layers in maps and apps

Leveraging Stream Services

This is a beginner-intermediate level technical workshop

ArcGIS Server

DesktopWeb Device

GeoEvent

Processor

Real-Time GISIntegration and exploitation of streaming data

• Integrates real-time

streaming data

into ArcGIS

• Performs continuous

processing and

real-time analytics

• Sends updates and alerts

to those who need it

where they need it

Real-Time GISIntegration and exploitation of streaming data

• Efficient real-time

display of live data in

web applications

ArcGIS Server

GeoEvent

Extension

DesktopWeb Device

Stream Service

Stream Layer

Real-Time GISIntegration and exploitation of streaming data

ArcGIS Server

GeoEvent

Extension

DesktopWeb Device

Stream Service

Stream Layer

10s of thousands of e/s

ArcGIS Spatiotemporal

Big Data Store

Spatiotemporal

Big Data Store

• Can ingest real-time

higher velocity data into

ArcGIS and can store it.

• Can visualize high

velocity & volume data

- in feature form

- in aggregate form

- live & historically

• Visualization can scale.

ArcGIS Enterprisewith real-time & big data capabilities

• Ingest high velocity real-time

data into ArcGIS

• Perform continuous analytics

on events as they are received

• Store observations in a

spatiotemporal big data store

• Run analytics on stored

observations

• Visualize high velocity &

volume data:

- as an aggregation

- as discrete features

• Notify those who need to

know about patterns of

interest

ArcGIS

Enterprise

DesktopWeb Device

GeoEvent

Server

visualization

live & historic

aggregates & features

spatiotemporal

big data store

map & feature servicestream service

live features

storage

ingestion

analytics

GeoAnalytics

Server

analytics

Stream Services

Real-time GIS – Two patternsStream services vs traditional feature services

• Feature layers pull from feature services

- Web apps poll to get periodic updates

• Stream layers subscribe to stream services

- Web apps subscribe to immediately receive data

- Low latency and high throughput

GeoEvent Server

Your

Applications

Stream Layer

Map Services

Feature Services

Feature Layer

feature layers

Update a Feature

Add a Feature

EGDB

Polling

(Pull)

Stream Services

Send Features to a Stream Service

ArcGIS Server

Two patterns, two important differences

Stream services vs traditional feature services

• Feature services persist their data in a Geodatabase

• Stream services broadcast their data without first persisting the data

GeoEvent Server

ArcGIS Server Your

Applications

Stream Layer

Map Services

Feature Services

Feature Layer

feature layers

Update a Feature

Add a Feature

EGDB

Polling

(Pull)

Stream Services

Send Features to a Stream Service

Part of configuring your GeoEvent output

Stream services publishing using GeoEvent Manager

Check to make sure you

have specified the correct

GeoEvent Definition …

Part of configuring your GeoEvent output

Stream services publishing using GeoEvent Manager

Your stream service

schema is based on your

GeoEvent Definition

Click ‘Publish Stream Service’

Part of configuring your GeoEvent output

Stream services publishing using GeoEvent Manager

Part of configuring your GeoEvent output

Stream services publishing using GeoEvent Manager

Optional store latest - uses

companion feature service

Optional related features –

facilitates stationary

sensors with dynamic

values

Click ‘Publish’

Part of configuring your GeoEvent output

Stream services publishing using GeoEvent Manager

Click the service to open its

REST specification page…

Stream services in the ArcGIS REST Services Directory

Click the JSON link to view

a more complete

specification of the

service…

Click the Subscribe link to

jump to an HTML page and

see the JSON being

broadcast by the service…

Stream services in the ArcGIS REST Services Directory

Stream services broadcast Esri Feature JSON

Subscribe to view stream service data

Started, stopped, and configured using ArcGIS Server Manager

Stream service administration

Stream service preview

Stream Layers

ArcGIS Server

Feature service

Stream LayerWhat is it?

• A layer in the Javascript API

- Available since version 3.6

• Draws data on map using client-side graphics

Stream Layer

GET

HTTPPUSH

Web

Socket

esri/layers/GraphicsLayer

esri/layers/FeatureLayer

esri/layers/StreamLayer

ArcGIS Server

GeoEvent

Server

Feature Layer

RESPONSE

Geometry

• Attributes

• Symbol

Graphic

Graphics Layer

Advantages to using stream layersFast, efficient

• More responsive and more efficient than feature layers

• Stream layers display immediately and refresh automatically

• Data is only sent to the client once

• Messages are sent without extra headers

Feature Layer

RequestResponse

ArcGIS Server

Feature service

Stream Layer

ArcGIS Server

GeoEvent Server

DataEstablish

Connection

Stream layer requirementsBrowser support for web sockets / JavaScript application support

• ArcGIS GeoEvent Server

- Stream services are published as GeoEvent output connectors

• Web Browser that supports Web Sockets

- http://caniuse.com/websockets

• Network support for the Web Socket protocol

- ws:// wss://

• No custom plug-in required: Standard JavaScript

implementation

10 +

6 +

6 +

14 +

DEMOStream Services and Stream Layers

Dynamic GeoFences

• A GeoEvent Service configures the flow of GeoEvents

• Processors and Filters rely on GeoFences for spatial analytics

• GeoFence synchronization can receive data broadcast from a stream service

Stream services enable dynamic GeoFences

Applying real-time analytics

• GeoFences imported once or

continually using Synchronization

Rules

• GeoFences come from feature

services or stream services

• GeoFences can be active or inactive

- Boolean value

- Date range

• GeoFences usually polygons

- Can also be points or lines

Stream services enable dynamic GeoFences

Applying real-time analytics

Spatial Operators

intersect disjoint

touches contains

crosses equals

overlaps within

• Spatial operators at 10.2:

• Spatial operators at 10.3 and

beyond:

inside outside

enter exit

33.322336

-117.416323

Dynamic GeoFences

33.322125

-117.398225

Dynamic GeoFences

DEMODynamic Geofences

Web Maps and Apps

• ArcGIS Online and Portal for ArcGIS Web Maps

• ArcGIS Online and Portal for ArcGIS web application templates

• Web applications built using Web AppBuilder

• Your own web apps that use the ArcGIS API for JavaScript

What can I use to consume stream services?

Support for stream services in the 10.3, 10.4 and 10.5 product releases

Web Maps and Apps…

Web Maps

Real-time data in a Web Map

• Add menu…

• Add layer from web…

• Add stream service URL

Real-Time Data in a Web Map

Real-Time Data in a Web Map

• Stop/Start

• Spatial Filter

• Clear

Streaming Controls

Symbology

• Change layer’s appearance

Symbology

• Heading

• Tracks

• Filter on data content

- i.e. “WHERE” clause

Streaming ControlsReal-Time Data in a Web Map

Heading between 45 and 135 (east) Heading between 225 and 315 (west)

• Each user has unique session

• Stream service knows what to send to each user

User-specificReal-Time Data in a Web Map

Web Maps and Apps…

Web App Templates

• “Share” button

• “Create a Web App”

Share in ArcGIS Online / Portal

Real-Time Data in Web App Templates

Web App Templates

• “Configurable Apps” tab

• Many to choose from

• Many are theme-focused

- Compare

- Elevation Profile

- Impact Summary

• Can publish, download

or preview

“Pre-packaged” Web Apps

Web Maps and Apps…

Web AppBuilder

• Also via the “Share” button

• Also use “ Create a Web App”

Real-Time Data in Web AppBuilder AppsShare in ArcGIS Online / Portal

Web AppBuilderPre-packaged Web App …

• “Web AppBuilder” tab

• Enter a title, add some tags,

enter a summary

• “Get Started”

Builder

WYSIWYGWeb AppBuilder

• “What You See Is What You

Get” user interface

• Style/Colors

• Layout

• Widgets

• Map, extent

• Branding

• Previews

• Launch

• Discoverable

Web AppBuilderPublished

Web AppBuilderStream Widget

• Stream Widget allows simple

configuration of stream layer in

your web app

• Stop/Start

• Clear

• Draw previous observations

• Spatial Filter

• Attribute Filter

Web AppBuilderStream Widget

Web Maps and Apps…

Custom Web Application

Two Lines of Code!!

Real-time data in Your Own Web App

• Any app using Esri

JavaScript API

• Dojo “require”

• Constructor

streamLayer = new StreamLayer(https://geoeventsample3.esri.com:6443/arcgis/rest/services/SeattleBus/StreamServer, {purgeOptions: { displayCount: 10000 },infoTemplate: new InfoTemplate("Attributes", "${*}")

});

require(["esri/map","esri/toolbars/draw",

"esri/layers/StreamLayer","esri/InfoTemplate","esri/graphic","esri/symbols/SimpleFillSymbol","esri/symbols/SimpleLineSymbol","dojo/_base/Color","dojo/on","dojo/domReady!"

Streaming AccuWeather Services

Paul Raymond

Director Product Management

SkyGuard Division

AccuWeather Enterprise Solutions

• Doppler radar-based products

• Update every few minutes

• Can be streamed

• Can be processed using ArcGIS GeoEvent Extension for

Server

Real-time weather solutions

61| ©2016, AccuWeather Enterprise Solutions, Inc. Confidential and Proprietary

• Attributes of radar provide excellent sense for

– 0.5 km Pixel size radar, updated every 5 minutes

– Severity of thunderstorms

– Potential for hail and hail size determination

– Potential for storm rotation and tornado development

– Can be displayed in all Esri products

Doppler Radar

62| ©2016, AccuWeather Enterprise Solutions, Inc. Confidential and Proprietary

StormPath™ from AccuWeather

63| ©2016, AccuWeather Enterprise Solutions, Inc. Confidential and Proprietary

• Uses Radar to improve storm attributes

– Predicted path of storm

– 15-, 30, and 45-minute timelines

– Potential for storm rotation and tornado development

– Improvements based on proprietary AccuWeather technology

StormPath™ from AccuWeather

64| ©2016, AccuWeather Enterprise Solutions, Inc. Confidential and Proprietary

• MinuteCast™

– Minute-by-minute precipitation forecast for the next 2 hours.

– Utilizes same radar for StormPath

– USA, Canada, Europe and Japan (soon China)

What’s else is new?

65| ©2016, AccuWeather Enterprise Solutions, Inc. Confidential and Proprietary

• We set

MinuteCast at (4)

four locations

along the route of

the marathon.

• It was raining for

most of the race.

• Allows racers to

know start and

stop of rain

MinuteCast™ - Boston Marathon 2015

66| ©2016, AccuWeather Enterprise Solutions, Inc. Confidential and Proprietary

Add AccuWeather Widget to Dashboard

67| ©2016, AccuWeather Enterprise Solutions, Inc. Confidential and Proprietary

• Improved “Store Latest” publishing to a spatiotemporal big data store

or relational data store

• Specify an existing feature service – or publish a new one for use with

Related Features and Store Latest capabilities

• Stream service validation for TrackID fields and necessary Credentials

• Unique GeoFence Synchronization parameters for Feature and

Stream services

Stream Services

What’s new in 10.5

Please Take Our Survey on the Esri Events App!

Select the session

you attended

Scroll down to find

the survey

Complete Answers

and Select “Submit”

Download the Esri Events

app and find your event

Print Your Certificate of Attendance

Print stations located in the 140 Concourse

Monday

12:30 PM – 6:30 PM

GIS Solutions Expo,

Hall B

5:15 PM – 6:30 PM

Expo Social,

Hall B

Tuesday

10:45 AM – 5:15 PM

GIS Solutions Expo,

Hall B

6:30 PM – 9:30 PM

Networking Reception,

Smithsonian National Air

and Space Museum

Questions?