event driven streaming analytics - demostration on architecture of iot

EVENT DRIVEN STREAMING ANALYTICSDemonstration on Architecture of IoT

Lei Xu 2016.09

1.DEMO SCENARIO

Taxi/Devices

Coordinates & Payment Info

Devices generate transactions after payment

Taxi/Devices

Data FlowCoordinates & Payment Info

IoT Gateway

Applications & Services

Data Management

Cloud Infrastructure

IoT Connectivity Platform

Online Analytics • Dropoff-district Count during the

last 30 min • Real-time Taxi Dropoff Events • 10 Most Frequent Routes(Pickup-

area Dropoff-area) during the last 30 minutes

• 10 Most Profitable areas within the last 15 minutes

Offline Analytics • 10 Most Frequent Routes by week day • 10 Most Profitable Areas by week day • Most Profitable Hours at the 10 Most

Profitable Areas by week day

Transaction being sent to IoT Platform for Analytics

Thousands of Browsers Hundreds of BI Clients

Data ManagementMulti-Channel Data Integration

Structured & Unstructured Data

Analytics & Machine Learning

Cloud InfrastructureCloud Native Platform

Scalability & High Availability

Security & Governance

Visualize DataReporting, BI & User Apps

History Data & Real-time Data

Multiple Devices & Platforms

IoT GatewayDevice Management

Authentication & Authorization

Gateways & Interfaces

Apps & ServicesRule Engine & Process Mgmt.

Complex Event Processing

Business & Data MicroServices

IoT Platform

Components

2.REQUEST DRIVEN FLOW

Storing Enriched Data

Time Based Enrichment &

Filtering

Receiving & Storing Raw Events

Time Based Batch Aggregation

Storing Aggregated Data

Sending Raw Events

Polling Requests for Latest Data

Taxi/Devices

Traditional Request Driven Flow

Event-driven Flow

Request-driven Flow

Time-based Flow

Analytics

BI Visualization

DRAWBACKS & CHALLENGES• Polling + Request/Response Style => Overhead

• Time Based Batch Process => High Latency

• Architecture - Monolith or SOA will work well with HA?

• How to scale & how to maintain system performance with lots of user requests?

• How to scale & how to maintain system performance with huge amount of data?

• Portable for both Cloud and On-Premise environments?

REQUEST DRIVEN & EVENT DRIVENRequest Driven Event Driven

How is an action being taken?

As a response to a specific request

Triggered by the fact of a specific situation

When is an action being taken?

When the request is being processed

Determined by the context of the situation

What happens when the event/request occurs? A response is produced

The event can be ignored, increment the state, trigger an internal derive event, or

trigger a solution

http://epthinking.blogspot.jp/2012/12/more-on-request-driven-vs-event-driven.html

3.EVENT DRIVEN FLOW

Publishing/Subscribing

Enriched Events

Enrichment &

Filtering

Receiving Raw Events

Real-time Aggregation/

Analytics

Publishing/Subscribing

Aggregated Events

Sending Raw Events

Events Visualization

Online AnalyticsTaxi/Devices

Expectations

• Event Driven

• Low Latency

• Scalability

• High Availability

• Portability

• Light Weight

Event-driven Flow

Publishing/Subscribing

Enriched Events

Enrichment &

Filtering

Receiving Raw Events

Real-time Aggregation/

Analytics

Publishing/Subscribing

Aggregated Events

Sending Raw Events

Events Visualization

Online & Offline Analytics

Storing Data for Analytics

Analytics

BI Visualization

Taxi/Devices

Event-driven Flow

Request-driven Flow

Publishing/Subscribing

Enriched Events

Enrichment &

Filtering

Receiving Raw Events

Real-time Aggregation/

Analytics

Publishing/Subscribing

Aggregated Events

Sending Raw Events

Events Visualization

Machine Learning Loopback to Streaming

Storing Data for Analytics

Analytics & Machine Learning

BI Visualization

Taxi/Devices

Event-driven Flow

Request-driven Flow

4.EVENT DRIVEN ARCHITECTURE

Binding Abstraction Layer

Messaging Options

Data Pipeline

Data Microservices

Data

Spring Cloud Stream App

Spring Cloud Stream App

Spring Cloud Stream App

Spring Cloud Stream/Data Flow as Event Driven Architecture

Data

Filter, Enrich, Transform, Aggregation, Rules, CEP, Analytics

Publishing/Subscribing Geocoded Events

Reverse Geocoding

Receiving Raw Events

Real-time Aggregation/Analytics

Publishing/Subscribing Aggregated Events

Sending Raw Events

Events Visualization

Simulator

HTTP Source

MongoDB Processor

GemFire Sink

GemFire Source

WebSocket Sink

WebSocket Sink

Reverse GeocodingReal-time Aggregation/Analytics

Messaging

Apps & Services – Online Analytics

Publishing/Subscribing Geocoded Events

Reverse Geocoding

Receiving Raw Events

Real-time Aggregation/Analytics

Publishing/Subscribing Aggregated Events

Sending Raw Events

Events Visualization

Simulator

HTTP Source

MongoDB Processor

GemFire Sink

GemFire Source

WebSocket Sink

WebSocket Sink

Reverse GeocodingReal-time Aggregation/Analytics

Messaging

BI Visualization

Analytics & Machine Learning

Storing Data for Analytics

HDFS Sink

SQL Query

Apps & Services – Online & Offline Analytics

MPP SQL on HDFS

Publishing/Subscribing Geocoded Events

Reverse Geocoding

Receiving Raw Events

Real-time Aggregation/Analytics

Publishing/Subscribing Aggregated Events

Sending Raw Events

Events Visualization

Simulator

HTTP Source

MongoDB Processor

GemFire Sink

GemFire Source

WebSocket Sink

WebSocket Sink

Reverse GeocodingReal-time Aggregation/Analytics

Messaging

BI Visualization

Analytics & Machine Learning

Storing Data for Analytics

MPP SQL on HDFS

HDFS Sink

SQL QueryMachine Learning

Apps & Services – Machine Learning Loopback to Streaming

Machine Learning

5.ONLINE ANALYTICS LOCAL DEPLOYMENT

MongoDB Processor

Real-time Aggregation/Analytics

Spring Boot App

Spring Cloud Stream App

Spring Cloud Stream App

HTTP Source

GemFire Sink

GemFire Source

WebSocket Sink

Simulator

Spring Cloud Stream App

Spring Cloud Stream App

Spring Cloud Stream App

Java CLI App

Messaging

WebSocket Sink

Spring Cloud Stream App

MVW App

Data Flow

Deployment

DeploymentReverse Geocoding

MongoDB Processor

Real-time Aggregation/Analytics

Spring Boot App

Spring Cloud Stream App

Spring Cloud Stream App

HTTP Source

GemFire Sink

GemFire Source

WebSocket Sink

Simulator

Spring Cloud Stream App

Spring Cloud Stream App

Spring Cloud Stream App

Java CLI App

Messaging

WebSocket Sink

Spring Cloud Stream App

MVW App

Data Flow

Deployment

Scale Out When NecessaryReverse Geocoding

MongoDB Processor

Real-time Aggregation/Analytics

Spring Boot App

Spring Cloud Stream App

Spring Cloud Stream App

HTTP Source

GemFire Sink

GemFire Source

WebSocket Sink

Simulator

Spring Cloud Stream App

Spring Cloud Stream App

Spring Cloud Stream App

Java CLI App

Messaging

WebSocket Sink

Spring Cloud Stream App

MVW App

Data Flow

Deployment

RuntimeReverse Geocoding

6. PRODUCTION LEVEL DEPLOYMENT

MQTT Source

Enrichment Processor

IMDG Sink

Machine Learning

Processor

WebSocket Sink

MPP/HDFS Sink

Real-time IMDG Sink

MPP/HDFS Source

Data Pipelines

IMDG Source

Broker by RabbitMQ

WebSocket by RabbitMQ

Front-end Services

Spring Cloud Stream Apps

MVW App Spring Cloud Stream Apps

Data MicroServicesBusiness MicroServices

……

Containerized Applications

Taxi/Devices

• Event Driven Spring Cloud Stream

• Low Latency Messaging by RabbitMQ Real-time by GemFire

• Scalability Apps Scale-Out by Containers Governance by Data Flow Services Scale-out

• High Availability Platform HA by Cloud Native Platform or PaaS

• Portability Apps & Services portable to other Cloud Native Platforms

• Light Weight Loosely Coupled MicroServices

or or etc.

Back-end ServicesRabbitMQGeode

MongoDB HAWQ

or

Spring Cloud Data Flow

7.DEPLOYMENT & RUNTIME SCREENSHOTS

• Real-time Taxi Dropoff Events

• Dropoff-district Count

• 10 most frequent routes

• 10 most frequent routes

8. LOCAL DEPLOYMENT DEMO VIDEO