internet of things (iot) and big data

BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

Internet of Things (IoT) and Big Data30.9.2016 – DOAG 2016 Big Data Days

Guido Schmutz

Guido Schmutz

Working for Trivadis for more than 19 yearsOracle ACE Director for Fusion Middleware and SOACo-Author of different booksConsultant, Trainer, Software Architect for Java, SOA & Big Data / Fast DataMember of Trivadis Architecture BoardTechnology Manager @ Trivadis

More than 25 years of software development experience

Contact: [email protected]: http://guidoschmutz.wordpress.comSlideshare: http://www.slideshare.net/gschmutzTwitter: gschmutz

Internet of Things (IoT) and Big Data2

Agenda

1. Introduction

2. Towards an IoT Architecture3. IoT Refererence Architecture

4. Summary


Introduction


Internet of Things (IoT) Wave

Internet of Things (IoT): Enabling communication between devices, people & processes to exchange useful information & knowledge that create value for humans

Term was first proposed by Kevin Ashton in 1999

Source:TheEconomistSource:Ericsson,June2016


Reasons why IoT opportunity is occurring now ?

Affordable hardware

• Costs of actuators & sensors have been cut in half over last 10 years

Smaller, more powerful hardware

• Form factors of hardware have shrunk to millimeter or even nanometer levels

Ubiquitous & cheap mobility

• Cost for mobile devices, bandwidth and data processing has declined over last 10 years

Availability of supporting tools

• Big data tools & cloud based infrastructure have become widely available

Mass market awareness

• IoT has surpassed a critical tipping point• Vision of a connected world has reached

such a followership that companies have initiated IoT developments

• Commitment is irreversible


The Sensing-as-a-Service Model


Towards an IoT Architecture


Key Challenges for building an IoT application

1. Connect: How to collect data from intelligent devices?

• Abstract complexity associated with device connectivity• Standardize integration of devices with enterprise

2. Analyze: How to analyze IoT data?

• Reduce noise and detect business event at real-time• Enable historical big-data analysis

3. Integrate: How to integrate IoT data & events with enterprise infrastructure?

• Make enterprise processes IoT friendly• Allow enterprise & mobile applications to control devices


Today) Existing Service-/API Architecture as a base

19

MobileApps

DB

Rich(Web)ClientApps

DB

APIG

atew

ay

EnterpriseServiceBu

s(ESB)/Da

taIntegration

EnterpriseApps

WS

ExternalCloudServiceProviders

BPMandSOAPlatform

Event

BusinessLogic/Rules

BusinessIntelligence

Services

WS

Event

Processes

Visualization

Analytics

DB

REST/SOAP

REST/SOAP

REST/SOAP

SOAP

Various

SQL

SOAPREST

ServiceBu

sOracleDa

taIntegrator

APIG

atew

ay

SOASuite

BPMSuite

BusinessActivityMonitoring

Internet of Things (IoT) and Big Data19 =oneway =request/response

REST/SOAP

REST/SOAP

IoT 1a) Reuse exiting Service-/API-based Architecture

IoT SmartDevices

20

MobileApps

DB

Rich(Web)ClientApps

DB

EnterpriseServiceBu

s(ESB)/Da

taIntegration

EnterpriseApps

WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

REST/SOAP

REST

REST

JMS/REST

SOAP

Various

SQL

SOAP REST

WebSocketJMS

ServiceBu

sOracleDa

taIntegrator

APIG

atew

ayAP

IGatew

ay

JMS

JMS

WeblogicJM

S

SOASuite

BPMSuite



IoT 1a) Challenges


• Do IoT devices contain enough resources to communicate directly over the internet (HTTP or JMS) ?

• Should the device only collect data (sense) or is there also the way back necessary (actuator) ?

• Can JMS be used from external devices (firewalls allow traffic over JMS) ?

• How many IoT devices are planned short and long term? How often do they send to the backend? Are the JMS server as well as the ESB capable to deal with the resulting message volume ?

• What are the operations on IoT messages / events, only simple transformations, filter and routing operations?

REST/SOAP

REST/SOAP

IoT 1b) Reuse existing Service-/API-based Architecture

IoT SmartDevices

22

MobileApps

DB

Rich(Web)ClientApps

DB

APIG

atew

ay

EnterpriseServiceBu

s(ESB)/Da

taIntegration

EnterpriseApps

WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

RESTHTTP

REST

REST HTTP

JMSAP

IGatew

ay

ServiceBu

sOracleDa

taIntegrator

SOAP

Various

SQL

SOAP REST

WebSocketJMS

JMS

JMS

REST/SOAP

WeblogicJM

S

SOASuite

BPMSuite



REST/SOAP

REST/SOAP

IoT 2) Adding Event Hub and optional IoT Gateway

23

MobileApps

DB

Rich(Web)ClientApps

DB

ESB/D

ataIntegration

IoTDevices

IoTGateways

IoT SmartDevices

EnterpriseApps

WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

REST

Kafka/MQTT/REST

REST

Kafka/MQTT/REST

MQTT

Kura

Even

tHub

Kafka

ServiceBu

s

OracleDa

taIntegrator

REST

REST

SOAP

Various

SQL

SOAP REST

WebSocketKafka

JMS

JMS

APIG

atew

ayAP

IGatew

ay

REST

Kafka

REST/SOAP

REST

REST

SOASuite

BPMSuite


MQTT


MQTT

=oneway =request/response

How to implement an Event Hub? Apache Kafka to the rescue

Distributed publish-subscribe messaging system

Designed for processing of high-volume, real time activity stream data (logs, metrics collections, social media streams, …)

Topic Semantic

does not implement JMS standard

Initially developed at LinkedIn, now part of Apache

Kafka Cluster

Consumer Consumer Consumer

Producer Producer Producer


Oracle’s Service Bus as a consumer of Kafka

ServiceBus12c

CloudApps

BusinessService Cl

oud

ProxyServiceKa

fka

Clou

dAP

I

MobileApps Pipeline

Routing

KafkaSensor/IoT

WebApps

BusinessService RE

ST

BusinessService W

SDL

BackendAppsRE

ST

BackendAppsW

SDLProxy

ServiceKafka Pipeline

Routing

Database

DB CDC

StreamProcessing


IoT 2) Solutions & Challenges


Solutions

• Event Hub solves the potential scalability issue of JMS

• IoT Gateway makes sure that lightweight sensors can connect to the internet / send their data

Challenges

• Where to do complex analytics on the events? Is it scalable?

• Can we really send all data down to backend? Network bandwidth?

REST/SOAP

REST/SOAP

IoT 3) Adding Stream Processing / Analytics in Backend

27

MobileApps

DB

Rich(Web)ClientApps

DB

(ESB

)/DataIntegration

IoTDevices

IoTGateways

IoT SmartDevices

Even

tHub

StreamProcessing

ESP/CEP

DB

DB

Even

tHub

EnterpriseApps

WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

StreamAnalytics

ServiceBu

s

OracleDa

taIntegrator

Kafka

MQTT Kafka

Kafka

SOAP

Various

SQL

SOAP REST

WebSocketKafka

JMS

JMSAP

IGatew

ayAP

IGatew

ay

REST

MQTT

REST

REST

Kafka

Kafka/MQTT/REST

Kafka/MQTT/REST

REST

Kafka

REST/SOAP

REST

REST

Kura

SOASuite

BPMSuite



Kafka

MQTT



Solutions

• Stream Processing handles complex analytics on events in a scalable manner before sending events to ESB / backend systems

Challenges

• Can we really send all data down to backend? Network bandwidth?

Oracle’s Stream Analytics as consumer of Kafka/MQTT

OracleStreamAnalytics

StreamAnalytics

Kafka

Kafka

MobileAppsKafka

Sensor/IoT

WebApps

MachineData

DB CDC

Kafka

MQTT


REST/SOAP

REST/SOAP

IoT 4) Adding Industry 4.0 Data Sources (machine data)

30

MobileApps

DB

Rich(Web)ClientApps

DB

(ESB

)/DataIntegration

IoTDevices

IoTGateways

IoT SmartDevices

Even

tHub

StreamProcessing

ESP/CEP

DB

DB

Even

tHub

EnterpriseApps

WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

StreamAnalytics

ServiceBu

s

OracleDa

taIntegrator

Kafka

MQTT

Kafka

Kafka

SOAP

Various

SQL

SOAP REST

WebSocketKafka

JMS

JMSAP

IGatew

ayAP

IGatew

ay

REST

MQTT

REST

REST

Kafka

Kafka/MQTT/REST

Kafka/MQTT/REST

REST

Kafka

REST/SOAP

REST

REST

Kura

SOASuite

BPMSuite



I4.0Machine

DB CDC GoldenGate

Kafka/MQTT/REST

Kafka

MQTT

GoldenGate Gateway

Oracle’s GoldenGate for Change Data Capture ofexisting database


MachineData

DB

Kafka

OracleGoldenGate

Delivery

Capture

Pump

JMS

MachineData

DB OracleGoldenGate

Delivery

Capture

Pump HBase

HDFS

REST/SOAP

REST/SOAP

IoT 5) Adding Stream Processing / Analytics at Edge

32

MobileApps

DB

Rich(Web)ClientApps

DB

(ESB

)/DataIntegration

IoTDevices

IoTGateways

IoT SmartDevices

Even

tHub

StreamProcessing

ESP/CEP

DB

DB

Even

tHub

EnterpriseApps

WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

StreamAnalytics

ServiceBu

s

OracleDa

taIntegrator

Kafka

Kafka

Kafka

SOAP

Various

SQL

SOAP REST

WebSocketKafka

JMS

JMSAP

IGatew

ayAP

IGatew

ay

Kafka/MQTT/REST

REST REST

Kafka

Kafka

REST

RESTREST

REST/SOAP

REST

MQTT

SOASuite

BPMSuite


ESP/CEP

EdgeAnalytics

MQTT


Kafka/MQTT/REST

MQTT


I4.0Machine

DB CDC GoldenGate

Kafka/MQTT/REST

Kafka



Solutions

• Stream Processing at the edge / gateway allows to reduce the amount of messages send to the backend (cloud / on premises) if necessary

Challenges

• What if we need the raw events persisted? Where do to that?

REST/SOAP

REST/SOAP

IoT 6) Adding Raw Data Storage and Batch Analytics

34

MobileApps

DB

Rich(Web)ClientApps

DB

(ESB

)/DataIntegration

IoTDevices

IoTGateways

IoT SmartDevices

Even

tHub

Even

tHub

EnterpriseApps

WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

ServiceBu

s

OracleDa

taIntegrator

Kafka

KafkaKafka

SOAP

Various

SQL

SOAP REST

WebSocket

JMS

JMSAP

IGatew

ayAP

IGatew

ay

REST REST

Kafka

Kafka

SQL

RESTREST

REST/SOAP

StreamProcessing

ESP/CEP

DB

DB

BigDataProcessingHDFS

BatchProcessing

DB

Kafka

Kafka

HDFS

ESP/CEP

EdgeAnalytics

MQTTMQTT

StreamAnalytics

Hadoop/Spark

OracleBigDataAppliance

SOASuite

BPMSuite



Kafka/MQTT/REST

Kafka/MQTT/REST


I4.0Machine

DB CDC GoldenGate

MQTT

Kafka/MQTT/REST

Kafka



Solutions

• Adding Big Data platform allows to store all raw data in the distributed file system in a scalable and reliable manner

Challenges

• How can we leverage the Big Data platform for more than just storing raw data? How does it combine with the stream processing?

Continuous Ingestion / Fan-In from the Edge

DBSource

BigData

Log

StreamProcessing

IoT Sensor

EventHub

Topic

Topic

REST

Topic

IoT GW

CDCGW

Conn

ectCDC

DBSource

Log CDC Native

IoT Sensor

IoT Sensor

37

DataflowGW

Topic

Topic

Queue

MQTTGW

Topic

DataflowGW

Dataflow

TopicRE

ST37FileSource

Log

Log

Log

Social

Native


Topic

Topic

Challenges for Ingesting Sensor Data

Internet of Things (IoT) and Big Data

• Multitude of sensors

• Multiple Firmware versions

• Bad Data from damaged sensors

• Data Quality

38

REST/SOAP

REST/SOAP

IoT 6a) Adding Data Mining / Machine Learning and Model execution

40

MobileApps

DB

Rich(Web)ClientApps

DB

(ESB

)/DataIntegration

IoTDevices

IoTGateways

IoT SmartDevices

Even

tHub

Even

tHub

EnterpriseApps

WS


BPMandSOAPlatform

Event

BusinessLogic/Rules


Services

WS

Event

Processes

Visualization

Analytics

DB

ServiceBu

s

OracleDa

taIntegrator

Kafka

KafkaKafka

SOAP

Various

SQL

SOAP REST

WebSocket

JMS

JMSAP

IGatew

ayAP

IGatew

ay

REST REST

Kafka

Kafka

SQL

RESTREST

REST/SOAP

StreamProcessing

ESP/CEP

DB

DB

BigDataProcessingHDFS

BatchProcessing

DB

Kafka

Kafka

HDFS

ESP/CEP

EdgeAnalytics

MQTTMQTT

StreamAnalytics

Hadoop/Spark

OracleBigDataAppliance

SOASuite

BPMSuite



Kafka/MQTT/REST

Kafka/MQTT/REST


I4.0Machine

DB CDC GoldenGate

MQTT

Kafka/MQTT/REST

Kafka

IoT Reference Architecture


IoT Services

IoT Logical Reference Architecture

IoTDevice

Sensor

Actuator

IoT Gateway

Storage

UIApp

StreamingAnalytics

EnterpriseApplications

BPMandSOAPlatformStreaming

Analytics

Storage

EndpointManagement

EventHub

ServiceBus

EventHub Event

Hub

ServiceBus

BigData/BI

Storage

Services Processes

UIApp

Storage

BulkAnalytics UI

BulkAnalytics UI

Storage

StreamingAnalytics

ServiceBusAP

I

RESTSOAPHTTPKAFKA

MQTTCoAPXMPPDDSAMQPKAFKAWIFI

BLEZigBeeWIFIWired


IoT Services

IoT Logical Reference Architecture – Oracleon premises

IoTDevice

Sensor

Actuator

IoT Gateway

Storage

UIApp

StreamingAnalytics



Analytics

Storage

EndpointManagement

EventHub

ServiceBus

EventHub Event

Hub

ServiceBus

BigData/BI

Storage

Services Processes

UIApp

Storage

BulkAnalytics UI

BulkAnalytics UI

Storage

StreamingAnalytics

ServiceBusAP

I

RESTSOAPHTTPKAFKA


BLEZigBeeWIFIWired

EdgeAnalytics


SOASuite

BPMSuiteServiceBus

OracleDataIntegrator

StreamAnalytics

BigDataAppliance

StreamAnalytics

ServiceBus

APIG

atew

ayInternet of Things (IoT) and Big Data44

OracleIoT CSGateway

OracleIoT CSClientLibrary

IoT Services

IoT Logical Reference Architecture – OracleCloud Services

IoTDevice

Sensor

Actuator

IoT Gateway

Storage

UIApp

StreamingAnalytics



Analytics

Storage

EndpointManagement

EventHub

ServiceBus

EventHub Event

Hub

ServiceBus

BigData/BI

Storage

Services Processes

UIApp

Storage

BulkAnalytics UI

BulkAnalytics UI

Storage

StreamingAnalytics

ServiceBusAP

I

RESTSOAPHTTPKAFKA


BLEZigBeeWIFIWired

EdgeAnalytics

OracleBICS

OracleBigDataCS

OracleSOACS

OracleIntegrationCS

OracleIoT CS

OracleStreamingAnalyticsCS

OracleMessagingCS

OracleBigDataDiscoveryCS

OracleMobileCS


OracleIoT CSGateway

OracleIoT CSClientLibrary

OracleProcessCS

OracleDataFlowMLCS

BigDataPreparationCS

ApplicationContainerCS

ContainerCS

IoT Services

IoT Logical Reference Architecture – MicrosoftAzure

IoTDevice

Sensor

Actuator

IoT Gateway

Storage

UIApp

StreamingAnalytics



Analytics

Storage

EndpointManagement

EventHub

ServiceBus

EventHub Event

Hub

ServiceBus

BigData/BI

Storage

Services Processes

UIApp

Storage

BulkAnalytics UI

BulkAnalytics UI

Storage

StreamingAnalytics

ServiceBusAP

I

RESTSOAPHTTPKAFKA


BLEZigBeeWIFIWired

AzureIoTHu

b

AzureStreamingAnalytics

AzureServiceBu

s

AzurePowerBI

HDInsight

AzureStorage

AzureIoT GatewaySDK Az

ureEventH

ub

AzureStorage

AzureMachineLearning


IoT ServicesIoT

Device

Sensor

Actuator

IoT Gateway

Storage

UIApp

StreamingAnalytics



Analytics

Storage

EndpointManagement

EventHub

ServiceBus

EventHub Event

Hub

ServiceBus

BigData/BI

Storage

Services Processes

UIApp

Storage

BulkAnalytics UI

BulkAnalytics UI

Storage

StreamingAnalytics

ServiceBusAP

I

RESTSOAPHTTPKAFKA


BLEZigBeeWIFIWired

IoT Logical Architecture – Open Source


Summary


Summary

Treat events as events! Infrastructures for handling lots of events are available!

IoT tends to make Big Data / Fast Data infrastructures necessary

Know your use case/requirements to choose the right architecture!• Can my existing backend landscape handle the new IoT load?

• Do I have to handle huge amount of events in “real-time”?

• Do I need to filter/aggregate data before invoking existing backend systems?

• Do I want to do Advanced Analytics (predictive analytics) where historical information is necessary?

• What is the network bandwidth between device/gateway and cloud/backend?

• Centralized or Decentralized IoT solution?


J


Guido SchmutzTechnology Manager

[email protected]
