iot arhitecture, database and big data - aceiot · gateway –centric iot cloud (1/2) iot...

IoT Arhitecture, database and big

data

23th- 27th April,

2018

Rwanda

Abdur Rahim

Open IoT

FBK CREATE-NET

IoT Architecture and platform

Platform overview 3

Two main platforms

IoT Gateway platform (Gateway-Centric

Cloud-Bring Cloud functionalities in Gateway)

IoT integrated Cloud/PaaS platform (Cloud-

centric IoT- Bring IoT functionalities in Cloud)

Gateway – Centric IoT Cloud (1/2)

IoT infrastructure will provide the opportunities to take services,

workloads, applications and large amounts of data and deliver

it all to the edge of the network.

Processing and storage of data close to users/near to sources

To distribute data to move it closer to the end-users to eliminate latency,

numerous hop, and support mobile computing and data streaming

Creating dense geographical distribution

Gateway- Centric IoT Cloud (2/2)

This approach are useful when service is provisioned from the

data coming from same location

Highly real time applications

Reduce network traffic and cost

Supporting end-users security

Data process and service execute locally (distributed cloud

processing, sub-work flow, data aggregation locally)

What is gateway platform?7

IoT Gateway centric architecture 8

Driving the functionalities to the edge

Intelligent gateway9

Data acquisition, integration and rules activation,

providing dynamic intelligence at the edge

Industrial IoT gateway10

Intel IoT Gateway11

Gateway comparison summary 12

https://www.loriot.io/gateways.html

Kura architecture 13

http://eclipse.githu

b.io/kura/doc/intr

o.html

Open source gateway14

Kura and camel integration

https://camel.apache.org/kura.html

Agile micorservice Gateway15

Cloud-centric IoT PaaS platform

Bring IoT data in the cloud

Processing and computing the data and deploy

management tools in cloud

This approach this good if service are provided

among objects located in multiple location

Giving several example (market platform, project

platform..)

Example of IoT platform

Bosch IoT Cloud18

https://www.bosch-si.com/products/

Most known IoT cloud platform19

Architecture

Architecture: functional domains and

components

Functional domain Component Technology choice

Application platform Orchestrator Deis

Execution environments Docker

RAD Node-RED

UI manager Elastic search, Kibana (Freeboard, D3.js)

IoT platform IoT bridge Specific protocols/libraries

Pre-Process FI-WARE CEPHEUS

Sensor registry Not selected yet

Sensor discovery Not selected yet

Big data platform Data streaming Orion

Storage manager MongoDB

Data analytic FlinkML, Mlib

Security & privacy Identity manager OpenAM/Gluu

Authorization manager OpenAM/Gluu

Privacy manager OpenAM/Gluu

Big data platform

orchestrator

App

compileApp

Read

manifest

Request

sensorStart

app

deployIoT PF

Manifest

reader

App Data

stream

Status

manager

Web

GUI

WorkersBig data

runtime

Stream

broker

Analytic

IoT platform

Sensor

finder

Sensor

registry

IoT

brokerIoT

bridge

Subscribe

Raw

data

data

Conn. req.Big

data PF

stream

sensor.

req.

Status

manager

Web

GUI

Pre-

process

PaaS platform

Developer

App +

manifest

WAZIUP Cloud platform

orchestrator

Cloud execution environment

Se

rvic

e

Se

rvic

e

Se

rvic

eApp user

Service

Service

Service

create push

deploy

deploy

Local deployment

App

Gateway Sensor

Local

PC

uses

With/Without internet

24

App deploy: local and global

Developer

App +

manifes

t

WAZIUP Cloud platform

orchestrator

Cloud execution environment

Se

rvic

e

Se

rvic

e

Se

rvic

eApp user

Service

Service

Service

create push

deploy

deploy

Local deployment

Ap

p

Gateway Sensor

Local

PC

uses

Data flows 1

WAZIUP Cloud platform

Cloud execution environment

Se

rvic

e

Se

rvic

e

Se

rvic

eApp user

access

Gateway

Data flow 2

WAZIUP Cloud platform

orchestrator

Cloud execution environment

Se

rvic

e

Se

rvic

e

Se

rvic

eApp user

Service

Service

Service

access

deploy

deploy

Data flows 2

WAZIUP Cloud platform

Cloud execution environment

Se

rvic

e

Se

rvic

e

Se

rvic

eApp user

Service

Service

access

Gateway

Databases requirements

• Run in gateway (low resources)

• Run in Cloud

• Run in local PC

• Synchronize in best-effort

• Support queries from local app

Big data technologies

• Data processing:

• Spark

• Flink

• Data broker:

• Kafka

• Orion

• Data Mining:

• Spark Mlib

• H20.ai

• Data Storage

• MongoDB

• Apache HBase

Databases

Two basic types

NoSQL

SQL

31

Data model

Key/Value

Memcached, Dynamo

Tabular

Big Table

Document Orieneted

MongoDB, CouchdB, JSON stores

32

NoSQL

A form of database management system

that is non-relational

System are often schema less, avoid joins

& are easy to scale

33

But why Choose NoSQL

Amount of data stored is on the

up and up

The data we store is more

complete than before

All the data is need to be easy to

be able to add/remove servers

without any disruption of services

34

MongoDB

Document base

Schema-less

Highly scalable

Easy replication & sharing

35

Basic of Big data and database

BIG data is always not well understood

Smart Data

Big data: 3V’s (volume)

Volume

Large data size• What does mean size?

– Not gigabytes

– Most likely not a few terabytes

– Possibly not 10’s of terabytes

– Probably 100’s of terabytes

– Definitely petabytes

Big Data: 3V’s (velocity)

Velocity

real-time

near-real time

streaming data flow

Big Data: 3V’s (variety)

Variety diverse data (structure and

unstructured, diverse data models and

query languages, diverse data sources

Some make it 4V’s

IoT in BIG data

IoT presents challenges in combination of all BIG data characteristics (3Vs/4Vs)

Most challenging IoT applications impact both Velocity & Volume and sometimes also Variety (situation and context)

Today..

GE each day gathers 50 million pieces of data from 10 million sensors, off equipment worth $1 trillion

A wearable sensor produces about 55 million data points pro day (challenge for storage), whereas some medical wearable's (like ECG) produce up to 1000 events per second (challenge for real-time processing)

IoT real-time Big data

Real-time Big Data (Fast data)

Real-time Data coming from (mainly)

Monitoring systems (e.g. Nagios)

Sensors

Stationary sensors (environment)

Embedded (e.g. in mobile devices)

Wearable sensors (e.g. HR monitors)

extreme velocity

mobile data streams

extreme resource

constraints

EMERGING

GREAT BUSINESS OPPORTUNITIES

e.g. there should be about 250 Million Wearable Health & Fitness Sensing Devices by 2017.

The market for sports and fitness apps will cross $400 million in 2016

BIG Data is nothing without BIG

business value insight

IoT is a huge opportunity for BIG data

..and opposite

IoT without BiG DATA is first generation IoT 1.0

IoT with BIG DATA is the third generation IoT 3.0 (vision of future world)

=+

IoT BIG Data BIG Value

© 2016 Abdur Rahim Biswas

IoT and Big Data

Content

Big data introduction

IoT big data requirements and platform

IoT big data technologies and tools

IoT BIG data applications

Deep understanding (observe of behavior of many thing””, gain important insight

Health example (understanding the cause of diseases/comorbidities/indicators)

Real-time actionable insight (Real-time analytic, detect and react in real-time)

Health example (real-time fall detection and potential reaction for aging population)

Performance optimization (configuration, energy, health-care)

Health example (Improve overall healthcare efficiency)

Proactive and predictive functional applications

Health example (proactive and prediction identification of diagnostic in healthcare applications (before thing occur)

Deep understanding applications

challenges

This vision boils down to solve multiple challenges:

to store all the events (Velocity & Volume Challenge);

to run analytical queries over the stored events; (Velocity & Volume Challenge)

to perform analytics (data mining and machine learning) over the data to gain insights (Velocity & Volume & Variety Challenge);

Real-time actionable Insights

Real-time detection and action represent multiple challenges

How to make reliable knowledge and decision from BIG Data? (Veracity and Verity)

How to process (real-time process and data interpretation) the streaming/real-time events on the fly (Velocity challenges)

How to store the events in the operational database (Velocity challenges)

How to correlate streaming events with store data in the operational database (velocity and Volume challenges)

Performance optimization challenges

When the “old way” of processing data just doesn’t work effectively

How we store the diverse set of BIG data, from mobile/sensors/server? (much data

How we move that much data

How we extract, load & transform that much data

How we explore and analyze that much data

How we process and get meaningful insights from that much data

Prediction challenges

Context

Long-period of time

IoT Big data platform requirements

CognitiveScalable

Real-time Unified view

Data

sourceData

source

Architecture of IoT Big data platform

Major big data analytic platform

IoT Big data framework- INTEL

Traditional methods Big data

Centralize Distributed

More power More machines

Summarize data Keep all data

Transform and store Transform on demand

Pre-define schema Flexible/no-schema

Move data toward compute Move compute towards data

Less data/more complex algorithms More data/simple algorithms

Philosophical differences of Big data analytic

© 2016 Abdur Rahim Biswas

IoT and Big Data

Content

Big data introduction

IoT big data requirements and platform

IoT big data technologies and tools

BIG data tools

Hadoop ecosystem- well known tools

Hadoop: The disruptive technology at

the core of Big data

Batch and stream processing

Hadoop MapReduce Batch Processing

Storm Streaming Stream Processing

Stream Processing

Handles data at high velocity

If Hadoop is the ocean, streams are the firehose

Processing in near real-time

Storm

Storm

Storm is a distributed data processing system whose processing is based on elements called spouts and bolts

The topology consists out of spouts which are message originators for the rest of the topology

The processing elements in the topology are called bolts which can be interconnected with an internal pub/sub mechanism

Bolts can also deliver their results to other systems such as DBMS’s, legacy systems or applications.

Lambda architecture combine

Complex Architectures Using Many Big

Data Technologies

Contact

Abdur Rahim

Project Coordinator

Create-Net, Italy

Email: arahim@fbk.eu

68