30052018 - iot for the bi professional - intelligentcloud.dk · iot ”the internet of things (iot)...

Expo Sponsors

Event Sponsors

Expo Light Sponsors

IoT for the BI professional

David L. Bojsen - Principal Architect

What to expect

∠ Level 200 session∠ Which basically means PowerPoint and talking

∠ Enthusiastic approach to (I)IoT∠ Want to spread the IoT-flu

What is IoT

IoT”The Internet of Things (IoT) is the network of physical devices,

vehicles, home appliances and other items embedded withelectronics, software, sensors, actuators, and connectivity

which enables these objects to connect and exchange

Data”

Source: en.wikipedia.org/wiki/Internet_of_things

What is IIoT

IIoT”The term Industrial Internet of things (IIoT) is often

encountered in the manufacturing industries, referring to the industrial subset of the IoT”

Source: en.wikipedia.org/wiki/Internet_of_things

Internet of Things opportunity

Examples of IIoT scenarios

Auto

What does that mean to us?

Auto

But not your typical datasource∠ IoT sensor data typically consists of time base measurements∠ Storing, indexing, querying, analyzing, visualizing can be challenging∠ Especially considering “IoT scale”

Devices

Storage

IoT Architecture

IoT Solutions Have a Common Pattern

ActionsThings Insights

A more realistic view…

ActionsThings Insights

Field Gateway

Device Connectivity & Management

Analytics & Operationalized Insights

Presentation & Business Connectivity

Dev

ices

RTOS

, Lin

ux, W

indo

ws,

Andr

oid,

iOS

Protocol Adaptation

Cold Path AnalyticsAzure HDInsight, AzureML

Azure Data Lake, Data Lake Analytics

Hot Path Business LogicService Fabric

Azure Functions

Field Gateway

Protocol Adaptation

Cloud GatewayIoT Hub

Hot Path AnalyticsAzure Stream Analytics

Azure HDInsightAzure Time Series Insights

Power BI

App Service

Logic Apps

Web Apps

Mobile Apps

Notification Hubs

BizTalk Services

• Bi-directional device-cloud • Telemetry

ingestion

• Command & Control• Device Registry

& Identity• Device Mgmt• HTTP, AMQP,

MQTT

High level IoT Architecture

Field Gateway




Dev

ices

RTOS

, Lin

ux, W

indo

ws,

Andr

oid,

iOS

Protocol Adaptation




Azure Functions

Field Gateway

Protocol Adaptation




Power BI

App Service

Logic Apps

Web Apps

Mobile Apps

Notification Hubs

BizTalk Services


ingestion



MQTT

Lets focus on the stuff we already know

Hot Path Analysis

Hot path analysis

∠ Azure StreamAnalytics

Query Language Declarative queries in a SQL like language

Easy to author and deploy

Temporal semanticsAll operators respect, and some use, the temporal properties of events

Built-in operators and functionsFilters, projections, joins, windowed (temporal) aggregates, text and date manipulation

∠ SELECT

∠ FROM

∠ WHERE

∠ GROUP BY

∠ HAVING

∠ CASE WHEN THEN ELSE

∠ INNER/LEFT OUTER JOIN

∠ UNION

∠ CROSS/OUTER APPLY

∠ CAST

∠ INTO

∠ ORDER BY ASC, DSC

SAQL – Language & Library

• WITH• PARTITION B• OVER

• DateName, DatePart, Day• Month, Year, DateDiff• DateTimeFromParts, DateAdd

• TumblingWindow• HoppingWindow• SlidingWindow

• SUM• COUNT• AVG• MIN• MAX• STDEV• STDEVP• VAR• VARP• TopOne

• Len, Concat, CharIndex• Substring, Lower• Upper, PatIndex

• Lag IsFirst,Last• CollectTop

• ABS, CEILING, EXP, FLOOR• POWER, SIGN, SQUARE• SQRT

SELECT TimeZone, COUNT(*) AS Count FROM TwitterStream TIMESTAMP BY CreatedAtGROUP BY TimeZone, TumblingWindow(second,10)

Tell me the count of tweets per time zone every 10 seconds

1 5 4 26 8 6 5

Time(secs)

1 5 4 26

8 6

A 10-second Tumbling Window

3 6 1

5 3 6 1

SELECT Topic, COUNT(*) AS TotalTweetsFROM TwitterStream TIMESTAMP BY CreatedAtGROUP BY Topic, HoppingWindow(second, 10 , 5)

Every 5 seconds give me the count of tweets over the last 10 seconds

1 5 4 26 8 7

A 10-second Hopping Window with a 5-second “Hop”

4 26

8 6

5 3 6 1

1 5 4 26

8 6 5 3

6 15 3

SELECT Topic, COUNT(*) FROM TwitterStreamTIMESTAMP BY CreatedAtGROUP BY Topic, SlidingWindow(second, 10)HAVING COUNT(*) > 10

Give me the count of tweets for all topics which are tweeted more than 10 times in the last 10 seconds

1 5

A 10-second Sliding Window

8

8

51

9

51 9

1

Hot path analysis

∠ Azure Stream Analytics∠ Azure Time Series Insight

Azure Time Series Insights

∠ IoT scale time-series data store

∠ Schema-less store, just send data

∠ Easy IoT Hub connection∠ Store, query and visualize

billions of events

∠ Simple and fast navigation

Hot path analysis

∠ Azure Stream Analytics∠ Azure Time Series Insight∠ Azure HDInsight

Big Data in Azure

Data Lake StoreHyper-scale Storage optimized for analytics

IaaS Hadoop

Hadoop distros on Azure VMs

Blob Storage

HDInsight

Azure-managed Hadoop clusters

HadoopSparkHBaseStorm

Data LakeAnalytics

Big Data as a Service

Cold Path Analysis

Cold path analysis

∠ Azure Data Lake Store

Petabyte size files and Trillions of objects

Scalable throughput for massively parallel analytics

HDFS for the Cloud

Always encrypted, Role-based Security & Auditing

Enterprise-grade Support

Azure Data Lake StoreA No limits Data Lake that powers Big Data Analytics

The first cloud Data Lake for enterprises that is secure, massively scalable and built to the open HDFS standard. With no limits to the size of data and the ability to run massively parallel analytics, you can now unlock value from all your unstructured, semi-structured and structured data.

Cold path analysis

∠ Azure Data Lake Store∠ Azure Data Lake Analytics

Start in seconds, Scale instantly, Pay per job

Develop massively parallel programs with simplicity

Debug and Optimize your Big Data programs with ease

Virtualize your analytics

Enterprise-grade Support and Security

Azure Data Lake AnalyticsMassively parallel, extensible, analytics made simpleThe first cloud analytics service where you can easily develop and run massively parallel data transformation and processing programs in U-SQL, R, Python and .Net over petabytes of data. With no infrastructure to manage, process data on demand, scale instantly, and only pay per job.

DECLARE @endDate DateTime = DateTime.Now;DECLARE @startDate DateTime = @endDate.AddDays(-7);

@orders = EXTRACT

OrderId int, Customer string, Date DateTime, Amount float

FROM "/input/orders.txt"USING Extractors.Tsv();

@orders = SELECT * FROM ordersWHERE Date >= startDate AND Date <= endDate;

@orders = SELECT * FROM ordersWHERE Customer.Contains(“Contoso”);

OUTPUT @ordersTO "/output/output.txt"USING Outputters.Tsv();

U-SQL Example

DECLARE constant values

C# Expressions

RowSets enable dataflow programming

EXTRACT performs schema on read for files

OUTPUT for writing files

Built-in handling for CSV & TSV

Whole-script optimization

REFERENCE ASSEMBLY [ExtPython];

DECLARE @myScript = @"def get_mentions(tweet):

return ';'.join( ( w[1:] for w in tweet.split() if w[0]=='@' ) )

def usqlml_main(df):del df['time']del df['author']df['mentions'] = df.tweet.apply(get_mentions)del df['tweet']return df

";

@t = SELECT * FROM

(VALUES("D1","T1","A1","@foo Hello World @bar"),("D2","T2","A2","@baz Hello World @beer")

) AS D( date, time, author, tweet );

@m =REDUCE @t ON datePRODUCE date string, mentions stringUSING new Extension.Python.Reducer(pyScript:@myScript);

Python Extensions

Use U-SQL to create a massively distributed program.

Executing Python code across many nodes.

Using standard libraries such as numpy and pandas.

Cold path analysis

∠ Azure Data Lake Store∠ Azure Data Lake Analytics∠ Azure Machine Learning

NotebooksIDEs

Azure M achine Learning W orkbench

VS Code Tools for AI

Experim entation and M odel M anagem ent

Services

AZURE MACHINE LEARNING SERVICES

SparkSQL Server Virtual m achines GPUsContainer services

SQL ServerM achine Learning Server

O N - P R E M I S E S

E D G EAzure IoT Edge

TRAIN & DEPLOY OPTIONS

A Z U R E

Azure Machine Learning

Cold path analysis

∠ Azure Data Lake Store∠ Azure Data Lake Analytics∠ Azure Machine Learning∠ And good old SQL

SQL ServerThe platform of choice

Linux

101001010010

{ } T-SQL

Java

C/C++

C#/VB.NET

PHP

Node.js

Python

Ruby

Any data Any application

Anywhere Choice of platform

Field Gateway




Dev

ices

RTOS

, Lin

ux, W

indo

ws,

Andr

oid,

iOS

Protocol Adaptation




Azure Functions

Field Gateway

Protocol Adaptation




Power BI

App Service

Logic Apps

Web Apps

Mobile Apps

Notification Hubs

BizTalk Services


ingestion



MQTT


Field

Gateway

Device

Connectivity & Management

Analytics &

Operationalized Insights

Presentation &

Business Connectivity

De

vic

es

RTO

S, Li

nu

x, W

ind

ow

s, A

nd

roid

, iO

S

Protocol

Adaptation




Azure Functions

Field

Gateway

Protocol

Adaptation



Azure HDInsight

Azure Time Series Insights

Power BI

App

Service

Logic

Apps

Web

Apps Mobile

Apps

Notification

Hubs

BizTalk

Services

• Bi-directional

device-cloud

• Telemetry ingestion

• Command & Control

• Device Registry

& Identity

• Device Mgmt

• HTTP, AMQP,

MQTT

But what good is it without data from devices…

Connect & ControlAzure IoT Hub

Azure IoT Hub Device Management

IoT Hub

Device twin Device twin

Software

FirmwareConfiguration

Azure IoT Hub Device Management

IoT Hub

Device twin Device twin

DesiredReported

TagsMethods

Desired

Reported

Methods

JobsSchedule and broadcast Device twin changes across large fleets

Azure IoT Hub Message RoutesDe

vices

IoT Hub

IoT Solution

Routing Logic

Event Processing

Queue

StorageQueue

AlarmQueue

Remember email before inbox rules?

Device Connectivity

Field G atew ay

CoAP, A llJoyn, O PC

Custom Protocol G atew ay

(C loud Service, VM )VPN /ExpressRoute

O PC, H TTP, CoAP

Fie ld G atew ay*

CoAP, A llJoyn, O PC

IoT H ub

Custom Protocol G atew ay

(C loud Service, VM )

AM Q P, M Q TT, H TTPS

Custom Protocols

Data Path

Optional solution component

Azure IoT solution component

DeviceIoT C lient

DeviceIoT C lient

Device

IoT C lientDevice

Device

Device

AM Q P, M Q TT, H TTPS

* Field Gateway is currently a concept that may represent Microsoft, 3rd party or custom capabilities at a hardware or software level

Industrial protocols

∠ Ethernet/IP∠ Modbus∠ OPC UA∠ OPC DA

∠ Profibus∠ BACNet∠ KNX

∠ Zigbee∠ ProfiNetAnd many, many more

Source: en.wikipedia.org/wiki/List_of_automation_protocols

Things

IoT Pattern

Insights ActionsCloud Gateway

Things

IoT Pattern + Edge

Insights ActionsInsights ActionsCloud Gateway

Azure Machine LearningS p a r k

S Q L S e r v e r

V ir t u a l m a c h in e s G P U s

C o n t a in e r s e r v ic e s

N o t e b o o k sID E s

A z u r e M a c h in e L e a r n in g W o r k b e n c h

S Q L S e r v e r

M a c h in e L e a r n in g S e r v e r

ON-PREMI SES

EDGE COMPUTI NGA z u r e Io T E d g e

E x p e r im e n t a t io n a n d M o d e l M a n a g e m e n t

A Z U R E MA C H I N E L E A R N I N G S E R V I C E S

T R A I N & D E P L O Y O P T I O N S

AZURE

Field Gateway




Dev

ices

RTOS

, Lin

ux, W

indo

ws,

Andr

oid,

iOS

Protocol Adaptation




Azure Functions

Field Gateway

Protocol Adaptation




Power BI

App Service

Logic Apps

Web Apps

Mobile Apps

Notification Hubs

BizTalk Services


ingestion



MQTT


Field Gateway




Devi

ces

RTOS

, Linu

x, W

indow

s, An

droi

d, iO

S

Protocol Adaptation




Azure Functions

Field Gateway

Protocol Adaptation




Power BI

App Service

Logic Apps

Web Apps

Mobile Apps

Notification Hubs

BizTalk Services

• Bi-directional device-cloud • Telemetry ingestion



MQTT

Who (or what) are we building this for…

Questions ?

AZURE REGIONS

50Announced Azure regions world wideHyper-Scale Capacity

Get started building IoT solutions today

AZURE IOT SUITE REGIONS

14Azure IoT Suite regions

Thank youfor your

time

30052018 - iot for the bi professional - intelligentcloud.dk · iot ”the internet of things (iot)...

Documents