analyze, monitorand act on data monitorand act on data nayana singh program management, azure iot,...
TRANSCRIPT
Analyze, Monitor and Act on Data
Nayana SinghProgram Management, Azure IoT, January 2017
Microsoft Confidential
From endpoint to insight to action, across the enterprise, and around the world
Built on the industry’s leading cloud
Magic Quadrant Leader, Business Intelligence and Analytics Platforms
SecureEnd-to-end
From the endpoint, through the connection, to data, applications, and the cloud
OpenConnect anything
Any device, OS, data source, software, or service
ScalableGrow effortlessly
Millions of devices, terabytes of data, on-premises, in the cloud, in the most regions worldwide
FastStart in minutes
Preconfigured solutions for the most common IoT scenarios
Insights ActionThings
AZURE REGIONS
38Announced Azure regions world wide
Hyper-Scale Capacity3.5 Trillion Messages / Week
Hyper-Scale Azure Footprint
AZURE IOT REGIONS
12Azure IoT regions world wide
Elements of Azure IoT Suite
1.Connect and Manage
Devices & Gateways
Gateway & Devices
Preconfigured solutions
Connect and control
2. Analyze data & Generate
insights*
Real time analytics
Predictive analytics*
Data visualization
3. Integrate into business
systems
Workflow integration
Push and broadcast
notifications
ID and access
management
4. Secure IoT Infrastructure
5. Customize IoT Architecture
* Only applies to predictive maintenance
Elements of Azure IoT Suite
1.Connect and Manage
Devices & Gateways
Gateway & Devices
Preconfigured solutions
Connect and control
2. Analyze streaming data &
Generate predictive insights*
Real time analytics
Predictive analytics*
Data visualization
3. Integrate into business
systems
Workflow integration
Push and broadcast
notifications
ID and access
management
4. Secure IoT Infrastructure
5. Customize IoT Architecture
* Only applies to predictive maintenance
Apache Storm/Spark
Devices
RTO
S, L
inu
x, W
ind
ow
s, A
nd
roid
, iO
S
Gateway
On-Gateway
Analytics
On-Device App Analytics
In-Cloud Analytics
In-Cloud Hot-Analytics
In-Cloud Cold-Analytics
In-Cloud Analytics
Batch Analytics (Cold Path) INGEST PREPARE ANALYZE PUBLISH CONSUMEDATA SOURCES
Ex: Azure Data Lake, Azure SQL DB, Azure SQL DW, HDFS, Cassandra, HBase
Cloud Storage
Ex: Azure Data Factory, Oozie
Orchestration, Movement, Scheduling, Monitoring
Ex: Azure Data Lake Analytics, Hadoop, Azure Machine Learning, Mahout
Data Aggregation, Data Science, etc.
Ex: SQL Server, Oracle, DB2, MySQL
On-PremStorage
Ex: SQL Server, Oracle, DB2, MySQL
SaaS Data Sources Ex: Custom Code,
Azure Data Lake Analytics, Hadoop
Transform, Combine, Clean, etc.
Batch Data Movement
Ex: SSIS, Azure Data Factory, Sqoop, IoT Hub (via File Upload)
Presentation, Dashboarding
Ex: File Upload via IoTHub)
Near Real Time Analytics (Hot Path)INGEST PREPARE ANALYZE PUBLISH CONSUMEDATA SOURCES
Ex: Azure IoT Hubs, Kafka, Kinesis
Scalable device
data ingestion
Ex: Azure Stream Analytics, Storm, Spark
Stream Processing
Ex: Azure Data Lake, Azure SQL DB, Azure SQL DW, HDFS, Cassandra, HBase
Storage
Ex: Azure Data Factory, Azure Function, Oozie
Orchestration, Movement, Scheduling, Monitoring
Arc
hiv
al
Ex: Azure Machine Learning, Mahout
Machine Learning
Devices
Presentation, Dashboarding
Analytics with IoT Suite preconfigured solutions
Devices
Azure IoT Suite Remote Monitoring and Predictive Maintenance*
Back end
systems and
processes
Event Hub
Storage blobs DocumentDB
Web/
Mobile App
Stream
Analytics
Logic AppsIoT Hub Web JobsC# simulator
* Azure ML
Power BI
Real time analytics
Azure Stream Analytics
https://docs.microsoft.com/en-us/azure/stream-analytics/
Real-time Fraud Detection Streaming ETL Predictive Maintenance Call Center Analytics
IT Infrastructure and Network
Monitoring
Customer Behavior Prediction Log Analytics Real-time Cross Sell Offers
Fleet monitoring and Connected Cars Real-time Patient Monitoring Smart Grid Real-time Marketing
and many more…
Scenario Examples
Azure Stream Analytics
Rapid developmentUse simple SQL syntax to develop and deploy powerful analytics
Extensive out of the box integrationsReady integration with Blob Storage, ML, Power BI, Service Bus etc,
HyperscaleDistributed architecture can analyze millions of events a second
Uncover insights in real timePerform low latency analytics across multiple data streams
Enterprise GradeGuaranteed event delivery, auto recovery and 3x9s availability
Fully ManagedNo maintenance. No performance tuning. Zero upfront costs
Presentation & Action
Storage &Batch Analysis
StreamAnalytics
Event Queuing & StreamIngestion
Event production
IoT Hubs
Applications
Archiving for long term storage/ batch analytics
Real-time dashboard
Stream Analytics
Automation to kick-off workflows
Machine Learning
Reference Data
Event Hubs
Blobs
Devices &
Gateways
Streaming Pipeline
PowerBI
Write powerful analytics using SQL like language
Advanced job monitoringRich visual tooling to debug queries and
monitor jobs
Real time data enrichmentEasily join and score streaming data with
static reference data or function call-outs.
Out of order and late arrival policiesAccount for lag among gateways or
sensors through simple configurations
and settings
Predictive Analytics
Azure MachineLearning
https://docs.microsoft.com/en-us/azure/machine-learning/
Fully
managed
Integrated Collaborate Deploy in
minutes
No software to install,
no hardware to manage,
and one portal to view
and update.
Simple drag, drop and
connect interface for
Data Science. Tested
algorithms, integrated R
and Python support.
Share workspace to
collaborate across the
globe. View run history
and past results.
Operationalize models
in the cloud with a
single click. Monetize
in Machine Learning
Marketplace.
Steps to Build a ML Solution
Integrated predictive analyticsEmpower with proactive analysisMachine learning solutions enable powerful predictive analytics solutions, leveraging historical data and real time device ingestion input.
Predictive Maintenance WarningScheduled Maintenance Alert – Asset Sensors Indicate Critical Failure in (6)
Days.
Azure Machine Learning Ecosystem
Get/Prepare Data
Build/Edit Experiment
Create/Update Model
Evaluate Model Results
Publish Web
Service
Build ML Model Deploy as Web ServiceProvision Workspace
Get Azure
Subscription
Create
Workspace
Publish an App
Azure Data
Marketplace
https://Studio.azureml.nethttp://gallery.azureml.net
http://Azure.com/mlhttps://datamarket.azure.com/browse?query=machine+learning
AML - Drag & Drop + Best in Class Algorithms
Data Visualization
Power BI
https://docs.microsoft.com/en-us/azure/power-bi-embedded/
Power BI overview
Power BI REST APIPower BI Desktop
Prepare Explore ShareReport
SaaS solutionsE.g. Marketo, Salesforce, GitHub,
Google analytics
On-premises dataE.g. Analysis Services
Organizational content packsCorporate data sources, or external
data services
Azure servicesE.g. Azure SQL, Stream Analytics
Excel filesWorkbook data and data models
Power BI Desktop filesRelated data from files, databases,
Azure, and other sources
Data refresh
Visualizations
Live dashboards
Content packs Sharing & collaborationNatural language query
Reports
Datasets01001
10101
Manage in a familiar environment -Azure
Manually or programmatically provision workspaces
Create datasets and reports
Embed reports in your app
Control refresh behavior and credentials
Power BI report
Azure SQL
Data Warehouse
Azure SQL
Database
Your app
Power BI
Workspace
Power BI Desktop
Azure Portal
Power BI Embedded
Power BI for developers*
EmbedPower BI experiences
directly into your public
facing websites and blogs
ExtendPower BI and your reach
with organizational content
packs and custom visuals
Integrateuser-defined Power BI
experiences into your app
</>
*This is for Power BI embedded. More customizations are available with full Power BI
Web Job
Event
Hub
Document
DB
(Device
Registry)
IoT
Hub
Consumer
Group
Azure
Stream
Analytics
Job 1
Device
Info
Job 2
Telemetry
Azure
Storage
(Blob)
Telemetry
History
RUL
Output
Input
Dataset
Web Job
Event
Processor
Host
Web App
Dashboard
Device
Portal
Event
Hub
Simulated
Device
Azure ML
Trained
Model
Training
Data
Demo Summary
Azure IoT Hub sends data to Azure Stream Analytics
Edits jobs in ASA (Streaming versus Reference Data)
Using Azure ML to calculate the Remaining Useful life of the engine
Sinking data from ASA to AML
Sinking streaming dataset from ASA to Power BI
Visualizing data in PowerBI
Elements of Azure IoT Suite
1.Connect and Manage
Devices & Gateways
Gateway & Devices
Preconfigured solutions
Connect and control
2. Analyze streaming data &
Generate predictive insights*
Real time analytics
Predictive analytics*
Data visualization
3. Integrate into business
systems
Workflow integration
Push and broadcast
notifications
ID and access
management
4. Secure IoT Infrastructure
5. Customize IoT Architecture
* Only applies to predictive maintenance
Get started today
Explore IoT Documentation tab on Azure.com
Connect with your regional IoT team
View Preconfigured Solution Demo
Select a partner
Get Started Now
Go to www.InternetOfYourThings.com
Send Feedback
