![Page 1: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/1.jpg)
Leveraging the Cloud to Architect Digital Solutions
![Page 2: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/2.jpg)
Agenda
• State of the art technology for IoT
• Table Storage Demo
• ML Clustering and Classification
prototype
• Cortana Analytics
• Architecture for building today
• Patterns and anti-patterns
• Open Discussion
2
![Page 3: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/3.jpg)
What is Digital?
Digital is more than technology.
It involves connecting that technology with the right data science, devices, design and business strategy.
It is putting a customer, device, organization or business process at the center of real change in how businesses do things and how customers experience them. In how we engage, invent, build and buy everything.
It means creating value by uniting the physical world—seamlessly, efficiently, meaningfully—to the virtual one we’re building.
3
![Page 4: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/4.jpg)
Microsoft Azure IoT services
Devices Device Connectivity Storage Analytics Presentation & Action
Event Hubs SQL DatabaseMachine
LearningApp Service
Service BusTable/Blob
Storage
Stream
AnalyticsPower BI
External Data
SourcesDocumentDB HDInsight
Notification
Hubs
External Data
SourcesData Factory
Mobile
Services
BizTalk
Services
{ }
4
![Page 5: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/5.jpg)
>90,000New Azure customer subscriptions/month
1.5Trillion
Messages per month processed by Azure IoT
>500Million
Users in Azure Active Directory
777Trillion
Storage Transactionsper day
>1.5Million
SQL Databases running on Azure
>40%Revenue from
Start-ups and ISVs
Cloud Momentum
5
![Page 6: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/6.jpg)
Device Connectivity and Security
Data Ingestion and Command & Control
Stream Processing & Predictive Analytics
Workflow Automation
Dashboards and Visualization
Preconfigured Solutions
Azure IoT Suite
Managing Big Data
6
![Page 7: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/7.jpg)
Is IoT even a new thing?
•
Depending on who you ask, IoT is either:
Nothing new
A unicorn
Magic, and will
soon change
everything.
We’ve been
doing this
for 40 years
7
![Page 8: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/8.jpg)
Emerging Challenges for IT
• Scale• # devices >> # users, and growing fast
• Volume of data (and network traffic)
• Pace• Innovation pressure: analysis, command and
control, cost
• Skill pressure: data science, new platforms
• Environment• IT/OT collaboration
• Security and privacy threats
• Emerging standards
• New competitors
8
![Page 9: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/9.jpg)
Azure Machine Learning Conceptual Model
9
![Page 10: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/10.jpg)
• Data scientist– A highly educated and skilled person who can solve complex data problems by employing
deep expertise in scientific disciplines (mathematics, statistics or computer science)
• Data professional– A skilled person who creates or maintains data systems, data solutions, or implements
predictive modelling
Roles: Database Administrator, Database Developer, or BI Developer
• Software developer– A skilled person who designs and develops programming logic, and can apply machine
learning to integrate predictive functionality into applications
Machine learning roles
10
![Page 11: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/11.jpg)
Pattern: Think big. Start small
• Build to an architecture that will scale, but start
prototyping with a small number of devices.
• It’s hard to predict what data provides value --
which impacts which sensors and devices are
necessary -- until you build something.
• The options can be overwhelming: set crisp goals
up front and use those to define and refine.
• It’s much easier to work through device identity,
management/update and security at small scale.
.
11
![Page 12: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/12.jpg)
Pattern: Telemetry first
• It is very hard to predict in advance what data
will be useful.
• It is tempting, but likely inefficient to try for
business transformation in the first step.
• Think about not only device telemetry but
also diagnostic telemetry.
• Privacy and security implications of telemetry
are generally lesser than for command and
control.
12
![Page 13: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/13.jpg)
Telemetry today
• High scale data ingestion
stream processing
• Storage for cold-path
analytics
• Processing for hot-path
analytics
13
![Page 14: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/14.jpg)
Azure Table Storage
14
![Page 15: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/15.jpg)
Storage
15
![Page 16: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/16.jpg)
Traditional RDBMS vs. Table Storage
Value Metric
1000 kB kilobyte
10002 MB megabyte
10003 GB gigabyte
10004 TB terabyte
10005 PB petabyte
10006 EB hexabyte
10007 ZB zettabyte
10008 YB yottabyte
16
![Page 17: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/17.jpg)
Entity Properties
– Up to 1MB per entity
– PartitionKey & RowKey (only
indexed properties)– Uniquely identifies an entity
– Defines the sort order
– Timestamp – Optimistic Concurrency
– Exposed as an HTTP Etag
– Each property is stored as a <name, typed value> pair
– No schema stored for a table
– Properties can be the standard .NET types
– String, binary, bool, DateTime, GUID, int, int64, and double
17
![Page 18: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/18.jpg)
Purpose of the PartitionKey
– Entities in the same partition will be stored together– Efficient querying and cache locality
– Endeavour to include partition key in all queries
– Atomic multiple Insert/Update/Delete in same partition in a single transaction
– Target throughput – 500 tps/partition, several thousand tps/account
– Windows Azure monitors the usage patterns of partitions
– Automatically load balance partitions– Each partition can be served by a different storage node
– Scale to meet the traffic needs of your table
18
![Page 19: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/19.jpg)
Azure Machine Learning Conceptual Model
19
![Page 20: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/20.jpg)
Azure IoT Reference Architecture
Solution PortalProvisioning API
Identity & Registry Stores
Stream Event Processor
Analytics/
Machine
Learning
Data
Visualization &
Presentation
Device State Store
GatewayStorage
IP capable
devices
Existing IoT
devices
Low power
devices
Presentation Device and Event ProcessingData Transport
Devices and
Data Sources
Cloud
Gate-
way
Agent
Libs
Agent
Libs
Control System Worker Role
Agent
Libs
20
![Page 21: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/21.jpg)
Demo: ML Clustering and Classification of Data
![Page 22: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/22.jpg)
Table Storage Sharding
22
![Page 23: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/23.jpg)
23
![Page 24: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/24.jpg)
Azure Machine Learning Conceptual Model
24
![Page 25: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/25.jpg)
Solution PortalProvisioning API
Identity & Registry Stores
Stream Event Processor
Analytics/
Machine
Learning
Data Visualization
& Presentation
Device State Store
GatewayStorage
IP capable
devices
Existing IoT
devices
Low power
devices
Presentation Device and Event ProcessingData Transport
Devices and
Data Sources
Cloud
Gate-
way
Agent
Libs
Agent
Libs
Control System Worker Role
Agent
Libs
![Page 26: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/26.jpg)
![Page 27: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/27.jpg)
Data, Information, Knowledge, Wisdom Hierarchy
27
![Page 28: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/28.jpg)
![Page 29: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/29.jpg)
Cortana Analytics SuiteTransform data into intelligent action
29
![Page 30: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/30.jpg)
Azure Machine Learning Conceptual Model
30
![Page 31: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/31.jpg)
Azure IoT Reference Architecture
Solution PortalProvisioning API
Identity & Registry Stores
Stream Event Processor
Analytics/
Machine
Learning
Data
Visualization &
Presentation
Device State Store
GatewayStorage
IP capable
devices
Existing IoT
devices
Low power
devices
Presentation Device and Event ProcessingData Transport
Devices and
Data Sources
Cloud
Gate-
way
Agent
Libs
Agent
Libs
Control System Worker Role
Agent
Libs
31
![Page 32: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/32.jpg)
Visualizing ML Results in Power BI
32
![Page 33: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/33.jpg)
The need to know what could be…
33
![Page 34: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/34.jpg)
Deriving Business Value from Big Data
34
![Page 35: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/35.jpg)
Vision Analytics
Using past data to predict the future
Recommenda-
tion engines
Advertising
analysis
Weather
forecasting for
business
planning
Social network
analysis
Legal
discovery and
document
archiving
Pricing analysis
Fraud
detection
Churn
analysis
Equipment
monitoring
Location-based
tracking and
services
Personalized
Insurance
Machine learning & predictive analytics are core capabilities that can help create added business value
35
![Page 36: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/36.jpg)
Summary• Think big (architecture), but start
small (experiment, learn and
refine).
• Start with telemetry. Address
privacy, security and
manageability before moving to
command and control.
• Don’t interrupt the fast path and
create processing bottlenecks.
• Build to the reference architecture
to ease the move to IoT Suite.
36
![Page 37: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/37.jpg)
Future Topics
• Cortana Analytics Suite
• Azure Portal
• Scalable Cloud Applications
• PaaS development
• Event Hub
• Stream Analytics
• Machine Learning Studio
37
![Page 38: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/38.jpg)
38
![Page 39: Leveraging the Cloud to Architect Digital Solutions · Cortana Analytics Suite Transform data into intelligent action 29 Azure Machine Learning Conceptual Model 30 Azure IoT Reference](https://reader034.vdocuments.us/reader034/viewer/2022050215/5f610aacedfe5933474799d6/html5/thumbnails/39.jpg)
Q&A