advanced analytics & iot - neudesic...neudesic partnered with one of the nation’s largest...
TRANSCRIPT
![Page 1: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/1.jpg)
Presented by:
Orion GebremedhinDirector of Technology, Data & Analytics
Marc LobreeNational Architect, Advanced Analytics
ADVANCED
ANALYTICS & IOTARCHITECTURES
![Page 2: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/2.jpg)
THE
RIGHT TOOL FORTHE RIGHT WORKLOAD
Azure SQL DW
Storage blob
HDInsight
SSIS
RDBMS Data Stores
Unstructured data
Excel (Direct Access)
SSIS
EDW
Flat File Upload
Azure SQL database
Local Data Sources
On-PremisesReporting & Analytics
![Page 3: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/3.jpg)
REFERENCE ARCHITECTURE
HYBRID BIG DATA PROCESSING
Cube
SSIS
EDW
Tabular
ESA
Direct Access/ Report Model Level Integration
Data Layer Level Integration
On-Demand-Compute
Cloud Storage
AZCopyPowerShellSSIS
![Page 4: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/4.jpg)
ON PREMISES
BIG DATA IMPLEMENTATIONS
![Page 5: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/5.jpg)
USE CASE:
ETLOFFLOADING
• Have you outgrown your data delivery SLAs?
• Is your business frustrated with data delays?
• Get the right data at the right time.
![Page 6: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/6.jpg)
Neudesic partnered with one of the nation’s largest utility companies that recently
deployed Smar Utility Meters for power customers, nearly a million meters sending usage
data every 15 minutes.
The result: an Azure hybrid big data processing solution that enabled the customer to
perform gap analytics: a process for identifying gaps that exist in the power usage
readings, over 7x faster than their previous solution! Billions of Smart Meter reads get
processed to identify the nature and duration of the gaps to mitigate revenue losses.
![Page 7: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/7.jpg)
USE CASE:
REAL-TIMEANALYSIS
• Got end users that need data now?
• Provide business units the data they need at
the time they need it.
![Page 8: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/8.jpg)
REAL TIME
TRAFFIC MANAGEMENT
EventsHub
StreamAnalytics
Reference DataVehicle Registration
Toll Data
Toll Way – Event Generator
Toll Violations
Toll Violation Tickets
![Page 9: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/9.jpg)
Real-Time Analysis
On-premisesUsing Data Lake to capture all data for everyone.
HDFS
Kafka Logs
OLTP
PM B DM MDM
Spark MLlib
Machine learning
Kafka
![Page 10: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/10.jpg)
USE CASE:
INTERNETOF THINGS
• What action does your IoT device drive?
• Help guide end-users to the action they are
looking to take.
![Page 11: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/11.jpg)
VENDING
MACHINEMANAGEMENT
EventsHub
StreamAnalytics
Vending Transactions
EventsHub
Machine learning
Batch Predictions
Real-time Notifications
EventsHub
Vending Machine
Vending Machine
Vehicle Location Info
![Page 12: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/12.jpg)
REAL TIME
TRAFFICMANAGEMENT
EventsHub
StreamAnalytics
Reference DataVehicle Registration
Toll Data
Toll Way – Event Generator
Toll Violations
Toll Violation Tickets
![Page 13: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/13.jpg)
IOT
WEARABLE MANAGEMENTProcessing device data in real time.
AzureStream Analytics
Power BIDashboards
Power BI DatasetTemporal
AzureEvent Hub or
IOT Hub
APIDevice
HD Insight Spark SQL Analyze
![Page 14: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/14.jpg)
USE CASE:
ITERATIVEEXPLORATION
• What can we do with all of this data?
• Mine for answers-one question at a time.
![Page 15: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/15.jpg)
ITERATIVE
EXPLORATION
Build expert systems, move to supervised learning, and evolve to reinforced learning.
Azure Machine LearningAPI End Point
Web Service used for Orchestration
HD Insight Azure Data Warehouse
Power BI
![Page 16: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/16.jpg)
ITERATIVE
EXPLORATION
Monitor and remove noise from textual data.
Stream Analytics
Power BIDashboards
Machine LearningAPI End Point
Azure SQL DBKeyword Analytics
Power BI DatasetStatistical
Power BI DatasetTemporal
MediaServices
Event Hubs
Web Service used for Orchestration
![Page 17: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/17.jpg)
USE CASE:
SELF SERVICE
• Are your reports only telling half the story?
• Quickly deliver large datasets for ad hoc analysis.
![Page 18: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/18.jpg)
SELF SERVICE
Allowing business to fulfill their analytics needs.
Apache Hadoop Spark SQL AnalyzeSemi-structured Files
SQL Server
Service Bus
![Page 19: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/19.jpg)
HYBRID
SELF SERVICE
![Page 20: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/20.jpg)
HYBRID
SELF SERVICE
![Page 21: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/21.jpg)
USE CASE:
DATA AS ASERVICE
• Got savvy end users that need more data?
• Provide data scientists with what they need
while making it easy for the business user.
![Page 22: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/22.jpg)
Data-as-a-Service
USING AZUREUsing Data Lake to capture all data for everyone.
Data Sources
Loading Data Lake
RawData Lake
Building Data Streams
Self-ServiceCatalog
AzureStream Analytics
Power BIDashboards
Azure Blob Storage
AzureEvent Hub or
IOT Hub
APIDevice
SQL
Azure Data Lake Store
Azure ML
HDInsight Hive or Spark
Data Factory
Azure Data Catalog
Azure Data Lake Store
App Service
Click Stream Logs
Data Historian(PI Server)
Data Factory
Data Factory
![Page 23: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/23.jpg)
Advanced AnalyticsMethodology
![Page 24: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/24.jpg)
Solution Development Process
Visual AnalysisData Acquisition Model(s) Selection Model Comparison
Build Model + Web Service Location for SQL query
Understanding DataModel Creation + Testing
Integration in Data Strategy
Business Objective
Understanding Data Model Creation + Testing
Integration in Data StrategyConsumption Layer
![Page 25: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/25.jpg)
Model Selection: Supervised (we know the response).
Parametric
Regression• Linear
• Polynomial
• Stepwise
• Binomial
• Splines
• Partial Least Squares
• Generalized Linear Models
Classification• Logistic
• Linear / Quadratic Discriminant Analysis
Non Parametric
• K Nearest Neighbors
• Decision Trees
• Random Forests
• Boosting
• Neural Network
• Support Vector Machines
• Generalized Additive Models
*Some models can change (parametric/nonparametric) and (regression/classification)
• Moving Averages
• Exponential Smoothing
• ARIMA
• Regressions
Forecasting
![Page 26: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/26.jpg)
Model Selection: MAPE & RMSE & R^2
Mean Average Percent Error
Root Mean Square Error
Variation explained by Predictor
We want to choose the model that reduces the test error and has a high percent value for how much the predictors explains the response
![Page 27: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/27.jpg)
Examining Weather and Active Meters in the System by Time
Seasonality of temperature
Active Metes by timeTemperature by time
Constant increase of active meters
![Page 28: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/28.jpg)
Usage by Day of Week & Verse Temperature
Day of Month
Usa
ge &
Tem
p
Hourly Usage TrendsDay of Week Trend
Temp = RedUsage = Blue
![Page 29: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/29.jpg)
Auto-Regressive Integrated Moving Average
AR(p) = number of seasonal autoregressive termsI(d) = number of differencing termsMA(q) = number of seasonal moving average terms
m = periods inside frequency
ARIMA(p,d,q)x(P,D,Q)[m]
StationaryMean
&Variance
Avg. Temperature Time Series
![Page 30: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/30.jpg)
![Page 31: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/31.jpg)
Information Management
Big Data Storage
Apache Hadoop
Real-time intelligence
Machine learning
IoT
Dashboards and Visualizations
and more!
Ideate, chart your “quick wins,” ask questions and get answers to your real Big Data challenges.
It’s insightful, it’s easy and can be done from the comfort of your conference room.
www.neudesic.com/meetneat
NEXT STEP
BECOME THE BISUPERHERO
![Page 32: ADVANCED ANALYTICS & IOT - Neudesic...Neudesic partnered with one of the nation’s largest utility companies that recently deployed Smar Utility Meters for power customers, nearly](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec4cd03250a0e6b8828ef91/html5/thumbnails/32.jpg)
Orion [email protected]
Twitter: @oriongm
Marc [email protected]
BIG DATA &
Advanced AnalyticsRoadshow
Questions?