making data-driven decisions with a hana native agile datamart.sapevents.be/mdw/presentations/data...
TRANSCRIPT
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 1Customer
. . .nemeryphilippe@nemeryp
Making data-driven decisions with a HANA native agile datamart.
October 2016 [email protected]
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 2Customer
Marketplace Early Signal Detection Systempowered by SAP HANA Data Platform & Predictive Analytics
Link
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 3Customer
Predictive Maintenance
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 4Customer
Predictive Quality Insurance
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 5Customer
Forecasting - Predictions
?NOW
PAST FUTURE
Time-Window
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 6Customer
NOWPAST FUTURE
Sensor-1
Sensor-2
?
Forecasting - Prediction
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 7Customer
NOWPAST FUTURE
Time-Window
Sensor-1
Sensor-2
Action-Window
Action-Window: time
during which an
action can be taken
(e.g. 10 minutes)?
Forecasting - Prediction
Classification: Will there be an incident in 10 minutes?
Regression: What is the value in 10 minutes?
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 8Customer
NOWPAST FUTURE
Time-Window
Sensor-1
Sensor-2
Action-Window?
Forecasting - Prediction
Classification: Will there be an incident in 10 minutes?
Regression: What is the value in 10 minutes?
- Values Now S1, S2, …
- Average Value S1 T-0
- Average Value S1 T-1
- Average Value S2 T-0
- Average Value S2 T-1
- Min Value S1 T-1
- Max Value S2 T-1
- Deltas between To, T-1, T-2
ToT-1T-2
Devices
BO Dashboards
External
DataFinancial,
Sales
Big Data Lake Training of Predictive Models
SDS
Triggering of alarms/actions
based on predictive models
SDS: Smart Data Streaming
SDI - SDQ
Hana Vora
Real Time
Gateway
System
Real-time Stream
Analysis
Visual
Exploration
SAP HANA
Platform
Hadoop
Devices SDS
Triggering of alarms/actions
based on predictive models
SDS: Smart Data Streaming
Real Time
Gateway
System
Real-time Stream
Analysis
Smart Data Streaming
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 13Public
Event stream processing uses continuous queries
Database Queries Continuous Queries
Step 1:
Store the data
Step 2:
Query the dataStep 1:
Define the
continuous
queries and the
dataflow
Step 2:
Wait for data to arrive.
As it arrives, it flows
through the continuous
queries to produce
immediate results
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 14Public
Complex Event Processing extracts insight from events
Sensor readings – 10’s of thousands per second
Virtually no useful
information in a
single isolated event history
e.g. Compare
variance of trends
across multiple
sensors against
historical norms
Event window – e.g. 30 min
Alert
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 15
Stream Processing
Creating equidistant data and provisioning data stores
• Streams and windows, CCL script
• Create, enrich and aggregate equidistant data
• Publish raw data to hdfs and aggregates to HANA tables
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 16Public
Apply SAP Predictive Analytics Model within SAP Smart Data
Streaming
• Automated Analytics now supports smart
data streaming
• Generates CCL Code which can be
deployed to HANA SDS
HANA
Smart Data
Streaming
Predictive Analytics
Automated
Modeller
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 17Public
Overview of SDS integration options
SDS Project
Data sources
Alerts
Applications
Adapters
input output
Appplication
with SDS
connectivity
Application with
SDS
connectivity
SDS pub/sub library
Input adapters receive/fetch messages using a
protocol supported by the source, parse the
message, and publish it to SDS as an event.
Output adapters do the same – in reverse.
The SDS SDK provides a library that
can be embedded in adapters or
applications to connect to SDS and
publish/subscribe
Devices
BO Dashboards
External
DataFinancial,
Sales
Big Data Lake
SDS
Triggering of alarms/actions
based on predictive models
SDS: Smart Data Streaming
SDI - SDQ
Real Time
Gateway
System
Real-time Stream
Analysis SAP HANA
Platform
Hadoop
Hana Vora
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 19Customer
SAP HANA: The Real-time Platform of the Digital Economy
SAP HANA PLATFORMON-PREMISE | CLOUD | HYBRID
Web Server JavaScript
Fiori UX Graphic
Modeler
Data Virtualization ELT &
Replication
Application Services Integration & Quality Services
Columnar
OLTP+OLAP
Multi-Core &
Parallelization
Advanced
Compression
Multi-
tenancy
Multi-Tier
Storage
Spatial Graph Predictive Search
Text
Analytics
Data
Quality
Series
Data
Business
Functions
ALM
Processing Services
Database Services
Hadoop & Spark
Integration
Streaming
Analytics
Application Lifecycle
Management
High Availability &
Disaster Recovery
OpennessData
Modeling
Remote Data
Sync
Admin &
Security
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 20Customer
SAP HANA: The Real-time Platform of the Digital Economy
SAP HANA PLATFORMON-PREMISE | CLOUD | HYBRID
Web Server JavaScript
Fiori UX Graphic
Modeler
ELT &
Replication
Application Services Integration & Quality Services
Columnar
OLTP+OLAP
Multi-Core &
Parallelization
Advanced
Compression
Multi-
tenancy
Multi-Tier
Storage
Spatial Graph Search
Text
Analytics
Data
QualityBusiness
Functions
ALM
Processing Services
Database Services
Streaming
Analytics
Application Lifecycle
Management
High Availability &
Disaster Recovery
OpennessData
Modeling
Remote Data
Sync
Admin &
Security
Data
VirtualizationPredictive
Series DataHadoop & Spark
Integration
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 21Customer
Time Series Data in SAP HANA
Store
Support both equidistant and non-equidistant data
Support very high volumes of data using effective compression
techniques
Handle
Efficient grouping to different granularities (GROUP BY
SERIES_ROUND(…))
Built in SQL functions for efficient handling of Series Data
– SERIES_GENERATE;
– SERIES_DISAGGREGATE;
– SERIES_ROUND;
– SERIES_PERIOD_TO_ELEMENT;
– SERIES_ELEMENT_TO_PERIOD
Analyze
Analytic operations to be expressed naturally in SQL while
maintaining high performance
– AUTO_CORR, CROSS_CORR
– BINNING
– CUBIC_SPLINE_APPROX, LINEAR_APPROX
– DFT
– RANDOM_PARTITION
– SERIES_FILTER
– WEIGHTED_AVG
– Sliding window support
– {FIRST/NTH/LAST}_VALUE
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 22Customer
IBM DB2, Netezza,
Oracle, MS SQL
Server, Teradata,
SAP HANA, SAP
ASE, SAP IQ
Modeling & SQL Script
S AP H AN A P L AT F O R M
Smart Data Access
Virtual Tables
Smart Data Integration
Built-In Adapters Custom Adapters
ODataDB2, Oracle,
MS SQL Server,
Teradata, SAP HANA,
SAP ASE
Adapter Framework
Metadata
S AP H AN A P L AT F O R M
Smart Data Access Smart Data Integration
Access any data from any source
Devices
BO Dashboards
External
DataFinancial,
Sales
Big Data Lake
SDS
Triggering of alarms/actions
based on predictive models
SDS: Smart Data Streaming
SDI - SDQ
Real Time
Gateway
System
Real-time Stream
Analysis SAP HANA
Platform
Hadoop
Hana Vora
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 24Customer
SAP HANA VoraWhat’s Inside and What Does It Do?
Democratize
Data
Access
Enable
Precision
Decisions
Making
Simplify
Big Data
Ownership
SAP HANA Vora is an in-memory query engine which leverages
and extends the Apache Spark execution framework to provide
enriched interactive analytics on Hadoop. Drill Downs on HDFS
Compiled Queries
HANA-Spark Adapter
Unified Landscape
Open Programming
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 25Customer
Vora Time Series (in Vora 1.3 – GA End September)
Sequence of data points recorded over time
Can be equidistant or non-equidistant
Detect and correct errors / anomalies
Granularization
Standard aggregation
Analysis
Specify a series clause during table creation
Define the period column (timestamp)
Provide a compression definition (optional)
Define start/end of series (optional)
Define the series increment (optional)
Column Functions
Trend
Stddev
Median
Linear_Approx
Const_Approx
Cubic_Spline_Approx
Polynomial_Approx
Table Functions (coming soon!)
Auto_Corr
Cross_Corr
Histogram
DFT
Granulize
Devices
BO Dashboards
External
DataFinancial,
Sales
Big Data Lake Training of Predictive Models
SDS
Triggering of alarms/actions
based on predictive models
SDS: Smart Data Streaming
SDI - SDQ
Hana Vora
Real Time
Gateway
System
Real-time Stream
Analysis
Visual
Exploration
SAP HANA
Platform
Hadoop
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 27Customer
Visual Exploration
?
Demo
Any DataBase (Oracle, etc.)
Consume universes (.UNX) from SAP BI 4.x
Connect to SAP Business Warehouse (BW)
Support for Big Data sources (>15K columns) including Hadoop/Hive and Spark
Guided Timeseries ForecastingFeature Building
29
Choose the deployment.
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 30Customer
Predictive Analytics Overview
Automated
Guided - Automated
Data Manager
Modeler
Expert
Workbench
Data Manip.
Workflow
Visualization
HANA
APL PAL
R
HANA SDSHadoop
Hana
Processing & DB ServicesHANA
APL PAL
R
Automated Predictive Library
(+6 BP)
Predictive Analysis Library (+100 M)
Conclusions
© 2016 SAP SE or an SAP affiliate company. All rights reserved. 32Customer
SAP SDS – HANA Advanced Analytical Platform
SAPAnalytical Platform
Future-Proof
Analytical Roadmap
Series Data
Capture and analyze a sequence of successive data points made
over a time interval
.
Enterprise E2E Platform
Centralized administration, management & auditing
Advanced & Real-Time Analytics
Experience
Project Approach and Partner network
Different Users
Role-based and maturity-based approach
Flexibility and Agility