tdwi solution spotlight presentation slides
TRANSCRIPT
Helping Organizations Transform Data into Business Value with the Power of
Intelligence.
2
TDWI Events• Learn More About Upcoming Seminars & Conferences at
www.tdwi.org/events
• CBIP Certification• Exhibit hall featuring vendors showcasing the latest in
technology offerings• Valuable networking opportunities
Conferences• July 18-20 , Boston
• October 2-7, San Diego
• December 4-9, Austin
Seminars
• July 25-28 , Salt Lake City
• September 19-22, Washington DC
• September 26-29, Minneapolis
• October 17-20, New York
• Upside – The content hub where new BI & analytics articles are published daily.
• eLearning – Learn everything from data viz to predictive analytics anytime, anywhere online.
• Seminars – Three-day in-depth learning opportunities in a city near you.
• Conferences – A week long opportunity to network & learn from seasoned instructors.
• Onsite Education – TDWI Instructors providing hands-on training in your offices.
• More information: tdwi.org/education
• TDWI Premium Membership helps you:– Stay current on trends & best practices.– Validate your data direction & approaches.– Network with peers in the industry.– Gain access to a trusted, independent source
for actionable insights. – Save money on TDWI events.
• More information: tdwi.org/membership
Data Warehouse Modernization
A Response to Big Data, Advanced Analytics, and Other Business Opportunities
Philip RussomSenior Research Director for Data Management, TDWI
Agenda
PLEASE TWEET@pRussom, #TDWI, #DataWarehouse, #Modernization, #Analytics, #RealTime
• A Time of Great Change– The evolution of data, its
management, and its use• What is data warehouse
modernization?– What is its state?– Why is it important?
• Benefits and Barriers• Best Practices
– Organizing DW Mod projects– Modernization strategies
• Trends in DW Mod– DWEs & evolving architecture– Role of Hadoop
• Top 12 Priorities for DW Mod• How to get the TDWI report
this presentation is based on• Q&A
6
TRENDS DRIVING
Data Warehouse Modernization
• Data is evolving• Data management is evolving• Business use & leverage of data is evolving
Data is Evolving
• Exploding data volumes– Demands speed & scale
from DM platforms & solutions
• Big data is more than big– It’s new, diverse, loaded with opportunity
• Structural diversity– Coming from new sources, feeding new targets– Data types and structures that are new to you
• Generated more frequently– Demands use of event processing and real-time tech– Demands monitoring and reaction from business
Data Management is Evolving
• Emerging practices– Data exploration, discovery analytics– Data lakes, data hubs
• New data platforms– Columnar, appliances, Hadoop, cloud
• New tools and methods– Data prep, self-service data access– Event processing, true real time– Early ingestion, in-line analytics
Biz Use & Leverage of Data is Evolving
• More business value and organizational advantage– Decisions based on more and better facts– More complete views of customers– Operations move faster, based on fresher data– More competition based on analytics, with massive data– More analytics, in general, in more advanced forms
• Greater governance for biz compliance & data standards
“DW Modernization” takes many forms…• Additions to existing data warehouse environment (or ecosystem)
– New data subjects, sources, tables, dimensions, etc.– More server instances, nodes, bigger storage
• Upgrades– Newer versions of current
database or integration software– Bigger and faster hardware
• More standalone platforms & tools– Complement DW wo/replacing it– Tools for analytics, real time,
new data types, new interfaces– New appliances,
columnar databases, Hadoop, NoSQL, etc.
• Architectural Adjustments– Logical DW design across
multiple platforms– Extending data integration (DI)
• Rip and Replace– Decommission current DW platform or misc tools; migrate to others
11
ACCORDING TO TDWI SURVEY RESULTS
State of Data Warehouse Modernization
• Modernizing a DW is extremely important (58%) or moderately important (33%)
• DWs are evolving dramatically (22%) or evolving moderately (54%)
• DWs are fully (7%) or mostly up-to-date (41%)• DWs are somewhat (38%) or far behind (12%)• DWs are still very relevant (49%) or
somewhat relevant (39%) to what biz wants
12
SOURCE: TDWI 2016 DW Modernization Report, extracted from Figures 2, 4, 5, 6.
Data Warehouse Modernization
is mostly a problem
Data Warehouse Modernizationis mostly an opportunity
SOURCE: TDWI 2016 DW Modernization Figure 7.
• “One has to keep up with the volumes and variety of data that can enhance your analytical results for better decision making and customer service.”
– Data architect, Financial services, Africa• “The business landscape is constantly changing, and it’s evolving the
data requirements. If you do not change with the times, you will become obsolete.”
– Enterprise architect, Petroleum, Canada• “In the past, week-old data might have sufficed; but today we need
near real-time data.”– BI Manager, State/local government, USA
• [We need DW mod.] “to achieve low TCO, integrate with digital channels, support fast business decisions, allow complex analytics.”
– CTO team member, Financial services, Asia• [Our] “current solution was built five years ago on twenty-year-old
technology and patterns. Latency, performance, and scope all lag far behind today’s needs.”
– Data architect, Insurance, USA
SOURCE: TDWI 2016 DW Modernization Report, extracted from Figure 3
IN USERS’ OWN WORDS
Why is DW Modernization Important?
14
Biz Benefits of DW Modernization
0% 10% 20% 30% 40% 50% 60%
Complete views of customers and other important entities
Address new business requirements
Competitive advantages
Agile delivery of solutions, for nimble business responses
Operational efficiency of business
Fast and frequent report/analysis cycles, near real time
Business decision making, both strategic and operational
Analytics, including visualization and exploration
SOURCE: TDWI 2016 DW Modernization Report, top half of Figure 8
What are the top business tasks that would benefit from data warehouse modernization?
15
Challenges to DW Modernization
• Overcoming bad practices in data management– Poor stewardship/governance, data quality, metadata
• Personnel problems– Inadequate staffing, skills, experience
• Paying the price of modernization– Cost of implementation, hardware/software upgrades
• Complexity of architecture– Designing & managing a multi-platform systems environ
• Limitations of existing systems– Current environment won’t scale up to big data, ingest data fast enough
• Outmoded development environment and practices– Need tools that foster speed for agility, plus reuse for productivity
SOURCE: TDWI 2016 DW Modernization Report, extracted from Figure 9
16
TO SURVIVE THE CHALLENGES,
Recognize Standard DW Modernization ScenariosAND STAFF OR SCHEDULE THEM ACCORDINGLY.
• System modernization (53%)– upgrades and patches for
hardware/software servers or tools• Arbitrary modernization (47%)
– based on business needs of a specific project, or urgent request for info/analysis
• Non-data modernizations (44%)– modernizing reporting, analytics, data
integration• Optimization modernization (42%)
– performance tuning and similar tweaks• Continuous modernization (37%)
– quarterly updates, complete views, etc.• Disruptive modernization (21%)
– rip and replace platforms, tools, datasets
SOURCE: TDWI 2016 DW Modernization Report, Figure 10
DEFINITION
Data Warehouse Environments (DWE)
• Many enterprise data warehouses (EDWs) are evolving into multi-platform data warehouse environments (DWEs).
• Users continue to add additional standalone data platforms to their warehouse tool and platform portfolio.– New platforms = relational DBMSs based on columns,
appliances, clouds; real-time complex event processing; Hadoop• The new platforms don’t replace the core warehouse, because it is
still the best platform for the data that goes into standards reports, dashboards, performance management, and OLAP.
• Instead, the new platforms complement the DW, because they are optimized for workloads that manage, process, and analyze new forms of big data, non-structured data, and real-time data.
BEST PRACTICES
Modernization Strategies• Most common strategy – DW Augmentation (42%)
– Add more data platforms to DWE, to complement existing core DW• For only 15%, replacing DW’s primary data platform has been a strategy• 24% modernize on per case basis; 14% don’t have a strategy
19
SOURCE: TDWI 2016 DW Modernization Report, Figure 11
TRENDS in DW MODERNIZATION
Evolving DW Platform Architectures• Single-monolith DW architectures aren’t that common and are slipping away• Simple DWE (a few platforms) is now the norm for DW systems architecture• Future: we’re trending strongly toward complex DWEs (many platforms)
20
SOURCE: TDWI 2016 DW Modernization Report, Figure 15
Seventh Inning Stretch…
TRENDS in DW MODERNIZATION
Role of Hadoop in DW Modernization• Hadoop in DWEs is still rare today (15%), but will increase 5x (78%) in 3 years• Hadoop usually complements a primary DW platform – 17% today, 36% in 3yrs• Hadoop rarely replaces a DW – 1% today, 6% in three years
22
SOURCE: TDWI 2016 DW Modernization Report, Figure 16
• Hadoop offloads DW and extends DWE– Captures and manages big data at scale– Data landing & staging on steroids– Repository for detailed source data– Processing for analytics & data integration– Advanced forms of algorithmic analytics
(mining, graph, predictive)– ELT push-down processing– Manages multi- and unstructured data– Inexpensive compared to capacity based
pricing on average relational DW or DBMS
TRENDS in DW MODERNIZATION
How can Hadoop Modernize a DW?
Top 12 Priorities for DW ModernizationThese are recommendations, requirements, or rules that can guide you.1. Embrace change.2. Make realignment with business goals your top priority.3. Make DW capacity a high priority on the technology side.4. Make analytics a priority, too.5. Don’t forget the related systems that also need modernization.6. Don’t be seduced by new, shiny objects.7. Assume that you’ll need multiple
manifestations of modernization.8. Know the tools and techniques of modern DWEs.9. Adjust the large-scale architecture of your DWE.10. Reevaluate your DW platform.11. Consider Hadoop for various roles in the DWE.12. Develop plans and recurring cycles for DW modernization.
24
Download a free copy of the report that this presentation is based on
• Download the report in a PDF file at:
tdwi.org/bpreports
• Feel free to distribute the PDF file of any TDWI Best Practices Report
25
The Modern Data WarehouseSAP Solutions for a New Era
June 2016
Michael BreenPlatform ArchitectCustomer Innovation & Enterprise Platform Group, SAP
27© 2015 SAP SE or an SAP affiliate company. All rights reserved.
75%of global workforce
will beMillennials
We are entering into a new era of unprecedented change across a multitude of dimensions
5 billionpeople worldwide
will becomemiddle class
50%of the world’s population
will live underwater shortage
1.3 billionpeople on business & social networks today
50 billion connected devices and
“internet of things” by 2030
Rising Customer Expectations A Dramatically Changing Workforce Pressure on Resources
Network Effect/Explosion in Structured and Unstructured data
27
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 28Public
Data LakeData volumes will continue to grow to 6 billion petabytes, including unstructured data such as social networking data and low level IoT data. Mining the value from this data is essential
CloudCloud spending will surge by 25%, reaching over $100 billion. There will be a doubling of cloud data centers.
Internet of Things30 billion devices, sensors in 2020 –driving $8.9 Trillion in revenue. The need for real-time processing and analytics will explode
Mobile
CRM Data
Planning
Opportunities
Transactions
Customer
Sales Order
Things
Instant Messages
Demand
Inventory
Big Data
Sales Order
Things
Mobile
Demand
Big Data
CRM Data
CustomerPlanning Transactions
Key Trends
28
29© 2015 SAP SE or an SAP affiliate company. All rights reserved.
40% executives worry that their organizations will not keep pace with technology change and lose their competitive edge.
– McKinsey study, 2013”“
Complexity built up over decades limits the ability to innovate; radical simplification is needed to unlock the potential.
Drive business innovation
Keep the lights on
IT EnvironmentCollapse redundant infrastructure layers
User ExperienceEngage front line employees/customers
28%
72%ConsumptionFor immediate business impact
Forrester IT Survey, 2013
Drive business innovation
Keep the lights on
29
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 30Public
SAP’s Data Warehouse enables a revolutionary approach streamlines and simplifies data warehousing
Providing greater speed and scale along with agility for development and efficiency that reduces data movement and data preparation. SAP’s complete architecture offers:
A-z
Flexible Architecture Rapid DeploymentPre-packaged or Customize
30
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 31Public
e
Customized Data Warehouse
• Controlled schemas, often prepopulated with structure
• Lifecycle management of schemas
• High level languages and less programming
• More prebuilt tools to purpose
SAP HANA platform
Processing Engine
Application Function Lib. & Data Models
Integration Services
SAP HANA PLATFORMReal-time transactions + end-to-end analytics
Extended Application Services
HANA Smart Data Streaming
HANA Dynamic Tiering
• Usually depends on SQL tools and low-level programming
• Fewer controls on schema updates• Easier to change
- An integrated architecture that reduces data redundancy while keeping all information at hand
- Utilizes state-of-the-art in-memory techniques that furnish answers in-context, in real time
- Makes more data available at the right time to the right person at the right place in the business process
SAP Provides The Best of Both Approaches!
More
Deg
rees
of F
reed
omLess Time to Implement
SAP Gives You The Power of Both Custom and Packaged
Pre-Packaged Data Warehouse
HANA Advanced Analytics
31
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 32Public
Traditional Data Warehouses Just Copy Data AndCreate More And More Copies In Indexes
CO
PY
Business Data:ERP, CRM, SCM
Reference/Supplier Data
Data FabricData Remains in Place!
Hadoop /Social Media
Data Bloat slows the database & becomes hard to manage
Historical Data
Hadoop /Data Lake
ReferenceData
HANA Keeps Critical Data in Memory withoutCopies or Support Indexes
Business Data:ERP, CRM, SCM
Streams & Context
Computations & Management are Streamlined without bloating database
Cop
y sc
hedu
les
dela
y da
ta
Cub
es a
nd In
dexe
sta
ke ti
me
to b
uild
HANA Flexible Architecture Example: Data Fabric
Real TimeSmart Data Access
32
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 33Public
YARN
HDFS
Enable Precision DecisionsWith Contextual Insights In Enterprise Systems
Other Apps
Files Files Files
HANA-Spark Adapter for improved performance between distributed systems
Gain business coherence with business data and big data
Compiled queries enable applications & data analysis to work more efficiently across nodes
Familiar OLAP experience on Hadoop to derive business insights from big data such as drill-down into HDFS data
Compiled Queries
Spark Adapter
Drill Downs
SAP HANA in-memory platform
Vora
Spark
Vora
SparkIn-Memory
StoreApplication Services
Database Services
Integration Services
Processing Services
SAP HANA Platform
Vora
SparkHANA-Spark Adaptor
33
34© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Customer value delivered by SAP Data Warehouse
Internet of Things
Eliminate or reduce data movement
Fewer copies of data
Enterprise Wide Analytics
Simplified Architecture
Real-time Analytics
Data Lake
Access data across your enterprise
Unmatched federation of data without centralizing
In-memory performance gives answers in seconds, not hours
Reduced latency means current data is addressed not old data
Petabytes of historical data storage
Advanced analytics for mining non-traditional data
Extensive Hadoop and no-SQL support
Data management and analytics from device to enterprise
Streaming analytics
34
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 35Public
Winning Combinations
SAP HANA* and Intel® Xeon® processors help customers get the most from their growing data
*See the latest SAP HANA* certified OEMs and appliances: http://global.sap.com/community/ebook/2014-09-02-hana-hardware/enEN/index.htmlSoftware and workloads used in performance tests may have been optimized for performance only on Intel microprocessors.
Optimized for Flexibility
Deploy SAP HANA
On premises On demand/hybrid cloud
Built for Each Other
More transactions per minute
Collaborative Partnership
Using your platform of choice from 15 industry leading OEMs* & CSPson the Intel Xeon processor E7 v3 family1, 2
+
35
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 36Public
From 50m to 5m for failed tests…
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 37Public
“We have built a highly innovative and scalable platform for the future. We really see this solution as a game-changer for the automotive industry.” Dirk Zeller, Head of IT Consulting, Mercedes-AMG GmbH
… led to 15% increase in overall test cycle capacity
In the future, Reinhard Breyer, CIO of Mercedes-AMG GmbH, explained that, “This breakthrough innovation is just the start. Ultimately we want to monitor engine performance in customer vehicles.”
38© 2015 SAP SE or an SAP affiliate company. All rights reserved.
500Metrics analyzed to identify outliers
100%Accuracy
97%Confidence that a signal is a true positive
6 weeksProject duration
Anticipates consumer behavior
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 39Public
eBay Early Signal Detection System Powered by Predictive Analytics on SAP®
HANA
CompanyeBay
HeadquartersSan Jose, California
IndustryProfessional Service (Internet)
ServicesOnline Marketplace
Employees
31,500 (2012)
Revenue or BudgetUS$ 14.1 billion (2012)
Web Site
eBay.com
Business Challenges Increase ability to separate signal from noise to identify key changes to
the health of eBay’s marketplace Improve predictability and forecast confidence of eBay’s virtual economy Increase insights into deviations and their causes
Technical Challenges Detect critical signals from 100 PBs of data in eBay EDW Highly manual process because one model does not fit all the metrics
hence requires analyst intervention
Key benefits Automated signal detection system powered by predictive analytics on
SAP HANA selects best model for metrics automatically; increases accuracy of forecasts
Reliable and scalable system provides real-time insights allowing data analysts to focus on strategic tasks
Decision tree logic and flexibility to adjust scenarios allows eBay to adapt best model for their data
“HANA is valuable in the sense that it accelerates that speed to insight. HANA, with in-memory capability, with multicore, fast, lots of data, all of that coming together is how I think analytics is going to work broadly in the future.”
David Schwarzbach, VP&CFO eBay North America at eBay Inc.
“HANA system will free up all the bandwidth right now involved in figuring out what is going. The user just has to feed in their metric, doesn’t have to really worry about which algorithm is the best and be able to use the system because it is inherently intelligent and configurable.”
Gagandeep Bawa, Manager, North America FP&A at eBay Inc.
Determine with 100% Accuracythat a signal is positive at 97% confidence
Automated Early Signal Detectionsystem powered by SAP HANA
40© 2015 SAP SE or an SAP affiliate company. All rights reserved.
+ $10M Revenue
Inventory scheduling andre-allocation in real-time
- 3% Cost
Sense deviations in tire temperature and pressure
+ $17M Revenue
Help brands harness word-of-mouth from social media
- 50% Inventory
Enable greater supply chain control to improve inventory
- €500k Capital
Drive profitable decisions with real-time analysis
+ $1.1M Revenue
Faster, earlier intervention to reduce student drop-outs
100% Accuracy
Monitor marketplace health through automated signal detection
$10-25savings per win-back
Measure the value of marketing campaigns: promotions, customer loyalty, adoption rates
~2 secondExecution
Gain competitive advantage with predictive analysis
99% fasterETL Load time
Offer rapid social media analysis to track consumers and influencers
5M People
Implement austerity guidelines to achieve cost savings of ~25%, and provide better care
€ 1.2Mpotential savings
Improve out-of-stock and loss prevention through real-time analysis
- 5% total cost
Use real-time info to operate call centers: greater productivity, first-call resolution rate, and lower cost
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Thank you
© 2014 SAP AG or an SAP affiliate company. All rights reserved.
http://hana.sap.com/dwl
Michael [email protected]