driving big data · • better processing performance • extend existing edw capacity • meet...
TRANSCRIPT
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 1
Davy Nys, VP EMEA & APAC [email protected]
December 2013
Driving
Big Data
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 2
The New Reality Simplified Analysis for all Users
ANY Analytics
• Reports
• Dashboards
• Visualizations
• Discovery
• Predictive
Analytics
ANY Environment
• Data warehouses
• Data marts
• Stack vendors
• Cloud
• Embedded
Existing & New Data
Infrastructure &
Processes
ANY Data
• Relational
• Operational
• Big Data
• Data sources not yet
anticipated…
Billing
Location
Social
Media
Customer
Web
Network
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 3 © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 3
Emerging Funded Big Data Use Cases
ENHANCED 360 VIEW OF CUSTOMER • What makes them tick, why they buy, preferences
BIG DATA EXPLORATION • Find, visualize & understand all the data stored across silos
DATA WAREHOUSE AUGMENTATION • Optimize data warehouse – offload appropriate data
MACHINE & OPERATIONAL DATA ANALYSIS • Machine & ops data from sensors, meters, GPS devices…
SECURITY/INTELLIGENCE • Lower risk, detect fraud & monitor cyber security
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 4
Evolving Big Data Architectures
P D I
Existing
ETL Tool
or PDI EDW Data Marts
Analytics
Existing
ETL Tool
or PDI
Customer
Provisioning
Billing BI Tools
Location
Web
Social
Media
Network
Existing
Process
or PDI Hadoop
Cluster
NoSQL
P D I
Analytic DB
On-Demand Integration & Blending
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 5
Why Blending at the Source Matters Customer Experience Analytics for Loyalty and Revenue
Analytics
Analyze quality of service: • Network outages
• Dropped calls
• Poor quality
• Calls to support center
For profiles of customers: • Up for renewal
• Profitable
• Multiple agreements/services
• In competitive area
Determine best action to take: • Billing Credit
• Customer Coupon
• No Action
EDW
Existing ETL Tool
or PDI Customer
Billing
Provisioning
Call Detail Records from:
• Billing
• Payment
• Usage
NoSQL Network
Location
PDI
Call Detail Records from Network:
• Outages
• Drops
• Service Quality
PDI
Blend revenue-related and
quality-of-service data
together to find customers at
risk
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 6
Entry
Tra
nsfo
rm
Advanced
Op
tim
ize
A Spectrum of Big Data Use Cases What the Market is Deploying Today and Planning for Tomorrow
Data
Warehouse
Optimization
Streamlined
Data Hub
Big Data
Exploration
Customer
360 Degree
View Harnessing
Machine &
Sensor Data
Next
Generation
Applications
Internal Big
Data as a
Service
On-Demand
Big Data
Blending
Big Data
Predictive
Analytics
Use Case Complexity
Bu
sin
ess
Imp
act
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 7
Data Warehouse Optimization Shrink Data Costs & Boost Analytics Performance for Business Users
PDI
CRM & ERP
Systems
Other Data
Sources
Hadoop
Cluster
Data
Warehouse Analytical
Data Mart
Relational
Layer
PDI
PDI PDI
Why Do It? • Save data capacity & management
costs
• Empower business users to meet
their operational goals on time
Benefits • Lower data management costs
• Better processing performance
• Extend existing EDW capacity
• Meet batch window SLAs to
deliver fresh data to users
• Retain more data for analysis
Challenges • May require new coding skillsets
that are hard to find
• Reporting off ‘active archive’
requires a relational layer on top of
the big data store (such as Impala
or Stinger)
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 8
Streamlined Data Hub Drive a Sustainable Analytics Strategy with Big Data ETL at Scale
Transactions –
Batch & Real-time
PDI Enrollments &
Redemptions
Location,
Email, Other
Data
Hadoop
Cluster
PDI Analytical
Database
Analyzer
Reports
Benefits
• Establish usable analytics on diverse
sources at high volume (terabytes+)
• Speed queries substantially with
rapid ingestion & powerful
processing
• Reduce costs of ETL
Challenges
• Expansive integration project
• May require new coding skillsets that
are hard to find
• May call for swapping from a DW to
an Analytical DB, depending on
requirements
Why Do It?
• Give business users insight into all
data
• Scale ETL and data management
cost savings
• Next step after DW optimization
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 9
Big Data Exploration & Discovery Tap the Latent Value in Massive Data from Diverse Sources
PDI Social Media
Web/Mobile
Tracking
Hadoop
Cluster
Tracking
Data Mining
& Discovery Analytical
Database BI Tools
PDI
Benefits • Discover new useful information and
understand its value
• First step toward identifying trends
and drivers that can affect business
outcomes
• A low-risk place to start turning Big
Data into business value
Challenges • May require new coding skillsets that
are hard to find
• Must properly scope/contain the
costs of an exploratory project
• Data mining component may require
expensive skillsets – data scientists
and PhDs
Why Do It? • Understand the data you have
• Identify crucial patterns in your
business & operations
• Motivate high-impact projects
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 10
Internal Big Data as a Service Cost Effectively Scale Database Service Across Teams
PDI
Hadoop
Cluster
-or-
NoSQL
Transactional
data
PDI
Log data
Relational
Data
Other
sources
IT User
Access to
Data &
Analytics
Benefits • Scale productivity through
centralized data infrastructure
• Provide reliable service and
enterprise-grade SLAs across IT
organization
• Repurpose your high-value tech
experts to service a broader
stakeholder base – share expertise
Challenges • Hard to find skillsets to migrate data
into NoSQL
• Need to scope out reporting strategy
in addition to operational use of Big
Data for shared IT service
• Oriented primarily to IT and
developers – must still address
business user analytics approach
Why Do It? • Save costs by standardizing data
service across all IT teams
• Promote operational efficiency
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 11
Customer 360 Degree View A Blended View to Drive Revenue Growth and Service Improvements
NoSQL
CRM
System
Documents &
Images
Admin.
Info
Claims
Online
Interactions
Call Center
View
Research
Analysts
Predictive
Analytics
PDI PDI
Benefits • All customer touch point data in a
single repository for fast queries, &
all key metrics in a single location for
business users
• Blend previously isolated data and
avoid point-to-point integrations
• Boost customer service & revenue
Challenges • Transformative effort in both
technology implementation &
business planning/definition
• Complex data structures and ETL
tools for collecting & enriching data;
complex data schemas
• May require new coding skillsets that
are hard to find
Why Do It? • Learn how your customers perceive
your brand
• Boost revenue
• Lower churn
• Increase cross-sell & upsell
effectiveness
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 12
Harnessing Machine & Sensor Data Operational Intelligence to Spur Service Automation & Product Innovation
Device Network
High Velocity
Data Storage
Nodes
Message Queue
(Kafka)
Web Portal –
Dashboard,
Visualization, Admin
Message
Processing
(Storm)
NoSQL
Hadoop
Cluster PDI
PDI
Benefits • Can enhance revenue & cut costs
• Reduce cost of customer support &
increase customer satisfaction
• Optimize service offering according
to consumption patterns
• Ability to retain customers through
understanding their experience
Challenges • Having right skillsets for design,
implementation, & operation
• Project needs to be properly scoped
& defined to ensure scalability
• Often combines several different
emerging technologies
Why Do It? • Understand how your products are
used in the field
• Reduce service costs & churn
• Enable value-added product
innovation
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 13
On-Demand Big Data Blending Accelerate Analysis to Support Just-in-time Decisions
PDI
NoSQL PDI
DW Existing
ETL or
PDI Business
Analytics Integration
& Blending
Just-in-time
Customer
Provisioning
Billing
Network
Location Benefits
• Unlock value of near real-time data:
Act on it today, not next week
• Quickly react to customer behavior
• Improve operational effectiveness as
issues arise
• Connect to new data sources without
increasing database cost
Challenges • May require new coding skillsets that
are hard to find
• Proper scoping so that only time-
sensitive data that needs to be
analyzed on demand is streamed
directly ‘from the source'
Why Do It? • To analyze Big Data right away
• Support on-demand info needs
without sacrificing accuracy or
governance
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 14
Next Generation Apps Deliver Value via Architecture Innovation & Embedded Analytics
Business
Analytics
Server
Hadoop
Cluster
PDI
RDBMS
Data
Mart
Metadata
Analyzer
Dashboards
Reporting
Embedded Analytics
Content in Web App UI Benefits • Faster data processing for better
application performance
• Optimize service by data mining &
benchmarking on very large data
sets (i.e. customer base) on the fly
• Tap into high volume unstructured
data sources (i.e. Social) to make
apps more intelligent and flexible
Challenges • A 'bet the farm' strategy, a major
change in your app infrastructure
• Open ended - Requires original
thinking to create value proposition
• Heavy investment in skillsets
• Relatively long term project adds to
uncertainty
Why Do It? • To create a unique value
proposition
• Build competitive advantage
• Drive sales & win markets
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 15
Big Data Predictive Analytics Supercharge Predictions by Refining Models in Hadoop
PDI Social Media
Hadoop
Cluster
Tracking
Data Mining
& Predictive
Analytics
PDI
Predictive
Analytics
in-cluster
Web
Behavior
Data
Benefits • Data processing power, speed, and
scalability of Big Data stores can
facilitate increased accuracy for
outcome prediction
• Revenue enhancement and risk
reduction potential, depending on
specific use case
Challenges • Requires data scientists and PhDs -
expensive resources
• Usually a second or third use case,
after experience with respect to Big
Data has been developed
Why Do It? • Improve prediction of business
risks, like fraud or security
breaches
• Improve predictions of customer
behavior, like buying decisions
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 16
Entry
Tra
nsfo
rm
Advanced
Op
tim
ize
A Spectrum of Big Data Use Cases What the Market is Deploying Today and Planning for Tomorrow
Data
Warehouse
Optimization
Streamlined
Data Hub
Big Data
Exploration
Customer
360 Degree
View Harnessing
Machine &
Sensor Data
Next
Generation
Applications
Internal Big
Data as a
Service
On-Demand
Big Data
Blending
Big Data
Predictive
Analytics
Use Case Complexity
Bu
sin
ess
Imp
act
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 17 © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 17
Final Parting Thoughts
EXPECT ADOPTION TO DRIVE MORE USE CASES • Have a future proof architecture
KEEP DATA GOVERNANCE IN MIND • User flexibility is key, complex data blending should be architected
LEVERAGE EXISTING INFRASTRUCTURE • Optimize/augment data warehouse – offload appropriate data
AVOID WRITING LEGACY CODE • Flexibility, Time to value & Cost savings
AVOID DATA VENDOR/TYPE LOCK IN • New use cases, new data sources, new data types, ….
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 18
Thank You
blog.pentaho.com
@Pentaho @davynys
Facebook.com/Pentaho
Pentaho Business Analytics
JOIN THE CONVERSATION. YOU CAN FIND US ON: