2013.12.12 big data heise webcast
DESCRIPTION
Presentation about BigData from a German Webcast: http://business-services.heise.de/it-management/big-data/beitrag/big-data-technologie-einsatzgebiete-datenschutz-160.html?source=IBM_12_2013_IT_ConnTRANSCRIPT
Big Data Wilfried Hoge, Software IT Architect Big Data [email protected] @wilfriedhoge
How is Big Data transforming the way organizations analyze information and generate actionable insights?
Paradigm shifts enabled by big data Leverage more of the data being captured
TRADITIONAL APPROACH BIG DATA APPROACH
Analyze small subsets of Information
Analyze all information
Analyzed information
All available information
All available information analyzed
Paradigm shifts enabled by big data Reduce effort required to leverage data
TRADITIONAL APPROACH BIG DATA APPROACH
Carefully cleanse information before any analysis
Analyze information as is, cleanse as needed
Small amount of carefully
organized information
Large amount of
messy information
Paradigm shifts enabled by big data Data leads the way—and sometimes correlations are good enough
TRADITIONAL APPROACH BIG DATA APPROACH
Start with hypothesis and test against selected data
Explore all data and identify correlations
Hypothesis Question
Data Answer
Data Exploration
Correlation Insight
Paradigm shifts enabled by big data Leverage data as it is captured
TRADITIONAL APPROACH BIG DATA APPROACH
Analyze data after it’s been processed and landed in a warehouse or mart
Analyze data in motion as it’s generated, in real-time
Repository Insight Analysis
Data
Data
Insight
Analysis
How have most companies made information available for decision making across the enterprise?
Traditional enterprise data and analytics environments Typical enterprise data management environment
Staging area
Actionable insight
Enterprise warehouse Data mart
Data sources
Transaction and application data
Archive
Reporting and analysis
Predictive analytics and modeling
How are leading companies transforming their data and analytics environment to provide faster, better insights at reduced costs?
Next generation architecture Starts from the current data management environment …
Enterprise warehouse
Information governance
Data mart
Analytic appliances
Information ingestion and operational information
Data sources
SYSTEMS—SECURITY—STORAGE
Transaction and application data
Enterprise warehouse
Data mart
Analytic appliances
Actionable insight
Reporting, analysis, content analytics
Predictive analytics and modeling
Next generation architecture … adds new technologies and capabilities …
Exploration, landing and
archive
Enterprise warehouse
Information governance
Real-time analytics
Data mart
Analytic appliances
Information ingestion and operational information
Data sources
SYSTEMS—SECURITY—STORAGE
Transaction and application data
Machine and sensor data
Enterprise content
Social data
Image and video
Third-party data
Enterprise warehouse
Data mart
Analytic appliances
Actionable insight
Reporting, analysis, content analytics
Predictive analytics and modeling
Decision management
Discovery and exploration
Cognitive
+ +
Next generation architecture … to enable new applications
Exploration, landing and
archive
Enterprise warehouse
Information governance
Real-time analytics
Data mart
Analytic appliances
Information ingestion and operational information
Enhanced applications
Customer experience
Operations and fraud
Risk
Financial performance
New business models
IT economics
Data sources
SYSTEMS—SECURITY—STORAGE
Transaction and application data
Machine and sensor data
Enterprise content
Social data
Image and video
Third-party data
Enterprise warehouse
Data mart
Analytic appliances
Actionable insight
Reporting, analysis, content analytics
Predictive analytics and modeling
Decision management
Discovery and exploration
Cognitive
+ +
Sample use cases that leverage the new data management environment capabilities
Dublin City Centre; Robust and efficient citywide traffic awareness system, enhance rapid action on incidents
Need
• A budget effective solution to improve traffic awareness system.
• To bring accuracy in event detection, inferring traffic condition (road speed) and prediction of bus arrival.
• Challenge is to rightly analyze GPS data, which is typically high data throughput and difficult to capture
Benefits • Monitor 600 buses across 150 routes daily • Analyzes 50 bus location updates per
second , using InfoSphere Streams • Collects, processes, and visualizes location
data of all public transportation vehicles
Exploration, landing and
archive
Enterprise warehouse
Information governance
Real-time analytics
Data mart
Analytic appliances
Information ingestion and operational information
Enhanced applications
Customer experience
Operations and fraud
Risk
Financial performance
New business models
IT economics
Data sources
SYSTEMS—SECURITY—STORAGE
Transaction and application data
Machine and sensor data
Enterprise content
Social data
Image and video
Third-party data
Enterprise warehouse
Data mart
Analytic appliances
Actionable insight
Reporting, analysis, content analytics
Predictive analytics and modeling
Decision management
Discovery and exploration
Cognitive
+ +
Reporting, analysis, content analytics
Real-time analytics
Architecture for traffic awareness system Real-time analytics to enhance customer experience
Customer experience
Operations and fraud
Machine and sensor data
Constant Contact Transforming Email Marketing Campaign Effectiveness with IBM Big Data
Need • Analyze 35 billion annual emails to guide
customers on best dates & times to send emails for maximum response
Benefits • 40 times improvement in analysis performance
• 15-25% performance increase in customer email campaigns
• Analysis time reduced from hours to seconds
Architecture for email marketing Analyze to maximize response rates
Exploration, landing and
archive
Enterprise warehouse
Information governance
Real-time analytics
Data mart
Analytic appliances
Information ingestion and operational information
Enhanced applications
Customer experience
Operations and fraud
Risk
Financial performance
New business models
IT economics
Data sources
SYSTEMS—SECURITY—STORAGE
Transaction and application data
Machine and sensor data
Enterprise content
Social data
Image and video
Third-party data
Enterprise warehouse
Data mart
Analytic appliances
Actionable insight
Reporting, analysis, content analytics
Predictive analytics and modeling
Decision management
Discovery and exploration
Cognitive
+ + Transaction and
application data
Enterprise content
Exploration, landing and
archive
+ +
Analytic appliances
Predictive analytics and modeling
Reporting, analysis, content analytics
New business models
IT economics
Battelle, helping reduce energy costs and enhancing power grid reliability and performance Need • Assess the viability of one smart grid
technique called transactive control
Benefits • Engages consumers and responsive assets
throughout the power system to help optimize the system and better integrate renewable resources
• Provides the capability to analyze and gain insight from up to 10 PB of data in minutes
• Increases grid efficiency and reliability through system self-monitoring and feedback
• Enables a town to avoid a potential power outage
Architecture for smart grid Analytics to reduce costs and optimize grid
Exploration, landing and
archive
Enterprise warehouse
Information governance
Real-time analytics
Data mart
Analytic appliances
Information ingestion and operational information
Enhanced applications
Customer experience
Operations and fraud
Risk
Financial performance
New business models
IT economics
Data sources
SYSTEMS—SECURITY—STORAGE
Transaction and application data
Machine and sensor data
Enterprise content
Social data
Image and video
Third-party data
Enterprise warehouse
Data mart
Analytic appliances
Actionable insight
Reporting, analysis, content analytics
Predictive analytics and modeling
Decision management
Discovery and exploration
Cognitive
+ +
Machine and sensor data
Analytic appliances
Predictive analytics and modeling
Decision management
Real-time analytics
Customer experience
New business models
Operations and fraud
Trust is the most important aspect of a Big Data solution
Create foundation of trusted data
Understand usage and monitor compliance
Model exposure and understand variability
Trust the facts Ensure privacy and security
Make risk aware decisions
Trust It Be proactive about privacy, security and governance
• Data ac&vity monitoring of NoSQL, Hadoop, and Rela&onal Systems
• Masking of sensi&ve data used in Hadoop
• Ensures sensi&ve data is protected, encrypted and secure
Big Data Privacy and Security Protect a Wider Variety of Sources
InfoSphere Optim
InfoSphere Guardium
RDBMS
Hadoop
NoSQL
Data Warehouses
Application Data and Files
• Protec&on is a pre-‐requisite for the fundamental assump&on of big data – sharing data for new insight
• Automa&on enables protec&on without inhibi&ng speed
Data volumes continue to grow fast. New technologies will be needed in the future
Big Data brings Technical Challenges IBM invests in future solutions
Huge Volumes of Multimedia Data: New Storage Requirements
The Challenge of Moore’s Law: New Storage Methods Needed
IBM’s Approach: Start at the Atomic Level
how many atoms are needed to store
1-bit of data
What are your activities to leverage Big Data Analytics?
Your big data journey IBM can help wherever you are
Educate Focused on knowledge gathering and market observa&ons
25% Explore Developing strategy
and roadmap based on business needs and
challenges
47% Engage Pilo&ng big data
ini&a&ves to validate value and requirements
Execute Deployed two or more big data ini&a&ves and con&nuing to apply advanced analy&cs
22% 6%
Learn the technology & gain exper&se
Validate and realize business value Case studies, Whitepapers and Value Repors ibmbigdatahub.com
IBM Briefings, Solu&on Centers
Learning, explora&on downloads & test BigDatauniversity.com, YouTube Big Data Channel
IBM Roadmap Workshop • Priori&sed business use cases • Recommend plaRorm
Solu&on Design & Proof of Concept • Validate business value • Demonstrate capabili&es
Enterprise-‐wide big data ini&a&ves • Stampede – exper&se and skills to get value straight away
• Enterprise data plaRorm
THINK