data analytics & hpe haven big data platform - apsu · data analytics & hpe haven big data...
TRANSCRIPT
Data Analytics &HPE Haven Big Data Platform
Michael Flower
Accelerating innovation and time to value
695,000 status updates
98,000+ tweets
698,445 Google searches
1,820TB of data created
11million instant messages
168 million+ emails sent
YouTube
Viber
Qzone
Amazon Web Services
GoGrid
Rackspace
LimeLight
J ive Software
salesforce.com
Xactly
Paint.NET
Business
EducationEntertainment
Games
Lifestyle
Music
Navigation
News
Photo & Video
Productivity
Reference
Social Networking
Sport
Travel
Utilities
Workbrain
SuccessFactors
Taleo
Workday
Finance
box.net
TripIt
Zynga
Zynga
Baidu
Yammer
Atlassian
Atlassian
MobilieIronSmugMug
SmugMug
Atlassian
Amazon
AmazoniHandy
PingMe
PingMe
Associatedcontent
Flickr
Snapfish
Answers.com
Tumblr.
Urban
Scribd.Pandora
MobileFrame.com
Mixi
CYworld
Renren
Yandex
Yandex
Heroku
RightScale
New Relic
AppFog
BromiumSplunk
CloudSigma
cloudability
kaggle
nebula
Parse
ScaleXtreme
SolidFire
Zillabyte
dotCloud
BeyondCore
Mozy
Fring Toggl
MailChimp
Hootsuite
Foursquare
buzzd
Dragon Diction
SuperCam
UPS Mobile
Fed Ex Mobile
Scanner Pro
DocuSign
HP ePrint
iSchedule
Khan Academy
BrainPOP
myHomework
Cookie Doodle
Ah! Fasion Girl
PaperHost
SLI Systems
NetSuite
OpSource
Joyent
Hosting.com
Tata Communications
Datapipe
PPM
Alterian
Hyland
NetDocuments
NetReach
OpenText
Xerox
Microsoft
IntraLinks
Qvidian
Sage
SugarCRM
Volusion
Zoho
Adobe
Avid
Corel
Microsoft
Serif
Yahoo
CyberShift
Saba
Softscape
Sonar6
Ariba
Yahoo!
Quadrem
Elemica
Kinaxis
CCC
DCC
SCMADP VirtualEdge
Cornerstone onDemand
CyberShift
KenexaSaba
Softscape
Sonar6
Workscape
Exact Online
FinancialForce.com
IntacctNetSuite
Plex Systems
Quickbooks
eBay
MRM
Claim Processing
Payroll
Sales tracking & Marketing
CommissionsDatabase
ERP
CRM
SCM
HCM
HCM
PLM
HP
EMC
Cost Management
Order Entry
Product Configurator
Bills of MaterialEngineering
Inventory
Manufacturing Projects
Quality Control
SAP
Cash Management
Accounts ReceivableFixed AssetsCosting
Billing
Time and Expense
Activ ity ManagementTraining
Time & Attendance
Rostering
Service
Data Warehousing
The InternetGigabytes
Client/ServerMegabytes
Every 60 seconds
IBM
Unisys
Burroughs
Hitachi
NECBull
Fijitsu
Mainframe Kilobytes
Mobile, Social, Big Data & The CloudZettabytes
217 new mobile web users
Yottabytes
Time
Volu
me
of d
ata
Data
Technologygap
Human data
Machine data
Business data
Big DatashiftMobile apps
System logs
Data centers
Compliance archives
Internet of Things
Sensors
Social networking
Photo sharing
Wearable devices
Big Data Platform Guiding Principles
• Fully Harness 100% of data
• Seamlessly deploy and consume anywhere
• Scale AND speed without compromise
• Open, extensible & easy to adopt
• Economics that work
Industry-leading breadth & depth of capabilities
Contextualsearch
Dataexploration
Image/videoanalytics Geospatial
analytics
SQL onHadoopAccelerated
analyticsSentimentanalysis
Predictiveanalytics
Haven Big Data Platform
Access Explore Enrich Analyze Predict Serve ActAndmore...
Core big data business capabilities
On-premise In the Cloud
The OS for human informationHPE Intelligent Data Operating Layer (IDOL)
Single processing layer to handle the continuum of human information
Connect
Understand
Over 500 functions to derive actionable insights
Act & Automate
Form an understanding of information, including docs, emails, databases, social media, rich media, etc.
Access virtually any source of information
HPE IDOL - unique platform for Information
Only IDOL can handle the continuum of Human Information– Single processing layer for all data
– Continuous learning ability
– Superior speed, scalability, and simplicity
– Built in security & compliance functionality
– 400+ seamless connections to data repositories
– 1,000+ file types support
– Language independent
Ability to understand meaning makes us unique in the market
HPE Vertica – Analytical Data Warehouse
Columnar storage Compression MPP scale-out Distributed query ProjectionsSpeeds query time by reading only necessary data
Lowers costly I/O to boost overall performance
Provides high scalability on clusterswith no name node or other single point of failure
Any node can initiatethe queries and use other nodes for work. No single point of failure
Combine high availability with special optimizations for query performance
CPU
Memory
Disk
CPU
Memory
Disk
CPU
Memory
Disk
A B D C E A
Why is HPE Vertica different from other Analytical Platforms?
• Sentiment analysis• Social CRM / network analysis• Churn mitigation• Brand monitoring• Cross and Up sell• Loyalty & promotion analysis• Web application optimization
• Marketing campaign optimization
• Brand management • Social media analytics• Pricing optimization• Internal risk assessment • Customer behavior analysis• Revenue assurance
• Logistics optimization• Clickstream analysis• Influencer analysis• IT infrastructure analysis• Legal discovery• Equipment monitoring• Enterprise search
Big data use cases are business-driven and cut across a wide range of industries & functionsBig Data Opportunities across industries and use cases
Government Telecom Manufacturing Healthcare
• Drug development
• Scientific research
• Evidence based medicine
• Healthcare outcomes analysis
• Supply chain optimization
• Defect tracking
• RFID Correlation
• Warranty management
• Broadcast monitoring
• Churn prevention
• Advertising optimization
• Law enforcement
• Counter terrorism
• Traffic flow optimization
Horizontal Use Cases
Sources: IDC: 2012 “Worldwide Big Data Technology and Services Forecast: 2011-2015, Gartner: 2012 “Big Data Drives Rapid Changes in Infrastructure and $232 Billion in IT Spending Through 2016
Finance
• Fraud detection
• Anti-money laundering
• Risk management
Energy
• Weather forecasting
• Natural resource exploration
Leaders don’t make compromises
Use case: Smart / Safe Cities
• Deployment Environment – Haven• Ingest data from 2,000+ CCTV cameras in Auckland
NZ• View network of road and environmental sensors• Social media trending, broadcast monitoring, and
real time web news
• Phase 1: HP IDOL scene analysis and license plate recognition
• Phase 2: HP Vertica to uncover breaking trends and facilitate incident responses
• HP IDOL eduction sends interesting data to HP Vertica for statistical analysis and slice/dice
• Combine HP Vertica’s pattern-matching and graph-analysis at scale with HP IDOL’s ability to model concepts and enrich data
Improving public safety by detecting & investigating high-risk events & threats
Award* winning video analytics powered by HPE IDOL 2014 Best Video Analytics - Security Industry Association
Supercell
Challenge– Adopt real-time gaming data analytics platform
Solution– HPE Vertica Analytics Platform
Result– Queries reduced from two to four hours to minutes or
seconds– Solution successfully met cloud deployment requirement– Expanded data capacity from a few months to whole
lifetime of data– Analytics improved customer service by augmenting
player support
Mobile gaming company leads with creativity supported by data
Leveraging Big Data to make games more fun and social
Zynga
• World’s leading social game provider
• And growing rapidly web and mobile 3rd party games on the Zynga Platform
Three key metrics that drive the economics of social gaming
The Challenge
Churn– In this context, churn is defined as the loss rate of game players. Social gaming, because of its very
nature, can have an extraordinarily high churn rate. Typical estimates are that, on average, social games have a churn rate of 50% per month – meaning that half of the new players signing up today will be gone in a month.
Viral Coefficient– The viral coefficient is a measure of how effective current game players are at drawing New players – a
key capability enabled by social network platforms. For example, if 100 Mafia Wars users are likely to cause five of their friends to sign up in a given month, that would be a viral coefficient of 1.05.
Revenue Per User– Finally, there is expected revenue per user. This is an estimate of the lifetime revenue that a game
player will generate, based on an estimate of monthly revenue per user and the churn. For example, if the average monthly revenue is $5 per user, and churn is 50%, the expected revenue can be estimated as ($5 (the first month) + $2.50 (the second month) + $1.25, etc.) or approximately $10.
Time to get Smart!
Vertica Solution
Zynga by Numbers
Users
• ~260 million MAUs
• ~60 million avg DAUs worldwide
Game Data
• Vertica driven• ~60 billion rows/day • ~10TB daily semi-
structured data • ~1.5PB source data • Largest 230 2U nodes
Server Data
• Splunk • 13TB per day raw logs
from server and app logs
• Vertica or Hadoop for archives
HPE Vertica @ Zynga
Solution
–Graph Analysis–Social Games have different social “graph” than social network platform itself.
–Improving these interactions by guiding players to communicate appropriately with these two different types of relationships helps to increase revenue, reduce churn, and increase virality. In other words, to make every aspect of the game more profitable by improving the player experience significantly.
The first thing the Zynga team did was evaluate graph engines (dedicated software for graph analysis), however, none of the solutions they evaluated would operate at the necessary scale or performance. They quickly realized that Vertica would meet their needs, in the words of Dan McCaffrey – Director of Analytics Engineering at Zynga, “Vertica has MPP and scale solved. We can process data daily to produce an optimized graph.”
Why Partner with HPE Big Data Platform
• Technology based on land-and-expand strategy
• Industry-leading technologies platforms
• Value-Adds
• Services e.g. consultancy, development, data scientist
• Additional tools – Tableau, Talend, Logi Analytics …
• Hardware, hosted and/or managed services
• Comprehensive Enablement (sales/technical training & resources)
Thank youMichael Flower: [email protected]