big data use cases
DESCRIPTION
Everyone is awash in the new buzzword, Big Data, and it seems as if you can’t escape it wherever you go. But there are real companies with real use cases creating real value for their businesses by using big data. This talk will discuss some of the more compelling current or recent projects, their architecture & systems used, and successful outcomes.TRANSCRIPT
![Page 1: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/1.jpg)
1
Big Data Use Cases*
DevNexus Conference2/18/2013
*Fully buzzword-compliant title
![Page 2: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/2.jpg)
2
whoami• Brad Anderson• Solutions Architect at MapR (Atlanta)• ATLHUG co-chair• NoSQL East Conference 2009• “boorad” most places (twitter, github)• [email protected]
![Page 3: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/3.jpg)
3
Service Bureau
Client/Server
Application Service Provider
Cloud
B2B
Software-as-a-Service
Virtualization
Social Media
Mobile
Web 2.0
![Page 4: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/4.jpg)
4
BIG DATA
![Page 5: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/5.jpg)
5
![Page 6: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/6.jpg)
6
Business Value
![Page 7: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/7.jpg)
7
Business Value
![Page 8: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/8.jpg)
8
Big Data is not new!but the tools are.
![Page 9: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/9.jpg)
9
Ship the Function to the Data
SAN/NAS
data data data
data data data
data data data
data data data
data data data
function
RDBMS
Traditional Architecture
data
function
data
function
data
function
data
function
data
function
data
function
data
function
data
function
data
function
data
function
data
function
data
function
Distributed Computing
![Page 10: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/10.jpg)
10
Variation: Multiple MapReducesExample: Fraud Detection in User Transactions
LDA training
Transaction data
LDA scoring
HBase /MapR M7 Edition
G2 score
Candidate events for analyst review
95 %-ile LDA anomaly
MapReduce
http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation
![Page 11: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/11.jpg)
11
MapR Distribution for Apache Hadoop
Complete Hadoop distribution
Comprehensive management suite
Industry-standard interfaces
Enterprise-grade dependability
Higher performance
Pig
Hive
HBase
Mahout
Oozie
Whirr
Map Reduce
Cascading
Nagios
Ganglia
MapR Control System
MapR Data Platform
MapR Control System
MapR Data Platform
Flume
Sqoop
HCatalog
Zookeeper
Avro
Map
Reduc
e
![Page 12: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/12.jpg)
12
Big Data Ecosystem
![Page 13: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/13.jpg)
13
Use Case Company Data Source(s) Technique(s) Business Value
![Page 14: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/14.jpg)
14
Proactive Monitoring
![Page 15: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/15.jpg)
15
Server Telemetry Monitoring Logs Network Flow
Data Sources
![Page 16: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/16.jpg)
16
Pattern Recognition Proactive Monitoring Early Alert Delivery
Techniques
![Page 17: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/17.jpg)
17
Business Value
![Page 18: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/18.jpg)
18
Telecommunications Giant
ETL Offload
![Page 19: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/19.jpg)
19
Customer Records Contract Data Purchase Orders Call Center
Data SourcesTelecommunications
![Page 20: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/20.jpg)
20
Techniques
AnalyticsETL
Telecommunications
![Page 21: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/21.jpg)
21
Techniques
+
ETL (Hadoop) Analytics (Teradata)
Telecommunications
![Page 22: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/22.jpg)
22
Business ValueTelecommunications
![Page 23: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/23.jpg)
23
Customer Purchase History Merchant Designations Merchant Special Offers
Data Sources
Credit CardIssuer
![Page 24: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/24.jpg)
24
Techniques
PurchaseHistory
Merchant Information
Merchant Offers
RecommendationEngine Results
(Mahout)
PresentationData Store
(DB2)
App
App
App
App
App
Hadoop Export(4 hrs)
Import(4 hrs)
Credit CardIssuer
![Page 25: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/25.jpg)
25
Techniques
PurchaseHistory
Merchant Information
Merchant Offers
RecommendationEngine Results
(Mahout)
RecommendationSearch Index
(Solr)
App
App
App
App
App
Hadoop
IndexUpdate(2 min)
Credit CardIssuer
![Page 26: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/26.jpg)
26
Business Value
Credit CardIssuer
![Page 27: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/27.jpg)
27
Idle Alerts
Waste & Recycling Leader
![Page 28: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/28.jpg)
28
Truck Geolocation Data– 20,000 trucks– 5 sec interval
Landfill Geographic Boundaries
Data Sources
![Page 29: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/29.jpg)
29
Techniques
TruckGeolocation
Data
Realtime Stream Computation(Storm)
Batch Computation(MapReduce)
ImmediateAlerts
Tax ReductionReporting
HadoopStorage
Shortest PathGraph Algorithm
Route Optimization
![Page 30: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/30.jpg)
30
Business Value
![Page 31: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/31.jpg)
31
Fraud DetectionData Lake
![Page 32: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/32.jpg)
32
Anti-Money Laundering Consumer Transactions
Data Sources
![Page 33: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/33.jpg)
33
TechniquesAnti-Money Laundering
SystemConsumer Transactions
System
![Page 34: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/34.jpg)
34
Techniques
AML
Consumer Transactions
Data Lake(Hadoop)
Suspicious Events
Latent Dirichlet Allocation,Bayesian Learning Neural Network,
Peer Group Analysis
Analyst
![Page 35: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/35.jpg)
35
Business Value
![Page 36: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/36.jpg)
36
Machine LearningSearch Relevance
DNA Matching
![Page 37: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/37.jpg)
37
Birth, Death, Census, Military, Immigration records
Search Behavior Activity DNA SNP (snips)
Data Sources
![Page 38: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/38.jpg)
38
Techniques Record Linking Search Relevance Clickstream Behavior Security Forensics DNA Matching
![Page 39: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/39.jpg)
39
Business Value
![Page 40: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/40.jpg)
40
Traffic Analytics
![Page 41: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/41.jpg)
41
Inrix Road Segment Data– Avg Speed / minute / segment– Reference Speeds
Road Segment Geolocation Data
Data Sources
![Page 42: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/42.jpg)
42
Techniques Bottleneck Detection Algorithm Time Offset Correlations– Alternate Routes
Predictive Congestion Analysis– Growth & Term Assumptions
![Page 43: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/43.jpg)
43
![Page 44: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/44.jpg)
44
![Page 45: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/45.jpg)
45
Business Value
![Page 46: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/46.jpg)
46
Similar Characteristics Lots of Data Structured, Semi-Structured, Unstructured Varied Systems Interoperating
– Hadoop, Storm, Solr, MPP, Visualizations
Increase Revenue Decrease Costs
![Page 47: Big Data Use Cases](https://reader034.vdocuments.us/reader034/viewer/2022051513/547fb92ab4af9fb2618b4c02/html5/thumbnails/47.jpg)
47
Thank You