welcome to the age of data
DESCRIPTION
An introductory presentation on Big Data and Hadoop for bigdate.be - presented 11/Jan/2012 at Accenture (Brussels).TRANSCRIPT
![Page 1: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/1.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
Welcome to the age of data!BIGDATA.BE
![Page 2: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/2.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
who am i
» Steven Noels» Founder & VP Product» Makers of Lily: Interactive Big Data
platform» Open Source / Apache Software
Foundation» co-founder bigdata.be
2
![Page 3: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/3.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
Houston,we have
a problem.
![Page 4: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/4.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
We’re drowning.
![Page 5: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/5.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
Drowningin aSeaofData.
![Page 6: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/6.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
Mountains of Metadata.
![Page 7: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/7.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
The firehoseof UGC.
![Page 8: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/8.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
Still, we can’t make
much sense of it.
![Page 9: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/9.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
... and we throw a lot of
it away.
![Page 10: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/10.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
We regard DATA as cost.
![Page 11: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/11.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
But data is an opportunity.
![Page 12: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/12.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
Think about it.
![Page 13: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/13.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 13
advertisements
![Page 14: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/14.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 14
recommendations
![Page 15: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/15.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 15
fraud detection
![Page 16: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/16.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 16
eyeballs
![Page 17: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/17.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 17
churn
![Page 18: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/18.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
The future is fordatanerds.
![Page 19: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/19.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
This is what Big Data is about: new insights, new business.
![Page 20: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/20.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
3 issues forBIG DATA
![Page 21: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/21.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 21
moore
data
need:more
capacity
time
volu
me 1
![Page 22: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/22.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 22
solution:distributed
systems 1
![Page 23: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/23.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 23
1
![Page 24: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/24.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
distributed systems are hard.
1
![Page 25: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/25.jpg)
2IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 25
database
![Page 26: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/26.jpg)
2IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 25
database data warehouse
![Page 27: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/27.jpg)
2IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 25
database data warehouse analytics
![Page 28: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/28.jpg)
2IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 25
database data warehouse analytics
data shuffling, data duplication
![Page 29: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/29.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 26
“Top-performing organizations are twice as likely to apply analytics to activities.”
(MIT Sloan Management Review, Winter 2011)
3
![Page 30: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/30.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 27
enter
![Page 31: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/31.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 28
![Page 32: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/32.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 28
HBase
![Page 33: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/33.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 29
Disk
CPU
RAM1
serv
er
what is hadoop ?
![Page 34: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/34.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 30
RAM
CPU
DISK HDFS
MAP/REDUCE
HBASE
many servers
![Page 35: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/35.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 31
map/reduce
![Page 36: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/36.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
map/reduce
32
» Batch-oriented» Data locality (code is shipped around)» Heavy parallellization» Process management» Append-only files
![Page 37: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/37.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 33
Hadoop ecosystem
» Hadoop Common» Subprojects» Flume/SQOOP: Data collection systems
for large distributed systems.» HBase: A scalable, distributed database
that supports structured data storage for large/wide tables.
» HDFS: A distributed file system that provides high throughput access to application data.
» Hive: A data warehouse infrastructure that provides data summarization and ad hoc querying.
» MapReduce: A software framework for distributed processing of large data sets on compute clusters.
» Pig: A high-level data-flow language and execution framework for parallel computation.
» ZooKeeper: A high-performance coordination service for distributed applications.
» Mahout: machine learning libraries
![Page 38: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/38.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 34
(PIG, HIVE)
Coordination(ZOOKEEPER)
DataIntegration
(FLUME, SQOOP)
FastRead/Write
Access(HBASE)
Languages / Compilers(PIG, HIVE)
Workflow(OOZIE)
Scheduling(oozie)
Metadata(HIVE)
UI Framework(HUE)
SDK(HUE SDK)
High-level data model / easy API indexes
Search
CDH
Dev2Dev tutoring,
integrateddeployment
andenterprise
supportusage metrics, analytics & recommen-
dations
![Page 39: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/39.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 35
speed layer
serving layer
batch layer
real-time big data architecture
1. store master dataset (append-only)2. compute arbitrary views
1. random access to batch views2. updated by batch layer
1. compensate for high latency of updates to serving layer2. fast, incremental algorithms3. batch layer eventually overrides speed layerstorm
![Page 40: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/40.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 36
Hadoop, interactive.
Analytics Interacticsbatchstatic files
1018
interactivedata management
1015
(RDBMS)
109-12
![Page 41: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/41.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 37
data managementindexing
search
profileharvesting
smartinsights
interactiveaudiencemetrics
news & media
telecom
finan
ce
com
mer
ce
My baby: Lily.
![Page 42: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/42.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 38
The start of Lily.
![Page 43: Welcome to the Age of Data](https://reader035.vdocuments.us/reader035/viewer/2022081414/54b794354a7959db528b4b36/html5/thumbnails/43.jpg)
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org
Thank you !for your attentionfor your questions
» @stevenn