Revolution Confidential
T he R is e of Data S c ienc e in the age of B ig Data A nalytic sWhy Data Dis tillation and Mac hine L earning A ren’t E nough
David M S mithV P Marketing and C ommunityR evolution Analytic s
Revolution ConfidentialToday, we’ll dis c us s :
What is Data Science? Why machine learning isn’t enough Why Data Science works The Data Scientists Toolkit The Future of Big Data Analytics Closing thoughts and resources
2
Revolution Confidential
3© Dov Harrington, CC By-2.0http://www.flickr.com/photos/idovermani/4110546683/
Revolution ConfidentialWhere is it s afe to fis h near S an F ranc is co?
4San Francisco Estuary Institutehttp://www.sfei.org/tools/wqt
Revolution ConfidentialHurric ane S andy
Bob Rudishttp://rud.is/b/2012/10/28/watch-sandy-in-r-including-forecast-cone/
5
Revolution ConfidentialHurric ane S andy
Ed Chenhttp://blog.echen.me/hurricane-sandy-outages/
6
Revolution Confidential
When did Michael J acks on have his bigges t hits ?
New York Times, June 25 2009 (3 hours after Michael Jackson’s death)http://www.nytimes.com/interactive/2009/06/25/arts/0625-jackson-graphic.html 7
Revolution ConfidentialT hree E s s ential S kills of Data S c ientis ts
8Drew Conwayhttp://www.dataists.com/2010/09/the-data-science-venn-diagram/
Data IntegrationMashups
Applications
ModelsVisualizationPredictionsUncertainty
ProblemsData Sources
Credibility
EffectiveData
Applications
Revolution Confidential
9Image © Abode of Chaos, CC BY 2.0http://www.flickr.com/photos/home_of_chaos/6418989233/
Revolution ConfidentialMac hine learning (ML ) for predic tions
10
Res
pons
e
Feat
ures
Res
pons
es
MLscoring rules
Building the Model
Valid
atio
n se
t
Pre
dict
ions
scoring rules
Validating the Model
New
Dat
a
Pre
dict
ions
(sco
res)
scoring rules
Scoring new data
“Accuracy”
Revolution ConfidentialP roblem: A lac k of pers pec tive
11Image © 2010 David M Smith. Some rights reserved CC BY-2.0
Revolution ConfidentialP roblem: L ac k of c redibility
12
Revolution ConfidentialP roblem: C omplexity
13
Revolution ConfidentialData Science to the Rescue!
14
Revolution ConfidentialA ns wer Unas ked Ques tions
15Revolutions blog: “The Uncanny Valley of Big Data”http://blog.revolutionanalytics.com/2012/02/the-uncanny-valley-of-big-data.html
Revolution Confidential
16
“More data beats better algorithms, every time” – Google
“Companies that have massive amounts of data without massive amounts
of clue are going to be displaced by startups that have less data but more
clue.” -- Tim O’Reilly
Google Research, “The Unreasonable Effectiveness of Data”: http://googleresearch.blogspot.com/2009/03/unreasonable-effectiveness-of-data.html
Tim O’Reilly on Google+: https://plus.google.com/107033731246200681024/posts/4Xa76AtxYwdTechnoCalifornia: http://technocalifornia.blogspot.com/2012/07/more-data-or-better-models.html
F ill in knowledge gaps
Revolution ConfidentialAvoid ineffec tive reac tions
17Stupid Data Miner Trickshttp://nerdsonwallstreet.typepad.com/my_weblog/files/dataminejune_2000.pdf
S&P
500
Revolution Confidential
18© Henricks Photos CC-BY-ND 2.0http://www.flickr.com/photos/hendricksphotos/3240667626/
Revolution Confidential0. Data (B ig & Mes s y)
19
Revolution Confidential1. A language for programming with data
20
Download the White Paper
R is Hotbit.ly/r-is-hot
Revolution Confidential
21
Grant awards to homeless veterans FY09Data: Data.govAnalysis: Drew Conway
User-defined functions
Internet API interfaceXML parsing
Custom graphics
Data import and pre-processing
Iterative data processing
Revolution Confidential2. S peed. L ots and lots of s peed.
22
Variable Transformation
Model Estimation
Model Refinement
Model Comparison / Benkmarking
Feature SelectionSampling
AggregationData Predictions
Revolution Confidential
Core 0(Thread 0)
Core n(Thread n)
Core 2(Thread 2)
Core 1(Thread 1)
Multicore Processor (4, 8, 16+ cores)
DataData Data
Disk
Shared Memory
Us e all available c omputing c yc les
23
Revolution Confidential
Compute Node
Compute Node
Master Node
DataPartition
DataPartition
Compute Node
Compute Node
DataPartition
DataPartition
3. A lgorithms that don’t choke on B ig Data
PEMAs: Parallel External-Memory Algorithms24
BIGDATA
Revolution ConfidentialDrink les s c offee!
25
Single ThreadedNon-optimized
algorithms
OptimizedParallelizedAlgorithms
Revolution Confidential4. Move c ode to data (not vic e vers a)
26
Map-Reduce
RHadoop: http://bit.ly/RHadoop
Revolution ConfidentialB ig Data A pplianc es
27
More info: http://bit.ly/R-Netezza
Revolution ConfidentialP lay Nic e with Others
• Business Intelligence Tools• Web-based data apps• Reporting / Spreadsheets
Presentation Layer
• R
Analytics Layer
• Relational datastores• Unstructured datastores
Data Layer
28
Revolution ConfidentialWhat every data s c ientis t needs
Open-Source RRevolution R
EnterpriseInterface with multiple data sources
Exploratory data analysis
Wide range of statistical methods
High-speed computation
Big Data support
Data/code locality (Hadoop, etc.)
Print-quality data visualization
Scheduled batch production
Works in a multi-tool ecosystem
Integration into Data Apps
29
Revolution ConfidentialR evolution R E nterpris e: B ig-Data R
Open-Source RRevolution R
EnterpriseInterface with multiple data sources
Exploratory data analysis
Wide range of statistical methods
High-speed computation
Big Data support
Data/code locality (Hadoop, etc.)
Print-quality data visualization
Scheduled batch production
Works in a multi-tool ecosystem
Integration into Data Apps
30www.revolutionanalytics.com/products
Revolution Confidential
31Image © www.tinyplanetphotography.com
Revolution ConfidentialA nd … the future?
Even more data
Cloud computing
Demand for Data Scientists
Diverging paradigms for data analytics
32http://www.indeed.com/jobtrends
Revolution ConfidentialDiverging data paradigms
33
HadoopNoSQL
FilesClusters
Data Appliances
More data, better fault tolerance
Easier programming, better performanceExplorationModeling
StoragePreprocessing
Production
Revolution ConfidentialData S c ienc e in P roduc tion
Real-time Big Data Analytics: From Deployment to Production
Thursday, November 29, 201210:00AM - 11:00AM Pacific Time
www.revolutionanalytics.com/news-events/free-webinars/
34
Revolution ConfidentialB uilding Data S c ienc e Teams
DJ Patil in O’Reilly Radar: http://oreil.ly/I3H5fI
Statistics and Data Science graduates
Kaggle and Chorus
Revolution Analytics R Training: http://www.revolutionanalytics.com/services/training/
35
Revolution ConfidentialC los ing T houghts
Data Science process leads to more powerful, and more useful models
Data Scientists need a technology platform to think about, explore, and model data
Revolution R Enterprise is R for Big Data
36
Revolution ConfidentialR es ourc es
Revolution R Enterprise : R for Big Data www.revolutionanalytics.com/products
Rhadoop : Connecting R and Hadoop bit.ly/r-hadoop
Contact David Smith [email protected] @revodavid blog.revolutionanalytics.com
37
Revolution ConfidentialT hank you.
38
www.revolutionanalytics.com 650.646.9545 Twitter: @RevolutionR
The leading commercial provider of software and support for the popular open source R statistics language.
