introducon to data science with hadoop - dataedge 2018 · introducon to data science with hadoop...

1/59©Cloudera,Inc.Allrightsreserved.

Introduc;ontoDataSciencewithHadoopGlynnDurham|[email protected]


ShortandSweetHadoopWhatAboutSpark?MachineLearningTheFuture


DataScienceis…

• gatheringdata,poten;allyofmanytypesandfrommanysources,

• wranglingthatdataintousefulforms,and

• applyingsta;s;calprogrammingandmachinelearning,togainnewinforma;onfromthedata.


MachineLearningandDataVolume

“It’snotwhohasthebestalgorithmswhowins.It’swhohasthemostdata.”[BankoandBrill,2001]


Hadoopis…

• anopensourceso\warepla]ormfor• acquiring,storing,andprocessingmassivevolumesofdata,• economically.


TheAgeofMachineLearning


Theword“Hadoop”means

• achild’stoyor

• HadoopCoreor

• theHadoopEcosystem.


HadoopCore

• Afreeopensourceso\wareso\wareproject• Managedtransparentlyonline,attheApacheSo\wareFounda;on(ASF),apache.org

• Theprojectwasstartedin2006,basedonpapersfromGoogle,in2003and2004

• Consistsof:• HDFS(HadoopDistributedFileSystem),forstorage• HadoopMapReduce,forprocessing• YARN(YetAnotherResourceNego;ator)

hadoop.apache.org


HadoopCoremainfeatures:Filestorageandbatchprogramming


HDFSWrites


HDFSReads


GeneralFileInput/Output


HDFSStrengthsandWeaknesses• HDFSisgoodat:

• storingenormousfiles• storinglotsofdatareliably•  throughputonsequen;alwrites•  throughputonsequen;alreadsofafileorpartofafile

• HDFSisnotgoodat:• highspeed(lowlatency)randomreadsofpartsofafile

• HDFScannot:• updateanypartofafileoncewrijen**butyoucanalwayswriteanewfileand/ordelete,move,andrenamefilesanddirectories


MapReduce:Programmingwithsimplefunc;ons


MapReduceExample:WordCountCountthenumberofoccurrencesofeachwordoveralargeamountofinputdata• Thisisthe‘helloworld’ofMapReduceprogramming

map(String input_key, String input_value) foreach word w in input_value: emit(w, 1)

reduce(String output_key, Iterator<int> intermediate_vals) set count = 0 foreach v in intermediate_vals: count += v emit(output_key, count)


WordCount,con;nuedInputtotheMapper:

OutputfromtheMapper:

(3414, 'the cat sat on the mat') (3437, 'the aardvark sat on the sofa')

('the', 1), ('cat', 1), ('sat', 1), ('on', 1), ('the', 1), ('mat', 1), ('the', 1), ('aardvark', 1), ('sat', 1), ('on', 1), ('the', 1), ('sofa', 1)


WordCount,con;nuedIntermediatedatasenttotheReducer:

FinalReduceroutput:

('aardvark', [1]) ('cat', [1]) ('mat', [1]) ('on', [1, 1]) ('sat', [1, 1]) ('sofa', [1]) ('the', [1, 1, 1, 1])

('aardvark', 1) ('cat', 1) ('mat', 1) ('on', 2) ('sat', 2) ('sofa', 1) ('the', 4)


Sowejustcountedwords.Sowhat?

• Manyproblemsconformtothispajern:• Webloganalysis:map()emitsanIPaddressforeachweblogevent;reduce()countsoccurrencesforeachIPaddress

•  Indexing:Foreachdocument,map()emitseachtermofinterestpairedwiththedocumentID;reduce()collectsandemitsalldocumentIDsforeachterm

• Pagerankalgorithm:• Everywebpage(URL)ontheWebgetsanini;alscore.• map()dividesapage’sscoreamongallofitsoutlinks’URLs;reduce()sumsthereceivedscoresforeachURL.

•  Iterateonthisprocedure.


MapReduceChains


MapReduceatScale


MapReduceStrengthsandWeaknesses• MapReduceisgoodat:

• processingenormousvolumesofdata• scalingoutasyouaddmoremachines• con;nuingtocomple;on,evenwhensomemachinesdie

• MapReduceisnotgoodat:•  runninganyalgorithmyoucanwriteinpseudocode• algorithmsthatrequiresharedstateoverall**butmaybeyoucangetcleverwithyouralgorithmdesign

• MapReducecannot:•  runinreal;me:MapReducejobsarebatchjobs


YARN,YetAnotherResourceNego;ator


Sqoop:RDBMStoHadoopandBack

• UsesMapReducetorunconcurrentdatabasequeriesthatextractorinsertdatasqoop.apache.org


Flume:Inges;ngOngoingEventDataflume.apache.org


Kata:GeneralDataStreamingkata.apache.org


HBase:ANoSQLDatabaseSystem

• Ascalablekey/valuestore• Accommodatesgeneralbinarydata• Highvolume,highperformanceaccesstoindividualitems• Randomreadsandwrites• WeakerquerylanguagethanSQL(put,get,scan,delete)•  LacksACID-complianttransac;ons

hbase.apache.org


Kudu:Scalablestorageforstructureddatakudu.apache.org


Hive:MapReduce(orSpark)as“SQL”

•  Familiarlanguageandprogrammingparadigm

• ProvidesinterfacetomanySQL-complianttools

hive.apache.org


Pig:AnotherLanguageforMapReduce(orSpark)pig.apache.org


Impala:HighSpeedAnaly;csinHadoop

• Purpose-builtforhighspeedanaly;cqueries

• DoesnotuseMapReduceorSpark

• Usually5to30;mesfaster—some;mes100;mesfaster!

incubator.apache.org/projects/impala.html


AndMore

•  Serializa;onandefficientfilestorage:AvroandParquet

• Workflow:Oozie

avro.apache.org parquet.apache.org

oozie.apache.org


AndEvenMore…

•  Security:SentryandRecordService

• MachineLearninginMapReduce:Mahout

• And…mahout.apache.org

sentry.apache.org recordservice.io


Spark:AnImprovementonMapReduce

• OriginallyaresearchprojectatUCBerkeleyRADLab—latertheAMPLab,in2009

• AddressessomefundamentalpainpointsofMapReduce

• TheSparkStreamingsubprojectof2012addsnearreal-;meprogramming• using“micro-batches”asanadapta;onofbatchprogramming• acapabilityaltogetherlackinginHadoopMapReduce

spark.apache.org


Similari;esofMapReduceandSpark

• Processesmassivevolumesofdatawithascale-out,distributedframework• Theframeworkprovidesreliability,eveninthefaceofmachinefailure• Programmingwithstatelessfunc;ons• Reliesonexpensiveshuffletoreorganizedataforaggrega;on,joins,sor;ng•  S;lllacksasharedstateamongallprocesses• CanrununderYARNtoshareprocessingresources


ImprovedAPI

•  First-classAPIsinScala,Java,PythonandR• Data-flowprogrammingparadigm(likePig)•  Interac;veshell—greatforexploratorywork

•  ImprovedsupportforstructureddataandSQL-likeprocessing


ProcessingChains,Improved

func;on func;on func;on

func;onfunc;onfunc;on

EliminateI/O

EliminateI/O

ReduceI/O

Tasks,notnewprocesses(JVMs)Enhancedcachinginmemory


SparkMLlib:MachineLearninginSpark

•  SubprojectofSpark• Effec;velyreplacesMahoutformachinelearninginHadoopclusters•  Fromspark.apache.org,thefrontpage:

Butjustbeclearwhatyoumeanby“Hadoop”!


CommercialMessage#1


BigEcosystem

oozie.apache.org


CompleteBigDataPla]orm

• ClouderaManagercan•  install,monitor,manage,upgradeacoherentbundleoftheseprojectsandmore

• ClouderaDirectorcan• easilyconfigureanddeploythispla]ormoncloudservicesfromAmazon,Google,orMicroso\

•  !!!


MachineLearningAlgorithms

•  SupervisedLearning:• Startwithcorrectlylabeledrecords,andlearntoes;mateorpredictlabelsfornewrecords

• Con;nuouslabels:Regression• Discretelabels:Logis;cRegression,Classifiers

• UnsupervisedLearning:• Startwithunlabeledrecords,trytoteasepajerns(labels)outofthedata• Thereisnotasingle“correct”answerforlabeling• Con;nuouslabels:Collabora;veFilters(Recommenders)• Discretelabels:Clustering


LinearRegression:SupervisedLearningofaCon;nuousLabel


Logis;cRegression:SupervisedLearningofaBinaryLabel


Classifiers:SupervisedLearningofDiscreteLabels

Training:Cat

Training:Table

Scoring:???


Collabora;veFilters(Recommenders):UnsupervisedLearningofCon;nuousLabels


Clustering:UnsupervisedLearningofDiscreteLabels


SparkMLlib:MachineLearningonHadoop


CommercialMessage#2


MoreDSTeamsintheOrganiza;on

• Collabora;on,repeatabilitywithinteams• Differingsecurityrequirements• Differentpreferredprograminglanguages:Python,R,Scala• Differentso\warelibraries:Pandas,H2O,etc.• Evendifferentversionsofso\ware


ClouderaDataScienceWorkbench

• DevelopmentinPython,Scala,orR• Differingsecurityrequirements


DeepLearning


DeepLearningonHadoop

• DeepLearningreferstoacategoryofclassifieralgorithms,mostlyinventedin2006.

•  SparkMLlibdoesnothaveanydirectimplementa;onofDL.• Thereareseveraladdi;onalprojectsthatcanfitDLontoSpark/Hadoop:

• BigDL• Caffe• TensorFlow• DL4J


TheRoad—orRunway(!)—Ahead

•  Itisatruismthatorganiza;onstodayhavevaluableinsightshiddenintheirdatathatarewai;ngtobeuncovered.

• 90%ofalldatathatwillexistin2020hasyettobecreated.• Opensourceisheretostay.• Hadoopasadatasciencepla]ormisevolving,anditsuseisgrowingexponen;ally.


[email protected]

introducon to data science with hadoop - dataedge 2018 · introducon to data science with hadoop...

Documents