hadoop for sysadmins
TRANSCRIPT
A whirlwind tour of hadoop
By Eric Marshall For LOPSA-NJ
an all too brief introduction to the world of big data
Eric Marshall
I work for Airisdata; we’re hiring!
Smallest computer I lost sleep over: Sinclair-Timex Z81 – 1KB of memoryLargest computer I lost sleep over: SGI Altix 4700 – 1 TB of memory
Vocabulary disclaimerJust like your favorite swear word, which can act
like many parts of speech and refer to many a thing; hadoop vocabulary has the same problem
Casually, people refer to hadoop as storage, processing, a programming model(s), clustered machines. The same problem exists for other terms in the lexicon, so ask me when I make less sense than usual.
My plan of attackAn intro: the good, the bad and the ugly at
50,000 ft.2¢ tour of hadoop’s processing - map reduce2¢ tour of hadoop’s storage – hdfsA blitz tour of the rest of the hadoop ecosystem
Why did this happen?Old school –> scale up == larger costlier
monolithic system (or a small cluster there of) i.e. vertical scaling
Different approach – all road lead to scale out
Assume failuresSmart software, cheap hardwareDon’t move data; bring processing to data
The GoodSimple development (when compared to
Message Passing Interface programming )Scale – no shared state, programmer don’t need to know the topology, easy to add hardware
Automatic parallelization and distribution of tasks
Fault toleranceWorks with commodity
hardware Open source!
The Bad Not a silver bullet :( MapReduce is batch data processing the time scale is minutes to hours MapReduce is overly simplify/abstracted – you are stuck with the M/R model and it is hard to work smarter MapReduce is low level compared to high-level languages like SQL Not all work decomposes well into parallelized M/R Open source :)
The Ugly?Welcome to the rest of our talk!
First stop, Map Reduce
Hadoop’s MapReduce Lisp’s map and reduce plus the associative property applied to clusters.
Map()Imagine a number of servers with lists of first
names – What is the most popular name?Box 1-isabella William ava mia Emma Alexander Box 2-Noah NOAH Isabella Isabella emma EmmaBox 3-emma Emma Liam liam mason Isabella
Map() would apply a function to each element independent of order. For example, capitalize each word
(MapReduce is covered in greater detail in Chapter 2 of Tom White’s Hadoop – The Definitive Guide by O’Reilly)
Map()So we would have:
Box 1-Isabella William Ava Mia Emma Alexander Box 2-Noah Noah Isabella Isabella Emma EmmaBox 3-Emma Emma Liam Liam Mason Isabella
Map() could be apply function to make pairs For example, Isabella becomes (Isabella, 1)
Map()So we would have:
Box 1-(Isabella,1) (William,1) (Ava,1) (Mia,1) (Emma,1) (Alexander,1)
Box 2-(Noah,1) (Noah,1) (Isabella,1) (Isabella,1) (Emma,1) (Emma,1)
Box 3-(Emma,1) (Emma,1) (Liam,1) (Liam,1) (Mason,1) (Isabella,1)
Now we are almost ready for the reduce, but first the sort and shuffle
Shuffle/SortSo we would have:
Box 1-(Alexander,1) (Ava,1) (Emma,1) (Emma,1) (Emma,1) (Emma,1) (Emma,1)
Box 2-(Isabella,1) (Isabella,1) (Isabella,1) (Isabella,1)
Box 3-(Liam,1) (Liam,1) (Mason,1) (Mia,1) (Noah,1) (Noah,1) (William,1)
Now for the reduce, our function would sum all the of the 1s, and return name and count
ReduceSo we would have:
Box 1-(Alexander,1) (Ava,1) (Emma,1) (Emma,1) (Emma,1) (Emma,1) (Emma,1)
Box 2-(Isabella,1) (Isabella,1) (Isabella,1) (Isabella,1)Box 3-(Liam,1) (Liam,1) (Mason,1) (Mia,1) (Noah,1)
(Noah,1) (William,1)Now for the reduce, our function would sum all the of the 1s, and return name and count Box 1-(Alexander,1) (Ava,1) (Emma,5) Box 2-(Isabella,4) Box 3-(Liam,2) (Mason,1) (Mia,1) (Noah,2) (William,1)(https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html for similar coded in java )
(This architecture is covered in greater detail in Chapter 4 of Tom White’s Hadoop – The Definitive Guide by O’Reilly)
Map/reduce failuresCheck the job if: The job throws an uncaught exception. The job exits with a nonzero exit code. The job fails to report progress to the tasktracker for a configurable amount of time. (i.e. hung, stuck, slow)Check the node if: the same node keeps killing jobs…check the nodeCheck the Job tracker/RM if: jobs are lost or stuck and then they all fail
Instant MR testUm, is the system working?yarn jar /usr/hdp/2.3.2.0-2950/hadoop-
mapreduce/hadoop-mapreduce-examples.jar pi 10 100
(your jar most likely will be somewhere else)
(HDFS is covered in greater detail in Chapter 3 of Tom White’s Hadoop – The Definitive Guide by O’Reilly)
There is a HDFS CLIYou already know some of the commands:
hdfs dfs –ls /hdfs dfs –du /hdfs dfs –rm /hdfs dfs –cat /
There are other modes than dfs: dfsadmin, namenode, datanode, fsck, zkfc, balancer, etc.
HDFS failures Jobs fail: due to missing
blocks Jobs fail: due to moving
data due to down datanodes or huge ingest
Without NN HA – single point of failure for everything
Regular file system mayhem that you already know and love
plus the usual perms issues
HDFS failures Jobs fail: due to missing
blocks Jobs fail: due to moving
data due to down datanodes or huge ingest
Without NN HA – single point of failure for everything
Regular file system mayhem that you already know and love
plus the usual perms issues
The rest of the gardenDistributed Filesystems
- Apache HDFS
outliers:
- Tachyon
- Apache GridGain
- Ignite
- XtreemFS
- Ceph Filesystem
- Red Hat GlusterFS
- Quantcast File System QFS
- Lustre
Security
outliers:
- Apache Sentry
- Apache Knox Gateway
- Apache Ranger
Distributed Programming
- Apache MapReduce also MRv2/YARN
- Apache Pig
outliers:
- JAQL
- Apache Spark
- Apache Flink (formerly Stratosphere)
- Netflix PigPen
- AMPLab SIMR
- Facebook Corona
- Apache Twill
- Damballa Parkour
- Apache Hama
- Datasalt Pangool
- Apache Tez
- Apache Llama
- Apache DataFu
- Pydoop
- Kangaroo
- TinkerPop
- Pachyderm MapReduce
NewSQL Databases
outliers:
- TokuDB
- HandlerSocket
- Akiban Server
- Drizzle
- Haeinsa
- SenseiDB
- Sky
- BayesDB
- InfluxDB
NoSQL Databases
:Columnal Data Model
- Apache HBase
outliers:
- Apache Accumulo
- Hypertable
- HP Vertica
:Key Value Data Model
- Apache Cassandra
- Riak
- Redis
- Linkedin Volemort
:Document Data Model
outliers:
- MongoDB
- RethinkDB
- ArangoDB
- CouchDB
:Stream Data Model
outliers:
- EventStore
:Key-Value Data Model
outliers:
- Redis DataBase
- Linkedin Voldemort
- RocksDB
- OpenTSDB
:Graph Data Model
outliers:
- Neo4j
- ArangoDB
- TitanDB
- OrientDB
- Intel GraphBuilder
- Giraph
- Pegasus
- Apache Spark
Scheduling
- Apache Oozie
outliers:
- Linkedin Azkaban
- Spotify Luigi
- Apache Falco
10 in 10 minutes! Easier Programming: Pig, Spark SQL-like tools: Hive, Impala, Hbase Data pipefitting: Sqoop, Flume, Kafka Book keeping: Oozie, Zookeeper
Easier Programming
PigWhat is it: a high level programming language for data manipulation that abstracts M/R from YahooWhy: a few lines of code to munge dataExample: filtered_words = FILTER words BY word MATCHES '\\w+';
word_groups = GROUP filtered_words BY word;
word_count = FOREACH word_groups GENERATE COUNT(filtered_words) AS count, group AS word;
(Pig is covered in greater detail in Alan Gate’s Programming Pig by O’ReillyAnd in Chapter 16 of Tom White’s Hadoop – The Definitive Guide by O’Reilly)
SparkWhat is it: computing framework from ampLab, UC BerkeleyWhy: high level abstractions and better use of memory Neat trick: in-memory RDDs Example:
scala> val linesWithSpark = textFile.filter(line => line.contains("Spark"))
Or, in python:
>>> linesWithSpark = textFile.filter(lambda line: "Spark" in line) (Spark is covered in greater detail by Matei Zaharia et al. in Learning Spark by O’ReillyAlso of note is Advanced Analytics with Spark – it shows Spark’s capabilities well but moves way too quick to be truly useful. It is covered in Chapter 19 of Tom White’s Hadoop – The Definitive Guide by O’Reilly – lastest ed. Only)
SQL-ish
Hive/HQLWhat is it: a data infrastructure and query language from FacebookWhy: batched SQL queries against HDFS Neat trick: stores metadata so you don’t have toExample: hive> LOAD DATA INPATH ‘/user/work/input/BX-BooksCorrected.csv’ OVERWRITE INTO TABLE BXDataSet;
hive> select yearofpublication, count(booktitle) from bxdataset group by yearofpublication;
(Hive is covered in greater detail by Jason Ruthergenlen et al. in Programming HIve by O’Reilly. Instant Apache Hive Essentials How-To by Darren Lee by Packt was useful to me as tutorial.It is also covered in Chapter 17 of Tom White’s Hadoop – The Definitive Guide by O’Reilly)
ImpalaWhat is it: SQL query engine from ClouderaWhy: fast adhoc queries on subsets of data stored in hadoopExample:
[impala-host:21000] > select count(*) from customer_address;
(nada, let me know if you hit pay dirt)
HBaseWhat is it: a non-relational database from PowersetWhy: fast access to large sparse data setsExample:
hbase(main):001:0> create 'test', 'cf'0 row(s) in 0.4170 seconds
Hbase::Table – testhbase(main):003:0> put 'test', 'row1', 'cf:a', 'value1'0 row(s) in 0.0850 secondshbase(main):006:0> scan 'test'ROW COLUMN+CELL row1 column=cf:a, timestamp=1421762485768, value=value1
(HBase is covered in Chapter 20 of Tom White’s Hadoop – The Definitive Guide by O’ReillyAnd in covered in greater detail in Lars George’s HBase – The Definitive Guide by O’Reilly)
Data pipefitting
SqoopWhat is it: glue tool for moving data between relational databases and hadoopWhy: make the cumbersome easierExample:
sqoop list-databases --connect jdbc:mysql://mysql/employees –username joe --password myPassword
(HBase is covered in greater detail in Chapter 16 of Tom White’s Hadoop – The Definitive Guide by O’ReillyThere is also a cookbook that covered a few worthy gotchas: Apache Sqoop Cookbook Kathleen Ting by O’Reilly)
FlumeWhat is it: a service for collecting and aggregating logs Why: because log ingestion is tougher than it seemsExample:
# Define a memory channel on agent called memory-channel.agent.channels.memory-channel.type = memory
# Define a source on agent and connect to channel memory-channel.agent.sources.tail-source.type = execagent.sources.tail-source.command = tail -F /var/log/system.logagent.sources.tail-source.channels = memory-channel
# Define a sink that outputs to logger.agent.sinks.log-sink.channel = memory-channelagent.sinks.log-sink.type = logger
# Define a sink that outputs to hdfs.agent.sinks.hdfs-sink.channel = memory-channelagent.sinks.hdfs-sink.type = hdfsagent.sinks.hdfs-sink.hdfs.path = hdfs://localhost:54310/tmp/system.log/agent.sinks.hdfs-sink.hdfs.fileType = DataStream
# Finally, activate.agent.channels = memory-channelagent.sources = tail-sourceagent.sinks = log-sink hdfs-sink
(I haven’t read much on Flume; if you find something clever let me know!)
KafkaWhat is it: message broker from LinkedInWhy: fast handling of data feedsNeat trick: no need to worry about missing data or double processing dataExample: > bin/kafka-console-producer.sh --zookeeper localhost:2181 --topic test This is a messageThis is another message> bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginningThis is a messageThis is another message(I disliked the one book I read but I found the online docs very readable! http://kafka.apache.org/
Also check out the design docs http://kafka.apache.org/documentation.html#design )
Book keeping
OozieWhat is it: workflow scheduler from Yahoo BanglaloreWhy: because cron isn’t perfectExample:
oozie job -oozie http://localhost:8080/oozie -config examples/apps/map-reduce/job.properties -run
(Oozie is covered in greater detail in Islam & Srinivasan’s Apache Oozie: The Workflow Scheduler by O’Reilly)
ZookeeperWhat is it: a coordination service from YahooWhy: sync info for distributed systems (similar idea behind DNS or LDAP)Example:
[zkshell: 14] set /zk_test junkcZxid = 5ctime = Fri Jun 05 13:57:06 PDT 2009mZxid = 6mtime = Fri Jun 05 14:01:52 PDT 2009pZxid = 5
[zkshell: 15] get /zk_testjunkcZxid = 5ctime = Fri Jun 05 13:57:06 PDT 2009mZxid = 6mtime = Fri Jun 05 14:01:52 PDT 2009pZxid = 5
(Zookeeper is covered in greater detail in Zookeeper: Distributed Process Cooridination by O’ReillyAnd in Chapter 21 of Tom White’s Hadoop – The Definitive Guide by O’Reilly)
Distributed Programming- Apache MapReduce also MRv2/YARN
- Apache Pig
outliers:
- JAQL
- Apache Spark
- Apache Flink (formerly Stratosphere)
- Netflix PigPen
- AMPLab SIMR
- Facebook Corona
- Apache Twill
- Damballa Parkour
- Apache Hama
- Datasalt Pangool
- Apache Tez
- Apache Llama
- Apache DataFu
- Pydoop
- Kangaroo
- TinkerPop
- Pachyderm MapReduce
Distributed Filesystems- Apache HDFS
outliers:
- Tachyon
- Apache GridGain
- Ignite
- XtreemFS
- Ceph Filesystem
- Red Hat GlusterFS
- Quantcast File System QFS
- Lustre
NoSQL Databases:Columnal Data Model
- Apache HBase
outliers:
- Apache Accumulo
- Hypertable
- HP Vertica
:Key Value Data Model
- Apache Cassandra
- Riak
- Redis
- Linkedin Volemort
:Document Data Model
outliers:
- MongoDB
- RethinkDB
- ArangoDB
- CouchDB
:Stream Data Model
outliers:
- EventStore
:Key-Value Data
Model
outliers:
- Redis DataBase
- Linkedin Voldemort
- RocksDB
- OpenTSDB
:Graph Data Model
outliers:
- Neo4j
- ArangoDB
- TitanDB
- OrientDB
- Intel GraphBuilder
- Giraph
- Pegasus
- Apache Spark
NewSQL Databases
outliers:
- TokuDB
- HandlerSocket
- Akiban Server
- Drizzle
- Haeinsa
- SenseiDB
- Sky
- BayesDB
- InfluxDB
Data Ingestion
:SQL on Hadoop
- Apache Hive
- Apache HCatalog
outliers:
- Cloudera Kudu
- Trafodion
- Apache Drill
- Cloudera Impala
- Facebook Presto
- Datasalt Splout SQL
- Apache Spark
- Apache Tajo
- Apache Phoenix
- Apache MRQL
- Kylin
Data Ingestion
- Apache Flume
- Apache Sqoop
outliers:
- Facebook Scribe
- Apache Chukwa
- Apache Storm
- Apache Kafka
- Netflix Suro
- Apache Samza
- Cloudera Morphline
- HIHO
- Apache NiFi
Etc.
Service Programming and Frameworks
- Apache Zookeeper
- Apache Avro
- Apache Parquet
outliers:
- Apache Thrift
- Apache Curator
- Apache Karaf
- Twitter Elephant Bird
- Linkedin Norbert
Scheduling
- Apache Oozie
outliers:
- Linkedin Azkaban
- Spotify Luigi
- Apache Falcon
- Schedoscope
Security
outliers:
- Apache Sentry
- Apache Knox Gateway
- Apache Ranger
System Deployment
and Management
outliers:
- Apache Ambari
- Cloudera Manager
- Cloudera HUE
- Apache Whirr
- Apache Mesos
- Myriad
- Marathon
- Brooklyn
- Hortonworks HOYA
- Apache Helix
- Apache Bigtop
- Buildoop
- Deploop
And now a bit of common sense for sys-admin-ing Hadoop clusters
AvoidThe usual -
Don’t let hdfs fill up Don’t use all the memoryDon’t use up all the cpusDon’t drop the network<insert fav disaster>
Resource Exhaustion by usersHardware Failure (drives are the king of this
domain)
Um, backups?Usual suspects plus Namenode’s meta data!! (fsimage)Hdfs? Well, it would nice but unlikely (if so
distcp)Snapshots
Hadoop Management Apache Ambari
Cloudera Manager
MonitoringThe usual suspects plus… JMX support Jvm via jstat, jmap etc.hdfsMapredconf/hadoop-metrics.propertieshttp://namenode:50070/http://namenode:50070/jmx
User managementHdfs quotasAccess controls
Internal andexternal
MR schedulersFifo, Fair, Capacity
Kerberos can be used as well
Configuration /etc/hadoop/confLots of knobs! !Ojo! –
Lots of overrides Get the basic system solid before security and
performance Watch the units – some are in megabytes but some are
in bytes! Have canary jobs Ensure same configs are everywhere (including uniform
dns/host)
Want more?
(Disclaimer: I receive nothing from O’Reilly. Not even a Christmas card…)
FinThanks for listeningSlides:
http://www.slideshare.net/ericwilliammarshall/hadoop-for-sysadmins
Any questions?
What’s in a name?Doug Cutting seems to have been inspired by
his family. Lucene is his wife’s middle name, and her maternal grandmother’s first name. His son, as a toddler, used Nutch as the all-purpose word for meal and later named a yellow stuffed elephant Hadoop. Doug said he “was looking for a name that wasn’t already a web domain and wasn’t trademarked, so I tried various words that were in my life but not used by anybody else. Kids are pretty good at making up words.”
What to do?Combinations of the usual stuff:Numerical SummarizationsFiltering Altering Data OrganizationJoining DataI/O
federation
(Image from Chapter 2 of Eric Sammer’s Hadoop Operations by O’Reilly)