a hadoop overview

A Hadoop Overview

Outline

Progress ReportMapReduce ProgrammingHadoop Cluster OverviewHBase OverviewQ & A

Outline

Progress ReportMapReduce ProgrammingHadoop Cluster OverviewHBase OverviewQ & A

Progress

Hadoop buildup has been completed. Version 0.19.0, running under Standalone mode.

HBase buildup has been completed. Version 0.19.3, with no assists of HDFS.

Simple demonstration over MapReduce. Simple word count program.

Testing Platform

Fedora 10JDK1.6.0_18Hadoop-0.19.0Hbase-0.19.3One can connect to the machine using pietty

or putty. Host: 140.112.90.180 Account: labuser Password: robot3233 Port: 3385 (using ssh connection)

Outline

Progress ReportMapReduce ProgrammingHadoop Cluster OverviewHBase OverviewQ & A

MapReduce

A computing framework including map phase, shuffling phase and reduce phase.

Map function and Reduce function are provided by the user.

Key-Value Pair(KVP) map is initiated with each KVP ingested, and output

any number of KVPs. reduce is initiated with each key and its

corresponding values, and output any number of KVPs.

MapReduce(cont.)

What user has to do?

1. Specify the input/output format2. Specify the output key/value type3. Specify the input/output location4. Specify the mapper/reducer class5. Specify the number of reduce tasks6. Specify the partitioner class(dicussed later)

What user has to do?(cont.)

Specify the input/output format “Input/Output format” is class that translate raw data

and KVPs. Has to inherit class InputFormat/OutputFormat. Input format is required.

The most common choice is KeyValueTextInputFormat class and SequenceFileInputFormat class.

Output format is selective, the default is TextOutputFormat class .

What user has to do?(cont.)

Specify the output key/value type The KVP type output by reducer.

The Key type has to implements WritableComparable interface.

The Value type has to implements Writable interface.

Specify the input/output location The directory or for input files/output files. The input directory should exist and contain at least

one file. The output directory should not exist or be empty.

What user has to do?(cont.)

Specify the mapper/reducer class The two classes should extend MapReduceBase class. The map/reduce class should implement Mapper<K1,

V1, K2, V2>/Reducer<K1, V1, K2, V2> interface

Specify the number of reduce tasks Usually approximate the number of computing nodes. 1 if we want a single output file. 0 if we don’t need the reduce phase.

Note that we will not have our result sorted. The reducer class is not required in this case.

Map Phase Configuration

Element Required?

Default

Input path(s) YesClass to convert the input path elements to KVPs

Yes

Map output key class No Job output key classMap output value class No Job output value

classClass supplying the map function YesSuggested minimum number of map tasks

No Cluster default

Number of threads to run each map task

No 1

Reduce Phase Configuration

Element Required?

Default

Output path YesClass to convert the KVPs to output files

No TextOutputFormat

Job input key class No Job output key classJob input value class No Job output value

classJob output key class YesJob output value class YesClass supplying the reduce function YesThe number of reduce tasks No Cluster default

MapReduceIntro.javapublic class MapReduceIntro { protected static Logger logger = Logger.getLogger(MapReduceIntro.class); public static void main(final String[] args) { try { final JobConf conf = new JobConf(MapReduceIntro.class); conf.set("hadoop.tmp.dir","/tmp");

conf.setInputFormat(KeyValueTextInputFormat.class); FileInputFormat.setInputPaths(conf, MapReduceIntroConfig.getInputDirectory());

conf.setMapperClass(IdentityMapper.class);

FileOutputFormat.setOutputPath(conf, MapReduceIntroConfig.getOutputDirectory()); conf.setOutputKeyClass(Text.class); conf.setOutputValueClass(Text.class); conf.setNumReduceTasks(1); conf.setReducerClass(IdentityReducer.class); final RunningJob job = JobClient.runJob(conf);

if (!job.isSuccessful()) { logger.error("The job failed."); System.exit(1); }

System.exit(0); }}

Initial Configuration

Map Phase Configuration

Reduce Phase Configuration

Job Running

IdentityMapper.java

public class IdentityMapper<K, V> extends MapReduceBase implements Mapper<K, V, K, V> {

public void map(K key, V val, OutputCollector<K, V> output, Reporter reporter) throws IOException { output.collect(key, val); }}

Input type Output type

Discussed later

Collect output KVPs

IdentityReducer.java

public class IdentityReducer<K, V> extends MapReduceBase implements Reducer<K, V, K, V> {

public void reduce(K key, Iterator<V> values, OutputCollector<K, V> output, Reporter reporter) throws IOException { while (values.hasNext()) { output.collect(key, values.next()); } }}

The input value is an Iterator<V>!

Compiling

Using default java compiler Note that we have to supply –classpath parameter so

that the compiler can find the hadoop core libraries and other classes needed. $ javac –classpath $HADOOP_HOME/hadoop-0.19.0-core.jar:. –d .

Myclass.java

The hadoop core librariesThe location of other class files

Creating jar file

To create an executable jar file:1. Create a file “manifest.mf”

2. Type the command: $ jar –cmf MyExample.jar manifest.mf <list of

classes> Wildcard character * is also accepted.

Main-Class: myclassClass-Path: MyExample.jar

A white space!

A return carriage! White space separate list!

The driver class

Run the jar file

Using hadoop command. $ hadoop jar MyExample.jar <param list>

Remember that the output path should not exist. If the path exist, use rm path –r command.

A simple demonstration

A simple word count program.

Reporter

Outline

Progress ReportMapReduce ProgrammingHadoop Cluster OverviewHBase OverviewQ & A

Hadoop

Full name Apache Hadoop project. Open source implementation for reliable, scalable

distributed computing. An aggregation of the following projects (and its

core): Avro Chukwa HBase HDFS Hive MapReduce Pig ZooKeeper

Virtual Machine (VM)

Virtualization All services are delivered through VMs. Allows for dynamically configuring and managing. There can be multiple VMs running on a single

commodity machine. VMware

HDFS(Hadoop Distributed File System)

The highly scalable distributed file system of Hadoop. Resembles Google File System(GFS). Provides reliability by replication.

NameNode & DataNode NameNode

Maintains file system metadata and namespace. Provides management and control services. Usually one instance.

DataNode Provides data storage and retrieval services. Usually several instances.

MapReduce

The sophisticate distributed computing service of Hadoop. A computation framework. Usually resides on HDFS.

JobTracker & TaskTracker JobTracker

Manages the distribution of tasks to the TaskTrackers. Provides job monitoring and control, and the submission

of jobs. TaskTracker

Manages single map or reduce tasks on a compute node.

Cluster Makeup

A Hadoop cluster is usually make up by: Real Machines.

Not required to be homogeneous. Homogeneity will help maintainability.

Server Process. Multiple process can be run on a single VM.

Master & Slave The node/machine running the JobTracker or NameNode

will be Master node. The ones running the TaskTracker or DataNode will be

Slave node.

Cluster Makeup(cont.)

Administrator Scripts

Administrator can use the following script files to start or stop server processes. Can be located in $HADOOP_HOME/bin

start-all.sh/stop-all.sh start-mapred.sh/stop-mapred.sh start-dfs.sh/stop-dfs.sh slaves.sh hadoop

Configuration

By default, each Hadoop Core server will load the configuration from several files. These file will be located in $HADOOP_HOME/conf Usually identical copies of those files are maintained

in every machine in the cluster.

Outline

Progress ReportMapReduce ProgrammingHadoop Cluster OverviewHBase OverviewQ & A

HBase

The Hadoop scalable distributed database. Resembles Google BigTable. Not relational database. Resides in HDFS.

Master & RegionServer Master

For bootstrapping and RegionServer recovering. Assigning regions to RegionServers.

RegionServer Hold 0 or more regions. responsible for data transaction.

Hbase(cont.)

Row, Column, Timestamp

The data cell is the intersection of an individual row key and a column. Cells stores uninterrupted array of byte. Cell data is versioned by timestamp.

Row

Row (Key) is the primary key of database Can be consisted by arbitrary byte array.

Strings, binary data. Each row has to be distinguished. The table is sorted by row key. Any mutation action of a single row is atomic.

Column/Column Family

Columns are grouped into families, with which shares a common prefix. Ex: temperature:air and temperature:dew_point The prefix has to be a printable string. The column name can also be arbitrary byte array. Column family member can be dynamically added or

dropped. Column families must be pre-specified as table schemas.

HBase is indeed a column-family-oriented storing. The same column family will be stored together in any

file system.

Region

The table is automatically horizontally-partitioned into regions. That is, a region is a subset of data rows. Regions are stored in separated RegionServers. A region is defined by its first row, last row, and a

randomly generated identifier. The partition will be completed by the master

automatically.

Administrator Scripts

Administrator can use the following script files to start or stop server processes. Can be located in $HBASE_INSTALL/bin

start-hbase.sh / stop-hbase.sh hbase

hbase shell to initial a command line interface. hbase master / hbase regionserver

HBase shell command line

Type command help to get information. create ‘table’, ‘column family1’, ‘column family2’, …

put ‘table’, ‘row’, ‘column’, ‘value’ get ‘table’, ‘row’, {COLUMN=>…} alter ‘table’, {NAME=>‘...’}

To modify a table schema, we have to disable it first! scan ‘table’ disable ‘table’ drop ‘table’

To drop a table, we have to disable it first!! list

A Simple Demonstration

Command line operation

Operations

Create table (and its schema) Shell

create ‘table’, ‘cf1’, ‘cf2’,… create ‘table’, {NAME=>‘cf1’}, {NAME=>‘cf2’},…

APIHBaseAdmin admin = new HBaseAdmin(new HBaseConfiguration());HTableDescriptor table = new HTableDescriptor(“table”);table.addFamily(new HColumnDescriptor(“cf1:”));table.addFamily(new HColumnDescriptor(“cf2:”));admin.createTable(table);

Operations(cont.)

Modify table (and its schema) Shell

alter ‘table’, {NAME=>’cf’, KEY=>’value’, …}

API

Note that there will be exceptions if the table is not disabled.

HBaseAdmin admin = new HBaseAdmin();Admin.modifyColumn(“table”,”cf”, new HColumnDescriptor(…));Admin.modifyTable(new HTableDescriptor(…));

Operations(cont.)

Write data Shell

put ‘table’, ‘row’, ‘cf:name’, ‘value’, ts

APIHTable table = new Htable(“table”);BatchUpdate update = new BatchUpdate(“row”);update.put(“cf:name”,”value”);table.commit(update);

Operations(cont.)

Retrieve data Shell

get ‘table’, ‘row’, {COLUMN=>’cf:name’, …}

API

If we don’t know the row retrieved at prior, we can use Scanner object instead. Scanner scanner = table.getScanner(“cf:name”);

HTable table = new HTable(“table”);RowResult row = table.getRow(“row”);Cell data = table.get(“row”,”cf:name”);

Operations

Delete a cell Shell

delete ‘table’, ‘row’, ‘cf:name’

APIHTable table = new HTable(“table”);BatchUpdate update = new BatchUpdate(“row”);Udpate.delete(“cf:name”);table.commit(update);

Operations(cont.)

Enable/Disable a table Shell

enable/disable ‘table’

APIHBaseAdmin admin = new HBaseAdmin(new HBaseConfiguration());admin.disableTable(“table”);admin.enableTable(“table”);

Outline

Progress ReportMapReduce ProgrammingHadoop Cluster OverviewHBase OverviewQ & A

Q & A

Hadoop 0.19.0 API http://hadoop.apache.org/common/docs/r0.19.0/api/

index.htmlHBase 0.19.3 API

http://hadoop.apache.org/hbase/docs/r0.19.3/api/index.html

Any question?