hadoop 2.4 installing on ubuntu 14.04
TRANSCRIPT
![Page 1: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/1.jpg)
![Page 2: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/2.jpg)
facebook.com/akshath.kumar180
Twitter.com/akshath4u
in.linkedin.com/in/akshathkumar
HADOOP 2.4 INSTALLATION ON UBUNTU 14.04
![Page 3: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/3.jpg)
Hadoop is a free, Java-based programming framework that supports the
processing of large data sets in a distributed computing environment. It is part
of the Apache project sponsored by the Apache Software Foundation.
In this presentation, we'll install a single-node Hadoop cluster backed by the Hadoop
Distributed File System on Ubuntu.
HADOOP ON 64 BIT UBUNTU 14.04
![Page 4: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/4.jpg)
➢ INSTALLING JAVA
➢ ADDING A DEDICATED HADOOP USER
➢ INSTALLING SSH
➢ CREATE AND SETUP SSH CERTIFICATES
➢ INSTALL HADOOP
➢ SETUP CONFIGURATION FILES
➢ FORMAT THE NEW HADOOP FILESYSTEM
➢ STARTING HADOOP
➢ STOPPING HADOOP
➢ HADOOP WEB INTERFACES
➢ USING HADOOP
STEPS FOR INSTALLING HADOOP
![Page 5: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/5.jpg)
❖ Hadoop framework is written in Java!➢ Open Terminal in Ubuntu 14.04, and Following the Steps
INSTALLING JAVA
![Page 6: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/6.jpg)
ADDING A DEDICATED HADOOP USER
![Page 7: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/7.jpg)
➔ SSH has 2 main components :
◆ ssh : The command we use to connect to remote machines - the client.◆ sshd : The daemon that is running on the server and allows clients to connect to
the server.
➔ The ssh is pre-enabled on Linux, but in order to start sshd daemon, we need to install ssh first. Use this command to do that.
$ sudo apt-get install ssh
INSTALLING SSH
![Page 8: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/8.jpg)
➔ Hadoop requires SSH access to manage its nodes,we therefore need to configure SSH access to
localhost. So, we need to have SSH up and running on our machine and configured it to allow SSH
public key authentication.
CREATE AND SETUP SSH CERTIFICATES
![Page 9: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/9.jpg)
➔ The second command adds the newly created key to the list of authorized keys
so that Hadoop can use ssh without prompting for a password.
◆ We can check if ssh works
CREATE AND SETUP SSH CERTIFICATES
![Page 10: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/10.jpg)
➔ Type the following Command in Ubuntu Terminal.
➔ We want to move the Hadoop installation to the /usr/local/hadoop directory using
the following command
INSTALL HADOOP
$ wget http://mirrors.sonic.net/apache/hadoop/common/hadoop-2.4.1/hadoop-2.4.1.tar.gz
$ tar xvzf hadoop-2.4.1.tar.gz
$ sudo mv hadoop-2.4.1 /usr/local/hadoop
$ sudo chown -R hduser:hadoop hadoop
$ pwd
/usr/local/hadoop
$ ls
bin etc include lib libexec LICENSE.txt NOTICE.txt README.txt sbin
share
![Page 11: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/11.jpg)
➔ The following files will have to be modified to complete the Hadoop
setup
1. ~/.bashrc
2. /usr/local/hadoop/etc/hadoop/hadoop-env.sh
3. /usr/local/hadoop/etc/hadoop/core-site.xml
4. /usr/local/hadoop/etc/hadoop/mapred-site.xml.template
5. /usr/local/hadoop/etc/hadoop/hdfs-site.xml
➔ First of all we need to find the path where Java has been installed to set the
JAVA_HOME environment variable using the following command
SETUP CONFIGURATION FILES
$ update-alternatives --config java
There is only one alternative in link group java (providing /usr/bin/java):
/usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java
Nothing to configure.
![Page 12: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/12.jpg)
1. ~/.bashrc
We can append the following to the end of ~/.bashrc
SETUP CONFIGURATION FILES
![Page 13: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/13.jpg)
2. /usr/local/hadoop/etc/hadoop/hadoop-env.sh
★ We need to set JAVA_HOME by modifying hadoop-env.sh file
export JAVA_HOME=/usr/lib/jvm/java-7-
openjdk-amd64
★ Adding the above statement in the hadoop-env.sh file ensures that the
value of JAVA_HOME variable will be available to Hadoop whenever it
is started up.
SETUP CONFIGURATION FILES
![Page 14: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/14.jpg)
3. /usr/local/hadoop/etc/hadoop/core-site.xm
★ he /usr/local/hadoop/etc/hadoop/core-site.xml file contains configuration
properties that Hadoop uses when starting up. This file can be used to override
the default settings that Hadoop starts with.
$ sudo mkdir -p /app/hadoop/tmp
$ sudo chown hduser:hadoop /app/hadoop/tmp
SETUP CONFIGURATION FILES
![Page 15: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/15.jpg)
Open the file and enter the following in between the <configuration>
</configuration> tag.
SETUP CONFIGURATION FILES
![Page 16: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/16.jpg)
4. /usr/local/hadoop/etc/hadoop/mapred-site.xml
➔ By default, the /usr/local/hadoop/etc/hadoop/ folder contains the /usr/local/hadoop/etc/hadoop/mapred-site.xml.template file which has to be renamed/copied with the name mapred-site.xml
$ cp/usr/local/hadoop/etc/hadoop/mapred-site.xml.template
/usr/local/hadoop/etc/hadoop/mapred-site.xml
SETUP CONFIGURATION FILES
![Page 17: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/17.jpg)
➔ The mapred-site.xml file is used to specify which framework is being used for
MapReduce. We need to enter the following content in between the
<configuration></configuration> tag
SETUP CONFIGURATION FILES
![Page 18: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/18.jpg)
5. /usr/local/hadoop/etc/hadoop/hdfs-site.xml
➔ Before editing this file, we need to create two directories which will contain the namenode and the
datanode for this Hadoop installation. This can be done using the following commands
SETUP CONFIGURATION FILES
![Page 19: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/19.jpg)
➔ Open the file and enter the following content in between the <configuration></configuration> tag
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
<description>Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.
</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/local/hadoop_store/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/local/hadoop_store/hdfs/datanode</value>
</property>
</configuration>
SETUP CONFIGURATION FILES
![Page 20: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/20.jpg)
➔ Now, the Hadoop filesystem needs to be formatted so that we can start to use it.
The format command should be issued with write permission since it creates
current directory under /usr/local/hadoop_store/hdfs/namenode folder.
hduser@k:~$ hadoop namenode -format
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
…………………………………………………………………………. …………. ………………….. ……………………. ………..
………… …………………. …………………… …………………… ……………………………………. ………………. ………..
14/07/13 22:13:10 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at k/127.0.1.1
************************************************************/
FORMAT THE NEW HADOOP FILESYSTEM
The hadoop namenode -format command should be executed once before we start using Hadoop. If this command is
executed again after Hadoop has been used, it'll destroy all the data on the Hadoop file system.
![Page 21: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/21.jpg)
➔ Now it's time to start the newly installed single node cluster. We can use start-
all.sh or (start-dfs.sh and start-yarn.sh)
hduser@k:/home/k$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
14/07/13 23:36:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
………………. ………………….. ……………….. . ……. ………………………………….. ……………………
……………………………. ……………………………………………………………….. ……………………………….. ..
localhost: starting nodemanager,
logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-k.out
➔ check if it's really up and running.
hduser@k:/home/k$ jps
➔ Another way to check is using netstat
hduser@k:/home/k$ netstat -plten | grep java
STARTING
HADOOP
![Page 22: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/22.jpg)
➔ Type the following Command into the Terminal for Stopping
the Hadoop
$ pwd
/usr/local/hadoop/sbin
$ ls
distribute-exclude.sh httpfs.sh start-all.sh
----------------------- ------------- ---------- -----------
------------ -- -- - -- -------- - -- - - -- - - - - - ----
start-secure-dns.sh stop-balancer.sh stop-yarn.sh
➔ We run stop-all.sh or (stop-dfs.sh and stop-yarn.sh) to stop all the daemons
running on our machine:
$ /usr/local/hadoop/sbin/stop-all.sh
STOPPING HADOOP
![Page 23: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/23.jpg)
Thank You...
![Page 24: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/24.jpg)
US UK UAE
7002 Hana Road,
Edison NJ 08817,
United States of America.
90 High Street,
Cherry Hinton,
Cambridge, CB1 9HZ,
United Kingdom.
Suite No: 51, Oasis Center,
Sheikh Zayed Road, Dubai,
UAE
Email to [email protected] or Visit baabtra.com
Looking for learning more about the above
topic?
![Page 25: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/25.jpg)
India Centres
Emarald Mall (Big Bazar Building)
Mavoor Road, Kozhikode,
Kerala, India.
Ph: + 91 – 495 40 25 550
NC Complex, Near Bus Stand
Mukkam, Kozhikode,
Kerala, India.
Ph: + 91 – 495 40 25 550
Cafit Square IT Park,
Hilite Business Park,
Kozhikode
Kerala, India.
Email: [email protected]
TBI - NITC
NIT Campus, Kozhikode.
Kerala, India.
Start up Village
Eranakulam,
Kerala, India.
Start up Village
UL CC
Kozhikode, Kerala
![Page 26: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/26.jpg)
Follow us @ twitter.com/baabtra
Like us @ facebook.com/baabtra
Subscribe to us @ youtube.com/baabtra
Become a follower @ slideshare.net/BaabtraMentoringPartner
Connect to us @ in.linkedin.com/in/baabtra
Give a feedback @ massbaab.com/baabtra
Thanks in advance
www.baabtra.com | www.massbaab.com |www.baabte.com
![Page 27: Hadoop 2.4 installing on ubuntu 14.04](https://reader034.vdocuments.us/reader034/viewer/2022042514/55a615631a28abf4328b46de/html5/thumbnails/27.jpg)
Want to learn more about programming or Looking to become a good programmer?
Are you wasting time on searching so many contents online?
Do you want to learn things quickly?
Tired of spending huge amount of money to become a Software professional?
Do an online course @ baabtra.com
We put industry standards to practice. Our structured, activity based courses are so designedto make a quick, good software professional out of anybody who holds a passion for coding.