2.0

YARN

Hadoop Intro

● Apache Hadoop is an open-source software framework that supports data-intensive distributed applications.

● Supports running of applications on large clusters of commodity hardware.

● Task are divided into Map-Reduce framework

● Provides a distributed file system that stores data on the compute nodes.

Components of Hadoop 1.0

● JobTracker

● TaskTracker

● DataNode

● NameNode

● Secoundary NameNode

Why Hadoop 2.0

Drawbacks of Hadoop 1.0

● Cluster is tightly couple with Hadoop.

● Cascading failures,.

What is Hadoop 2.0

● Re-architectured Hadoop is complete overhaul of 0.23 branch.

● Introduced YARN and MR2.

● Enhanced resource scheduler.

● Efficient utilization of cluster by running apps apart from MR Jobs.

Components of Hadoop 2.0

● NameNode

● DataNode

● YARN

● MR2 Framework

What is Yarn ?

Yet-Another-Resource-Negotiator

Components of YARN

● ResourceManager

● NodeManager

● ApplicationMaster

● History Server

ResourceManager

The ResourceManager is the ultimate authority in Hadoop cluster. Which utilise resources among all the applications in the system. All the negotiations of resources are done from the ResourceManager.

Components of Resource Manager

Scheduler

The Scheduler is responsible for allocating resources to the various running applications.

ApplicationsManager

The ApplicationsManager is responsible for accepting job-submissions, negotiating the first container for executing the application specific ApplicationMaster and provides the service for restarting the ApplicationMaster container on failure.

NodeManager

The NodeManager is the per-machine agent who is responsible monitoring the resources for the respective machine it is running on and report the same to the ResourceManager.

Containers are allocated on NodeManager to perform the task assigned

ApplicationMaster

● It is a specific library for negotiating resources from the ResourceManager and working with the NodeManager(s) to execute the task on containers and the monitor the same.

● ApplicationMaster has the responsibility of negotiating resource containers from the Scheduler for the tasks.

● Provides communication port to users to communicate with Application Master.

History ServerThe history server provide users to get status on finished applications.

YARN Application Flow

YARN Solution

● Apache YARN, will provide a framework on which various application can execute.

● Hadoop backers expect that the advent of Yarn could open the floodgates for new applications being built to run on Hadoop.

● Various projects, like Apache Tez, have been created to do more advanced data processing compared to what MapReduce specializes in.

● YARN promotes effective utilization of resources while providing distributed environment for application execution

Current use case on YARN

Storm-YARNStreaming IN Hadoop: Yahoo! release

Storm-YARN enables Storm applications to utilize the computational resources in a Hadoop cluster along with accessing Hadoop storage resources such as HBase and HDFS.

Samza: Linked-In Release

Apache Samza is a distributed stream processing framework. It uses Apache Kafka for messaging, and Apache Hadoop YARN to provide fault tolerance, processor isolation, security, and resource management

Any Questions

Author: Abhishek KapoorTwitter: @kapoorSunny