Intro  to  YARN  Apps  Sandy  Ryza  

Introduc4on  

•  What’s YARN? •  YARN apps •  Building YARN apps

The  OS  analogy  

Traditional Operating System

Storage: File System

Execution/Scheduling: Processes/Kernel

Scheduler

The  OS  analogy  

Hadoop

Storage: Hadoop Distributed File System (HDFS)

Execution/Scheduling: YARN!

Goal:  Mul4tenancy  

•  Different types of applications on the same cluster

•  Different users and organizations on the same cluster

ResourceManager  (RM)  

•  Central service that tracks o  Nodes

§  Resources o  Applications o  Containers

•  Houses scheduler, which is in charge of all container placement decisions

NodeManager  (NM)  

•  One on every node •  Launches container processes •  Enforces resource allocations •  Monitors liveliness

Applica4on  Master  (AM)  

•  User/application code •  Every application instance has one •  Runs inside a container on the cluster •  Requests resources from ResourceManager

YARN  

ResourceManager

NodeManager NodeManager

Container

Map Task

Container

Application Master

Container

Reduce Task

JobHistoryServer Client

Processing  Frameworks  /  YARN  apps  

•  MapReduce o  Batch processing, fault tolerant

•  Impala o  Low latency SQL on Hadoop

•  Spark o  Load data into memory, great for iterative

algorithms •  Storm o  Stream processing

YARN  app  models  

•  Applica4on  master  (AM)  per  job  •  Most  simple  for  batch  •  Used  by  MapReduce  

YARN  app  models  

•  Applica4on  master  per  session  •  Runs  mul4ple  jobs  on  behalf  of  the  same  user  •  Recently  added  in  Tez  •  Spark  interac4ve  mode  

YARN  app  models  

•  Singleton  AM  as  permanent  service  •  Always  on,  waits  around  for  jobs  to  come  in  •  Used  for  Impala  

YARN/MR  Scheduling  

Fair Scheduler Decide which jobs to give resources to

ResourceManager

Decide which tasks to give resources to within a job

MapReduce Application Master

Scheduling  on  Hadoop  

ResourceManager

Application Master 1

Application Master 2

Node 1 Node 2 Node 3

Scheduling  on  Hadoop  

ResourceManager

Application Master 1

Application Master 2

Node 1 Node 2 Node 3

I want 2 containers with 1024 MB and a 1 core each

Scheduling  on  Hadoop  

ResourceManager

Application Master 1

Application Master 2

Node 1 Node 2 Node 3

Noted

Scheduling  on  Hadoop  

ResourceManager

Application Master 1

Application Master 2

Node 1 Node 2 Node 3

I’m still here

Scheduling  on  Hadoop  

ResourceManager

Application Master 1

Application Master 2

Node 1 Node 2 Node 3

I’ll reserve some space on node1 for AM1

Scheduling  on  Hadoop  

ResourceManager

Application Master 1

Application Master 2

Node 1 Node 2 Node 3

Got anything for me?

Scheduling  on  Hadoop  

ResourceManager

Application Master 1

Application Master 2

Node 1 Node 2 Node 3

Here’s a security token to let you launch a container on Node 1

Scheduling  on  Hadoop  

ResourceManager

Application Master 1

Application Master 2

Node 1 Node 2 Node 3

Hey, launch my container with this shell command

Scheduling  on  Hadoop  

ResourceManager

Application Master 1

Application Master 2

Node 1 Node 2 Node 3

Container

Should you build a YARN app?

•  MapReduce can’t run arbitrary DAGs? o  Use Spark

Should you build a YARN app?

•  MapReduce can’t store data in memory? o  Use Spark

Should you build a YARN app?

•  Iterative processing? o  Use Spark

Should you build a YARN app?

•  Have an existing distributed app that runs all tasks at once? o  Use distributed shell

When to build a YARN app

•  Allocating and releasing containers dynamically

•  Weird scheduling requirements o  Gang o  Complex locality

What YARN does for you

•  Deploys your bits •  Runs your processes •  Monitors your processes •  Kills your processes when they misbehave

What YARN does not do for you

•  Communication between your processes

AMRMClientAsync CallbackHandler handler = new CallbackHandler() {

public void onContainersAllocated(List<Container> containers) {

for (Container container : containers) {

startTask(container);

}

}

[... more methods]

}

AMRMClientAsync amClient = AMRMClientAsync.createAMRMClientAsync(1000, handler);

amClient.registerApplicationMaster(NetUtils.getHostName(), -1, “”);

amClient.addContainerRequest(

new ContainerRequest(

Resource.newInstance(1024, 1),

new String[] {“node1”, “node2”}, new String[] {“rack1”},

Priority.newInstance(2)));

NMClientAsync CallbackHandler nmHandler = new CallbackHandler() {

[... listen for containers stopped and started]

}

NMClientAsync nmClient = NMClientAsync.createNMClientAsync(nmHandler);

Launching Containers

public void startContainer(Container container) {

ContainerLaunchContext launchContext =

ContainerLaunchContext.newInstance(

localResources,

environment,

Arrays.asList(“sleep 1000”),

serviceData,

tokens,

acls);

nmClient.startContainerAsync(container, launchContext);

}

Local resources

HDFS

Node Container Container

file.txt

file.txt

Node Container Container

file.txt