what's new in hadoop 3.0

1

What‘s new in Hadoop 3.0Heiko Loewe@loeweh

2

Tributes

• Zhe Zhang, Erasure Coding• Akira Ajisaka, Script Rewrite• Junping Du, Yarn Timeline Service v2• And manny Others

3

Hadoop 3.0 Roadmap• Hadoop 3.x Releases• Planned for hadoop-3.0.0

Classpath isolation on by default HADOOP-11656 • hadoop-3.0.0-alpha1

– HADOOP• Move to JDK8+ • Shell script rewrite HADOOP-9902 • Move default ports out of ephemeral range HDFS-9427

– HDFS • Removal of hftp in favor of webhdfs HDFS-5570 • Support for more than two standby NameNodes HDFS-6440 • Support for Erasure Codes in HDFS HDFS-7285 • Intra-datanode balancer HDFS-1312

– YARN • YARN Timeline Service v.2 YARN-2928

– MAPREDUCE • Derive heap size or mapreduce.*.memory.mb automatically MAPREDUCE-5785

4

Release Feature2.0NameNode High-Availability,2.2Federation, Snapshots, NFS v3 Mount2.3Heterogenous Storage (Phase 1), In Memory Caching2.4Rolling Upgrades, Posix ACL2.5Extended Attributes2.6Hot Swap Volumes, Heterogeneous Storage (Phase 2), transp. Encryption2.7Files w/ variable Block Length, Inotify2.8Docker Container in Linux, Yarn ATS 1.5, changing resources on alloc Yarn Container2.9TBD

Release / Features / 2.X

5

What's Apache Hadoop 3?20142010 201

1201320122009 2015

2.2.0

Heterogeneous storage HDFS in-memory caching

2.3.0 2.5.0

2.4.0

HDFS ACLs

2.0.0-alpha

2.1.0-beta

branch-1 (branch-0.20)

1.0.0 1.1.0 1.2.1(stable)0.20.1 0.20.205

0.22.0Security

0.23.11(final)NameNode Federation, YARN

0.21.0New append

0.23.0

NameNode HAbranch-2

HDFS Snapshots NFSv3 support Windows

HDFS Rolling Upgrades Application History Server RM Automatic Failover

2.6.0YARN Rolling Upgrades Transparent Encryption Archival Storage

2.7.0

Hadoop2Drop JDK6 supportTruncate API

2016

branch-0.23

trunk

Hadoop3Hadoop 3 and 2 were diverged in 2011 (5 years ago!)

Hadoop1 (EOL)

Source: Akira Ajisaka

6

Break compatibility

Major version up is to clean up the code• Deprecated APIs can be removed only in changing major version

– @Public and @Stable Java API– REST API– Metrics/JMX– CLI– Environment variables

• Wire-compatibility can be broken– 2.X client cannot talk to 3.X server and vice versa

• Compatibility Guide:– Apache Hadoop 3.0.0-alpha1 – Apache Hadoop Compatibility

7

Erasure Coding

Click icon to add picture

8

Traditional Hadoop

• HDFS inherits 3-way replication from Google File System• Simple, scalable and robust

• 200% storage overhead• Secondary replicas rarely accessed

9

Erasure Coding Saves Storage

• Simplified Example: storing 2 bits

• Same data durability– can lose any 1 bit

• Pro: Half the storage overhead• Cons: Slower recovery

1 01 0Replication: 2 extra bits

XOR Coding: 1 0⊕ 1= 1 extra bit

10

Erasure Coding with m Data, n ParityReed Solomon Coding

11

Durability and EfficiencyData Durability = How many simultaneous failures can be tolerated?Storage Efficiency = How much portion of storage is for useful data?

  Data Durability Storage Efficiency Single Replica 0 100%3-way Replication 2 33%XOR with 6 data cells 1 86%RS (6,3) 3 67%RS (10,4) 4 71%

12

Continious Layout

13

Stripped Layout

14

Other Implementation

15

HDFS Roadmap

16

Erasure Coding: Current Status Phase 1: striping

layout C = 64KB (default)

Work for small files

No data locality Available on trunk Phase 2: contiguous layout C = 128MB (= HDFS Block

size) Not work for small files Data locality Now in progress (HDFS-8030)

Incoming Data

DataNode 1DataNode 2DataNode 3DataNode 4DataNode 5

・・・

Cell size (C)

16

17

Name Server ChangesMapping Logical and Storage Blocks

Too Many Storage Blocks?Hierarchical Naming Protocol:

18

Erasure Coding: Write files using (6,3)-Reed-Solomon

・・・・・・・・・

・・・

Write data to 9 DNs in parallel6 Data Blocks

18

DN1

DN6

DN7

Incoming Data

DN9

3 Parity Blocks

ECClient

19

Erasure Coding: Read files Read data from 6 DNs in

parallelDN1

ECClient

DN6

DN9

・・・・・・

・・・

19

20

Erasure Coding: Read files when DN fails Read data from arbitrary 6 DNs in

parallelDN1

ECClient

×DN6

DN7

DN9

・・・・・・

・・・

20

21

hdfs erasure command[loewe@loewe hadoop-3.0.0-alpha1]$ bin/hdfs erasurecode -helpUsage: hdfs erasurecode [generic options]

[-getPolicy <path>][-help [cmd ...]][-listPolicies][-setPolicy [-p <policyName>] <path>]

-getPolicy <path> : Get erasure coding policy information about at specified path

-help [cmd ...] : Displays help for given command or all commands if none is specified.

-listPolicies : Get the list of erasure coding policies supported

-setPolicy [-p <policyName>] <path> : Set a specified erasure coding policy to a directory Options : -p <policyName> erasure coding policy name to encode files. If not passed the default policy will be used <path> Path to a directory. Under this directory files will be encoded using specified erasure coding policy

22

Acceleration with Intel ISA-L

• 1 legacy coder– From Facebook’s HDFS-RAID project

• 2 new coders– Pure Java — code improvement over HDFS-RAID– Native coder with Intel’s Intelligent Storage Acceleration Library (ISA-L)

23

Benchmarks

24

Benchmarks

25

Benchmarks

26

Benchmarks

27

Yarn Timeline Service v2

Click icon to add picture

28

First, A bit of Vision…

• Evolution of Hadoop start with YARN• YARN Evolution will continue to drive Hadoop forward• Hadoop 3 will still use Yarn, but there are a lot of Improvements

Hadoop 3

29

Several important trends in age of Hadoop 3.0 +

YARN and Other Platform Services

StorageResource

Management SecurityServiceDiscovery Management

Monitoring

Alerts

IOT Assembly

Kafka Storm HBase Solr

Governance

MR Tez Spark …

Innovating frameworks:

Flink, DL(TensorFlow)

, etc.

Various Environments

On Premise Private Cloud Public Cloud

30

Yarn Architecture

Yarn

Resource Database

Scheduler

ApplicationTimeline Service

31

YARN Process Flow - Walkthrough

NodeManager NodeManager NodeManager NodeManager

Container 1.1

Container 2.4

NodeManager NodeManager NodeManager NodeManager

NodeManager NodeManager NodeManager NodeManager

Container 1.2

Container 1.3

AM 1

Container 2.2

Container 2.1

Container 2.3

AM2

Client2

Yarn

Scheduler

Yarn

Timeline Service Client (Query)

32

Why Timeline Service v2

• Scalability and reliability challenges– Single instance of Timeline Server– Storage (single local LevelDB instance)

• Usability– Flow– Metrics and configuration as first-class citizens– Metrics aggregation up the entity hierarchy

33

Highlights

v.1 v.2Single writer/reader Timeline Server Distributed writer/collector architectureSingle local LevelDB storage* Scalable storage (HBase)v.1 entity model New v.2 entity modelNo aggregation Metrics aggregationREST API Richer query REST API

34

Architecture

• Separation of writers (“collectors”) and readers• Distributed collectors: one collector for each app• Dedicated RM collector for RM-generated data• Collector discovery via RM• Pluggable storage with HBase as default storage

35

Distributed Collectors and Readers

36

New Entity Model

• Flows and flow runs as parents of YARN applicaSon enSSes• First-class configuraSon (key-value pairs)• First-class metrics (single-value or Sme series)• Designed to handle mulS-cluster environment out of the box

37

What is a flow

• A flow is a group of YARNapplications that are launched as parts of a logical app

• Oozie, Scalding, Pig, etc.– name:– “frequent_visitor_stat”– run id: 1466097809000– version: “b9b9068”

38

Metrics Aggregation

• Application level– Rolls up sub-application metrics– Performed in real time in the collectors

in memory• Flow run level

– Rolls up app level metrics– Performed in HBase region servers via

coprocessors• Offline aggregation (TBD)

– Rolls up on user, queue, and flow offline periodically

– Phoenix tables

39

More Cloud Friendly• Elastic

– Dynamic Resource Configuration• YARN-291• Allow tune down/up on NM’s resource in runtime

– Graceful decommissioning of NodeManagers• YARN-914• Drains a node that’s being decommissioned to allow running containers to finish

• Efficient– Support for container resizing

• YARN-1197• Allows applications to change the size of an existing container

40

More Cloud Friendly (Contd.)• Isolation

– Embrace container technology to achieve better isolation– Resource isolation support for disk and network

• YARN-2619 (disk), YARN-2140 (network)• Containers get a fair share of disk and network resources using Cgroups

– Docker support in LinuxContainerExecutor• YARN-3611• Support to launch Docker containers alongside process• Packaging and resource isolation

• Operation– Container upgrades (YARN-4726)

• ”Do an upgrade of my Spark / HBase apps with minimal impact to end-users”– AM Restart With Work Preserving

• MAPREDUCE-6608

41

• Add a native implementation of the map output collector– Sort, Spill and IFile serialization

• Prequisites– Built with -Pnative option– Custom writable types and comparators are not supported

• Setting<property name="mapreduce. job.map.output .co l lec tor.c las s" value="org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator">

Task level native optimization (MAPREDUE-2841)

42

Benchmark

• Release Note in the issue:– "For shuffle-intensive jobs this may provide speed-ups of 30% or more."

• Benchmarked with 3 slaves (m3.xlarge)– CentOS 7.2– 3.0.0-SNAPSHOT (revision 5865fe2b)

• A very shuffle-intensive wordcount job– Input: 2.6GB (compressed)– Shuffle: 14GB– Output: 10GB

42

43

Updated Web UI

44

Setup Timeline Service v2

• Set up the HBase cluster (1.1.x)– Add the timeline service jar to HBase– Install the flow run coprocessor– Create tables via TimelineSchemaCreator utility

• Configure the YARN cluster– Enable Timeline Service v.2– Add hbase-site.xml for the timeline collector and readers– Start the timeline reader daemon

45

Shell Script rewrite

Click icon to add picture

46

Directory Structure

47

Bin/hadoop Command

48

bin/hdfs Command

49

Shell Script Rewrite (HADOOP-9902)

• Hadoop and Shell Script– Launching daemons– Hadoop CLI

• Difficult to understand– What is the correct env var to set a option

• java classpath?• java.library.path?• GC options?

– How to add the option to the env var– We have to read almost all the shell scripts!

• New CLI is not completely downward compatible– Hadoop 3: bin/hdfs namenode ‐format– Hadoop 2: bin/hadoop namenode –format

Apache Hadoop 3.0.0-alpha1 – Apache Hadoop Compatibility

50

After rewriting the scripts ...

• Easy to understand• Because shell API doc is available

– Shelldoc maker generates docsfrom the scripts

– Similar to JavaDoc

Public API

Documentation by Akira Ajisaka build (trunk):http://aajisaka.github.io/hadoop-project/hadoop-project-dist/hadoop-common/UnixShellAPI.html

51

• Very similar to .bashrc– Read the API doc– Create your own ~/.hadoopXX

• hadoop-env : hadoop-env.sh for each user• hadooprc : called after shell env vars are configured

• And that's all :)

• ex.) Set additional classpath (.hadooprc)

hadoop_add_classpath /path / to /my/ jar

.hadoop-env and .hadooprc(HADOOP-11353, HADOOP-13045)

52

--debug option is available

CLASSPATH was overwritten!! (before HADOOP-13045)

Useful for troubleshooting

$ hadoop --debugversionDEBUG: DEBUG: DEBUG:( sn i p) DEBUG: DEBUG: DEBUG:( sn i p)DEBUG:

hadoop_parse_args: procesiong version hadoop_parse: asking c a l l e r toskip 1 HADOOP_CONF_DIR=/usr / local /hadoop/ e t c / hadoopApplying the u se r ' s

. hadooprc I n i t i a lCLASSPATH=/path/to/my/jar I n i t i a l i z e CLASSPATHI n i t i al

CLASSPATH=/usr / l o c a l / hadoop/ sha re / hadoop/common/lib/*

52

53

Many new features, bug fixes, improvements

• 'hadoop distch' to change the ownership and permissions on many files via MapReduce job

• 'hadoop jnipath' to print java.library.path• 'hadoop --daemon' instead of hadoop-daemon.sh

– ex.) hdfs --daemon status namenode– The return code for status is LSB-compatible– hadoop-daemon(s).sh are now deprecated

• .out files are now appended (not overwritten)– Allows external log rotation

• and many more– see https://issues.apache.org/jira/browse/HADOOP-9902

53

54

Derive heap size or mapreduce.*.memory.mb automatically (MAPREDUCE-5785)

• In Hadoop 2, two similar properties must be set :(– mapreduce.{map,reduce}.memory.mb

• The amount of memory to request from the scheduler for each task (ex. 2048)– mapreduce.{map,reduce}.java.opts

• Java options for YARN containers (ex. -Xmx2G)

• In Hadoop 3, either is enough– .java.opts is derived from .memory.mb and vice versa

• .java.opts = .memory.mb * mapreduce.job.heap.memory-mb.ratio• .memory.mb = .java.opts / mapreduce.job.heap.memory-mb.ratio

55

Intra Data-Node Balancer (HDFS-1312)

• Due to activities like deletesthe volumes of a DataNodemay become imbalance filled

DataNodeBlock PlacingPolicies

56

Intra Data-Node Balancer (HDFS-1312)

• Offline scripts existed to reblance a DataNode• HDFS-1312 introduces a online process that

rebalances the Volume of a DataNode• „hdfs diskbalancer“ Command

57

Multiple Name Nodes

Click icon to add picture

58

Support more than two NameNodes (HDFS-6440)

• Hadoop 2 now supports only 2 NameNodes– 1 active and 1 standby

• Hadoop 3 supports 2 or more standby NameNodes– provides additional fault-tolerance– avoids multiple standby NNs to checkpoint at the same time– # of standby should be small due to block report

(typically 3 or 5 NameNodes)

59

Old Layout

ZK Failover Controller

Active NameNode

ZK Failover Controller

Active NameNode

Zookeeper Zookeeper Zookeeper

Fencing

Monitors the health of the NN Participating in election of the active NN Coordinates transition process Fence the other NN, if it win election

The information whichNN is active is kept her

60

New Layout

ZK Failover Controller

Active NameNode

ZK Failover ControllerStandby

NameNode

Zookeeper Zookeeper Zookeeper

Fencing

Monitors the health of the NN Participating in election of the active NN Coordinates transition process Fence the other NN, if it win election

The information whichNN is active is kept her

ZK Failover ControllerStandby

NameNode… 4 .. 5

61

All Feature supported

• Checkpointing with NFS• Checkpoint with Quorum Journal Daemon• Manual Failover• Automatic Failover

62

Incompatible changes

• Many deprecated APIs will be removed– hftp/hsftp/s3 -> webhdfs/s3{n,a}– Metrics v1– org.apache.hadoop.Records– and more

• Improved CLI output– 'mapred job -list' shows the job name as well– 'hadoop fs -du' shows the raw disk usage, and aligned more unix-like– and more

• Search 'Incompatible change' flag– https://s.apache.org/sMO4

62

63

Bump up the versions of the libraries

• Drop JDK7 support (HADOOP-11858)• Dependency Hell

– Tomcat– Jetty– Jersey– Guava– Log4J– Jackson– And many more

common/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/hadoop/share/ hadoop/common/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/hadoop/share/hadoop/ common/lib/slf4j-api-1.7.10.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/local/hadoop/share/hadoop/common/lib/jsp- api-2.1.jar:/usr/local/hadoop/share/hadoop/common/lib/curator-client-2.7.1.jar:/usr/local/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:!/usr/local/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-collections-3.2.2.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/hadoop/ share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/common/lib/nimbus-jose-jwt-3.9.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/ hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/hadoop/share/hadoop/common/lib/junit-4.11.jar:/usr/local/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/share/hadoop/common/lib/htrace- core4-4.0.1-incubating.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/ netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-annotations-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local! /hadoop/share/hadoop/common/lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/json-smart-1.1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/hadoop/share/hadoop/common/lib/ java-xmlbuilder-0.4.jar:/usr/local/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/common/lib/jsch-0.1.51.jar:/usr/local/hadoop/share/hadoop/common/lib/curator-framework-2.7.1.jar:/usr/local/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jcip-annotations-1.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/ usr/local/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/hadoop/share/ hadoop/common/lib/hadoop-auth-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/usr/local/hadoop/share/ hadoop/common/lib/gson-2.2.4.jar:/usr/local/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/commons- compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/common/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/ usr/local/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/hadoop/share/hadoop/common/lib/asm-3.2.jar:/usr/local/ hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/ hadoop/common/lib/curator-recipes-2.7.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-3.0.0-SNAPSHOT-tests.jar:/usr/local/ hadoop/share/hadoop/common/hadoop-nfs-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/hadoop- hdfs-client-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/netty-all-4.1.0.Beta5.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/hpack-0.11.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/okio-1.4.0.jar:/usr/local/ hadoop/share/hadoop/hdfs/lib/okhttp-2.4.0.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-3.0.0-SNAPSHOT-tests.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop- mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop- mapreduce-client-app-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce- examples-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client- common-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT-tests.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar:/usr/local/hadoop/share/hadoop/yarn/lib/metrics-core-3.0.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/usr/

local/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/curator-test-2.7.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/fst-2.24.jar:/usr/local/hadoop/share/hadoop/yarn/lib/objenesis-2.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/local/hadoop/share/ha

doop/yarn

/lib/zookeeper-3.4.6

-tests.jar:/usr/lo

cal/ha

doop/share/hadoo

p/yarn/lib/guice-servle

0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-math-2.2.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-api-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-common-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/ share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/ yarn/hadoop-yarn-server-web-proxy-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn- registry-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.0.0-SNAPSHOT.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-client-3.0.0-SNAPSHOT.jar:/usr/ java/latest/lib/tools.jar

64

Classpath isolation (HADOOP-11656)

• Relaxing the "dependency hell"– Separate client and server jars– Client jar does not pull any third party dependencies

• If the isolation is done ...– We can safely upgrade the libraries in server code– In branch-2, the upgrade is incompatible :(

64

65

Thank YouFollow me on Twitter: @loeweh