8/13/2019 Hadoop Performance Tuning Guide

http://slidepdf.com/reader/full/hadoop-performance-tuning-guide 1/1

Shrinivas Joshi, Advanced Micro Devices, Inc. {shrinivas.joshi@amd.com}

Introduction

Users who have deployed and triedtuning their Hadoop clusters for the firsttime will certainly attest to the fact thatoptimizing Hadoop clusters is a

daunting task.

Apart from the nature andimplementation of Hadoop jobs,hardware, network infrastructure, OS, JVM, and Hadoop configurationproperties all have a significant impacton performance and scalability ofHadoop workloads.

Using TeraSort as a reference workload,this poster attempts to educate theaudience on challenges involved intuning performance of Hadoop setup,different tools available for monitoringand diagnosing performancebottlenecks, tuning best practices,empirical data on effect of varioustunings on performance, and somefuture directions.

Challenges

• Hadoop is a large and complicatedframework involving a number ofentities interacting with each otheracross multiple hardware systems.• Performance of Hadoop jobs issensitive to every component of thecluster stack: Hadoop configuration, JVM, OS, network infrastructure,underlying hardware, and possiblyBIOS settings.• Hadoop supports a large number ofconfiguration properties and a goodchunk of these can potentially impact

performance.• As with any large software system,diagnosing performance issues is acomplicated task.

Tools

• Ganglia and Nagios  – effective inmonitoring cluster-wide statistics.• Hadoop Vaidya – offers guidelines onmitigating framework-level bottlenecks.• Hadoop Core – offers simple andinsightful information on performance-affecting events.• AMD CodeAnalystTM, Oracle® SolarisStudio and Intel® VTuneTM

– profilersfor in-depth source-level analysis.•

AMD MultEvent, Linux perf, andOProfile – platform-level profilers forin-depth analysis of hardware-levelperformance events.

Tuning Highlights• On ClusterA,we achieved a speed-upfactorof 5.6X.• OnClusterB, we achieved a speed-upfactorof 2.2X.

Conclusion &Future Direction• Configuration tuning of all thecomponents of Hadoop stack is animportant exercise and can offer a hugeperformance payoff.• Different Hadoop workloads will have

different characteristics, so it isimportant to experiment with differenttuning options.• We want to perform a similar study inmulti-tenant environments and in thecloud environment.• We want to explore JVM and JDKoptimizations targeting peculiarcharacteristics of Hadoop frameworkand jobs running on top of it.

• On 64-bit Oracle JDK 6 update 25,compressed pointers are ON by default. Ifyou are using an older version of JDK andcompressed pointers are disabled,experiment with enabling them.• Compressed pointers reduce memoryfootprint.

 JVM Configuration Tuning • Biased-locking feature of Oracle HotSpot JDK improves performance in scenarioswith uncontended locks.• Given the architecture of Hadoopframework, biased locking shouldgenerally improve performance.

• When all the CPU cores on thehardware are not fully utilized, theprocessor could be downgrading CPUfrequency.• Experiment with ACPI and otherpower-related BIOS options.

BIOS Configuration Tuning • Modern AMD processors support afeature called HT assist (a.k.a. probefilters). This feature reduces traffic onmemory interconnects at the expense ofsome portion of L3 cache.• Experiment with this feature. Trydisabling it if your job is cache-sensitive.

• Native command queuing feature ofmodern hard drives helps improve I/Operformance by optimizing drive headmovement.• Experiment with AHCI option in BIOS,which can be used to enable NCQ mode.

• On some AMD processor-basedsystems, HyperTransportTM link speedand width are dynamically adjusted toreduce power consumption.• If memory bandwidth is a bottleneck,experiment with this option.

• Certain Linux distributions supportEXT4 as the default file-system type.• If you are using another type of filesystem, experiment with EXT4 filesystem.

OS Configuration Tuning 

• Using maximum possible map andreduce slots, identify the optimal numberof disks that maximizes I/O bandwidth.• Experiment with different HDFS block

sizes.• Identify heap usage and GCcharacteristics of framework processesand lock in their heap and GC settings.

Hadoop Configuration Tuning 

• Try to avoid or eliminate intermediatedisk I/O operations on reduce side bytuning heap sizes of reduce JVMs.• If the reduce functionality is not heap

heavy, adjust map output buffer size.• Tune framework-related resources suchas task tracker threads, data node, andname node handler count.

• Start with biggest-payoff properties.• Enable map output compression.• Experiment with different codec choices.In our experience, LZO codec performed

better across multiple workloads.• Experiment with JVM reuse policy.• This helps reduce disk & network I/Owait and increase CPU utilization.

• If memory is not a bottleneck, try toeliminate map-side spills by tuningio.sort.mb. At the least, reduce thenumber of spills.•

In case there are no spills, tuneio.sort.spill.percent.•This also helps reduce disk I/O waittime and increase CPU utilization.

Best PracticesHadoop:• Identify the right number of data disksyour job requires.• Observe Hadoop framework heap

usage and GC patterns and lock in heapand GC JVM flags for these processes.• Using default settings, start withHadoop configuration tuning.• Focus on properties shown to have abig impact, such as map/reduce outputcompression and JVM reuse policy.• Avoid or eliminate map-side spills.• Avoid or eliminate reduce-side diskI/O by tuning reduce JVM heap size,map output copier threads, mergefactor, and tasktracker threads handlingreduce-side requests.• If reduce functionality is not heap-heavy, try to maximize the map outputstorage buffers.• Try to maximize CPU usage by tuningnumber of map and reduce JVMs.• Tune other framework-relatedcomponents such as datanode andnamenode handler count.

 JVM:• Experiment with performance-affecting JVM flags such as biasedlocking, compressed pointers,AggressiveOpts, etc.• Perform a thorough Java GC analysisand tune GC related flags.

Operating System:• Experiment with various OS-leveltunings such as choice of FS, choice ofI/O scheduler, and limits on OSresources.

BIOS:• Finally, verify if any default BIOSsettings are negatively affectingperformance of your Hadoop jobs.

Cluster A• 6 DNs, 1 NN: 2 chips/6 cores per chip:AMD OpteronTM 2435 @2.6GHz• 16GB DDR2 800 RAM per node• 6 x 1TB Samsung Spinpoint F3 @7200• Ubuntu 11.04 Server x64, Oracle JDK6update 25 x64• Dataset size – 64 GB

Cluster B• 5 DNs: 2 chips/4 cores per chip: AMDOpteron 2356TM @2.3GHz• 1 NN: 4 chips/4 cores per chip: AMDOpteron 8356TM @ 2.6GHz• 64GB DDR2 667 and DDR2 800 RAM• 6 x 1TB Samsung Spinpoint F3 @7200• Ubuntu 11.04 Server x64, Oracle JDK6update 25 x64• Dataset size – 1 TB

• Experiment with AggressiveOpts flag.• Experiment with UseCompressedStringsflag.• Experiment with UseStringCache flag.• Experiment with UseNUMA flag.•Experiment with UseLargePages flag.

• Verify whether the JVM is running out ofcode cache. Increase code cache size ifnecessary.• Perform detailed GC log analysis andtuning.

• By default, every file read operationtriggers a disk write operation formaintaining last access time.• Disable this logging using noatime,nodirtaime FS attributes.• Experiment with other FS attributetuning such as extent, flex_bg, barrier etc.

• Linux kernels support 4 different types ofI/O schedulers – CFQ, deadline, no-op,and anticipatory.• Experiment with different choices of I/Oscheduler, especially CFQ and deadline.

100.00%

85.00%

0

0.2

0.4

0.6

0.8

1

1.2

FSattributes

   E  x  e  c  u   t   i  o

  n   t   i  m  e

Effect of noatime file system attribute

FS noatimeattributedisabled

FS noatimeattributeenabled

100.00%

85.00%

0

0.2

0.4

0.6

0.8

1

1.2

IOscheduler choices

   E  x  e  c  u   t   i  o

  n   t   i  m  e

Effect of I/O scheduler choices

"deadline" scheduler

"CFQ"scheduler

100.00%

91.00%

0

0.2

0.4

0.6

0.8

1

1.2

Filesystem type

   E  x  e  c  u   t   i  o

  n   t   i  m  e

Effect of EXT4 file system

Data disksusingEXT3 FS

Data disksusingEXT4 FS

100.00%97.00%

0

0.2

0.4

0.6

0.8

1

1.2

GC tuning

   E  x  e  c  u   t   i  o  n   t   i  m  e

Effect of GC tuning 

WithoutGC tuning

WithGC tuning

100.00%

81.80%

0

0.2

0.4

0.6

0.8

1

1.2

Map-sidespill configurations

   E  x  e  c  u   t   i  o  n   t   i  m  e

Effect of eliminating map-side spills &tuning io.sort.factor

Map-sidespillsoccuring

Map-sidespillseliminated

100.00%

71.80%

0

0.2

0.4

0.6

0.8

1

1.2

Mapoutput compressionconfiguration

   E  x  e  c  u   t   i  o  n   t   i  m  e

Effect of enabling map compression usingLZO

Compressiondisabled

Compressionenabled

100.00%94.00%

0

0.2

0.4

0.6

0.8

1

1.2

Reducesidebufferspaceconfiguration

   E  x  e  c  u   t   i  o  n   t   i  m  e

Effect of tuning reduce-side configurations

Withoutreduce sidetuning

Withreduce sidetuning

100.00%94.50%

0

0.2

0.4

0.6

0.8

1

1.2

Biasedlocking configuration

   E  x  e  c  u   t   i  o  n   t   i  m  e

Effect of biased locking 

Biased lockingdisabled

Biased lockingenabled

100.00%96.80%

0

0.2

0.4

0.6

0.8

1

1.2

Compressedpointers configuration

   E  x  e  c  u   t   i  o  n   t   i  m  e

Effect of compressed pointers

Compressedpointersdisabled

Compressedpointersenabled

100.00% 102.00%

0

0.2

0.4

0.6

0.8

1

1.2

Aggressiveoptimizations configuration

   E  x  e  c  u   t   i  o  n   t   i  m  e

Effect of JVM aggressive optimizations

Without-XX:+AggressiveOptsflag

With-XX:+AggressiveOpts flag

• Linux OS limits such as max open filedescriptors and epolllimits can affectperformance.• Experiment with these limits.

100.00% 101.00%

0

0.2

0.4

0.6

0.8

1

1.2

Linuxopen filedescriptors limitsettings

   E  x  e  c  u   t   i  o

  n   t   i  m  e

Effect of Linux OS resource settings

DefaultopenFDlimit(1024)

Openfd limitsetto 16K

100.00% 99.00%

0

0.2

0.4

0.6

0.8

1

1.2

Powersavingsettings inBIOS

   E  x  e  c  u   t   i  o  n   t   i  m  e

Effect of power-saving settings in BIOS

Powersavingenabled inBIOS

Powersavingsdisabled inBIOS

© 2011 Advanced Micro Devices, Inc.AMD, theAMD Arrowlogo, AMDOpteron andcombinationsthereof aretrademarks ofAdvanced Micro Devices, Inc. HyperTransportis a trademark ofHyperTransport Technology Consortium.

100.00%

51.00%

43.00%

0

0.2

0.4

0.6

0.8

1

1.2

Numberof data disks

   E  x  e  c  u   t   i  o  n   t   i  m  e

Scaling with number of data disks

1 disk

3 disks

5 disks

100.00% 98.00%

0

0.2

0.4

0.6

0.8

1

1.2

HyperTransport settings

   E  x  e  c  u   t   i  o  n   t   i  m  e

Effecting of tuning HyperTransport settings

HyperTransportlinkspeed andwidthsetto "auto"

HyperTransportlinkspeed setto2.2GHz and widthsetto 16

76.01

44.58

1.26

4.59

8.30

0.30

3.30

0

10

20

30

40

50

60

70

80

64 GB TeraSortonClusterA 1TB TeraSortonClusterB

   T  o   t  a   l  p  e  r  c  e

  n   t  a  g  e  g  a   i  n  s  a  s  c  o  m  p  a  r  e   d   t  o   t   h  e   b  a  s  e   l   i  n  e  e  x  e  c  u   t   i  o  n   t   i  m  e

ClusterConfigurations

BIOS

OS

 JVM

Hadoop