apache hadoop at yahoo! ready for business

32
Apache Hadoop at Yahoo! Ready for Business Chris Douglas Hadoop Team [email protected] @chris_douglas

Upload: wilton

Post on 24-Feb-2016

49 views

Category:

Documents


1 download

DESCRIPTION

Apache Hadoop at Yahoo! Ready for Business. Chris Douglas Hadoop Team [email protected] @ chris_douglas. Introductions. Apache Hadoop Full time contributor since May 2007 Committer, Member of PMC Apache Member Yahoo Hadoop HDFS, Performance, MapReduce development team. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Apache Hadoop at Yahoo! Ready for Business

Apache Hadoop at Yahoo!Ready for Business

Chris DouglasHadoop Team

[email protected]@chris_douglas

Page 2: Apache Hadoop at Yahoo! Ready for Business

2

Introductions

• Apache Hadoop– Full time contributor since May 2007– Committer, Member of PMC– Apache Member

• Yahoo– Hadoop HDFS, Performance, MapReduce development team

Page 3: Apache Hadoop at Yahoo! Ready for Business

3

Timeline of Apache Hadoop at Yahoo

• Hadoop is mission critical for Yahoo• Making Hadoop enterprise-ready for Yahoo

Page 4: Apache Hadoop at Yahoo! Ready for Business

4

The Team - Hadoop Development

Page 5: Apache Hadoop at Yahoo! Ready for Business

5

Code Contributions

Jan-0

2

Apr-02

Jul-0

2

Oct-02

Jan-0

3

Apr-03

Jul-0

3

Oct-03

Jan-0

4

Apr-04

Jul-0

4

Oct-04

Jan-0

5

Apr-05

Jul-0

5

Oct-05

Jan-0

6

Apr-06

Jul-0

60

500

1000

1500

2000

2500

3000

3500

4000

4500

Apache Hadoop Patches

yahoopowersetotherfacebookcloudera

Patc

hes

Page 6: Apache Hadoop at Yahoo! Ready for Business

6

Hadoop as a Service

Production

Research

Sandbox

99.2 99.3 99.4 99.5 99.6 99.7 99.8 99.9

99.85

99.47

99.69

Availability SLA

0

50,000

100,000

150,000

200,000

250,000

2006 -Qtr1

2006 -Qtr2

2006 -Qtr3

2006 -Qtr4

2007 -Qtr1

2007 -Qtr2

2007 -Qtr3

2007 -Qtr4

2008 -Qtr1

2008 -Qtr2

2008 -Qtr3

2008 -Qtr4

2009 -Qtr1

2009 -Qtr2

2009 -Qtr3

2009 -Qtr4

2010 -Qtr1

2010 -Qtr2

2010 -Qtr3

Total Nodes = 43,936Total Storage = 206 PB

13,687

22,334

7,803

0 5000 10000 15000 20000 25000

Production

Research

Sandbox

Nodes running Hadoop at Yahoo!

Over 43,000 nodes running Hadoop

Total Nodes = 43,936Total Storage = 206 PB

Page 7: Apache Hadoop at Yahoo! Ready for Business

7

Application Patterns

Page 8: Apache Hadoop at Yahoo! Ready for Business

Hadoop Usage at Yahoo!

Today

8

Thou

sand

s of

Ser

vers

Pet

abyt

es

44K Hadoop Servers206PB Raw Hadoop Storage

1M+ Monthy Hadoop Jobs

Page 9: Apache Hadoop at Yahoo! Ready for Business

9

Research to Mission Critical

Page 10: Apache Hadoop at Yahoo! Ready for Business

10

Themes

Economies of scale, per cluster• More users (diverse patterns)• More data• Fewer disruptions• Fewer operatorsBackwards Compatibility• API and semantic consistency• Feature consistency across shifts in deploymentReflect business environment in operational environment• Allow technology to complement users’ workflow• Express relationships between users in rules

Page 11: Apache Hadoop at Yahoo! Ready for Business

11

Evolution of Hadoop at Yahoo!

• Utilization at Scale• Security• Multi-tenancy• Super-size

09/09

04/09

04/11

04/10

Multi-Tenancy

hadoop-0.20 yhadoop-0.20 20.S Multi-tenant

HDFS

Federation

hadoop-next

09/10

CapacityScheduler

Security

4400+ patches on hadoop-0.20!

Apache HadoopYahoo Hadoop

Page 12: Apache Hadoop at Yahoo! Ready for Business

12

Utilization at Scale

04/09

04/11

04/10

Multi-Tenancy

hadoop-0.20 yhadoop-0.20 20.S Multi-tenant

HDFS

Federation

hadoop-next

09/10

CapacityScheduler

Security

Apache HadoopYahoo Hadoop

Page 13: Apache Hadoop at Yahoo! Ready for Business

13

Motivation - CapacityScheduler

• Exploit shared storage – Unified namespace

• Provide compute elasticity– Stop relying on private clusters and Hadoop on Demand (HoD)– Higher utilization at massive scale

Page 14: Apache Hadoop at Yahoo! Ready for Business

14

Hadoop On Demand (HoD)

[0,10)

Rack 0 Rack 1 Rack (N-1) Rack N

User0

...

<--------------------------(HDFS)-------------------------->

MASTER

Job0-0x2000x100

Each map() has a list of preferred hosts. The JobTracker (master) attempts to match each map to the closest host in its sub-cluster.

User1 Job1-0x500

Job1-1x500

Job1-2x500

MASTER

Page 15: Apache Hadoop at Yahoo! Ready for Business

15

CapacityScheduler

• Resource allocation in shared, multi-tenant cluster• A cluster is funded by several business units• Each group gets queue allocations based on their funding

– Guaranteed capacity– Control who can submit jobs to their queues– Set job priorities within their queues

• Support for “High-RAM” jobs

Challenges• Single master (JobTracker)• HoD feature replication

Page 16: Apache Hadoop at Yahoo! Ready for Business

16

CapacityScheduler

[0,10)

Rack 0 Rack 1 Rack (N-1) Rack N

User0

...

<--------------------------(HDFS)-------------------------->

MASTER

Job0-0x2000x100

User1 Job1-0x500

Job1-1x500

Job1-2x500

Page 17: Apache Hadoop at Yahoo! Ready for Business

17

CapacityScheduler - Queues

[0,5)

Rack 0 Rack 1 Rack (N-1) Rack N

JobTracker

...

JobTracker

Maps

Q0 Q1 Q2

Q0

Q1

Q2

Job5-0x1500x10

Job3-0x5000x250

Job3-1x250x1

Job<USER>-<JOB>

Job1-1x500

Job1-2x500

GuaranteedCapacity(Qi) =ClusterCapacityi

Job1-0x500

Job0-0x2000x100

Job2-0x200x100

Page 18: Apache Hadoop at Yahoo! Ready for Business

18

CapacityScheduler - Benefits

• Improved utilization and latency• Isolation in support of user

applications with high resource requirements

• Significantly better utilization of excess capacity– Mix SLA critical and ad-hoc

jobs• Predictable latencies

Job throughput

InputBytes throughput

OutputBytes throughput

0.00 0.50 1.00 1.50 2.00 2.50 3.00

Hadoop 20Hadoop 18

Normalized Throughput

936 GB/hr

MapSlot Utilization

ReduceSlot Utilization

0.0% 20.0% 40.0% 60.0% 80.0%

Hadoop 20

Slot Utilization (%)

CS

HoD

Hadoop 18

Page 19: Apache Hadoop at Yahoo! Ready for Business

19

Security

04/09

04/11

04/10

Multi-Tenancy

hadoop-0.20 yhadoop-0.20 20.S Multi-tenant

HDFS

Federation

hadoop-next

09/10

CapacityScheduler

Security

Apache HadoopYahoo Hadoop

Page 20: Apache Hadoop at Yahoo! Ready for Business

20

Motivation - Security

• Revenue bearing applications• Strong security for data on multi-tenant clusters

– Enable sharing clusters between disjoint kinds of users– Larger namespaces with diverse datasets

• Auditing– Access to data– Access and change management

Page 21: Apache Hadoop at Yahoo! Ready for Business

21

Secure Hadoop

• Kerberos based strong authentication– Client-based authentication introduced in hadoop-0.16 (2007)– Authenticate RPC and HTTP connections

• Multiple person-years of development• Integration with existing security mechanisms in Yahoo• Authorization

– Use HDFS Authorization– Add MapReduce Authorization

• CapacityScheduler and Job/Task log ACLs

Page 22: Apache Hadoop at Yahoo! Ready for Business

22

Multi-Tenancy

04/09

04/11

04/10

Multi-Tenancy

hadoop-0.20 yhadoop-0.20 20.S Multi-tenant

HDFS

Federation

hadoop-next

09/10

CapacityScheduler

Security

Apache HadoopYahoo Hadoop

Page 23: Apache Hadoop at Yahoo! Ready for Business

23

Motivation – Multi-Tenancy

• Growing demand for consolidation, unified namespaces– Economics of scale and operability– Several clusters of 4k nodes each

• Growing demand for stability– Isolation for applications– Shield framework from poorly designed or rogue applications

• Growing “creativity” of users– Features not developed with multi-tenancy or scale in mind– Particularly volatile research clusters

Page 24: Apache Hadoop at Yahoo! Ready for Business

24

Multi-Tenancy

• Limits ensuring availability of the Apache Hadoop service– Plug uptime vulnerabilities in the framework– Enforce best practices (Arun C Murthy)

http://developer.yahoo.com/blogs/hadoop/posts/2010/08/apache_hadoop_best_practices_a/ (http://s.apache.org/CaS)

• Shield clusters from poorly written applications– JT exposed to self-inflicted DDoS attacks, e.g. job counters– NameNode exposed to applications performing too many metadata

operations from the backend tasks• Shield users from one another

– Impose limits on utilization at worker nodes, e.g. memory/disk usage• Metrics and Monitoring• Operability tools for managing large groups of users

Page 25: Apache Hadoop at Yahoo! Ready for Business

25

Super-Sized Hadoop

04/09

04/11

04/10

Multi-Tenancy

hadoop-0.20 yhadoop-0.20 20.S Multi-tenant

HDFS

Federation

hadoop-next

09/10

CapacityScheduler

Security

Apache HadoopYahoo Hadoop

Page 26: Apache Hadoop at Yahoo! Ready for Business

26

Motivation – Super-sized clusters

• Massive storage and processing– Hardware gets more capable per dollar – Continued consolidation for economics and operability

• Unified namespaces. Again.• Better hardware

– More spindles/node and larger disks– 4k 2011 nodes == 12k 2009 nodes– Need to scale the HDFS master (NameNode)

Page 27: Apache Hadoop at Yahoo! Ready for Business

27

Opportunity:Vertical & Horizontal scaling

Horizontal scaling/federation benefits:– Scale– Isolation, Stability, Availability– Flexibility– Other Namenode implementations or non-HDFS namespaces

Namenode Horizontal: Federation

Vertical scalingMore RAM, Efficiency in memory usage

First class archives (tar/zip like)

Partial namespace in main memory

Page 28: Apache Hadoop at Yahoo! Ready for Business

28

HDFS Federation

• Redefine the meaning of a HDFS cluster– Scale horizontally by having multiple NameNodes per cluster

• Striping – Already in production– Shared storage pool– Shared namespace

• Striping – Mount tables in production– Helps availability – Better isolation

• 72 PB raw storage per cluster– 6000 nodes per cluster– 12TB raw, per node

Page 29: Apache Hadoop at Yahoo! Ready for Business

Block (Object) Storage Subsystem

29

NS1 Foreign NS n

... ... NS k

Nam

espa

ce

Datanode 1 Datanode 2 Datanode m... ... ...

Balancer

Block Pools

Pools nPools kPools 1

Blo

ck s

tora

ge

Block (Object) Storage Subsystem• Shared storage provided as pools of blocks• Namespaces (HDFS, others) use one or more block-pools• Note: HDFS has 2 layers today – we are generalizing/extending it.

Page 30: Apache Hadoop at Yahoo! Ready for Business

30

Availability

• Mission critical system• HDFS

– Faster HDFS restarts• Full cluster restart in 75min (down from 3-4 hrs)• NN bounce in 15 minutes• Part of the problem is the NameNode’s size – Federation will help

– Steps towards automated failover• Backup NN (AvatarNode)• Move state off the NN server so we can failover easily

– Federation will significantly improve NN isolation, availability, & stability • Availability for Map-Reduce framework and jobs

– Continued operation across HDFS restarts

Page 31: Apache Hadoop at Yahoo! Ready for Business

31

/* TODO */

• Issues in public clusters– Yahoo controls access to its clusters and its users have a common

hierarchy for resolving issues– Funding for clusters spans business units, but not companies

• Deploying and operating clusters– Yahoo employs a professional operations team experienced in

managing Apache Hadoop and its “quirks”– Not a “turn-key” product. Its largest users customize the platform to

meet their needs. This is a good thing!

Page 32: Apache Hadoop at Yahoo! Ready for Business

32

Questions?

Thanks!