hado "ops" or had "oops"

Proprietary amp Confidential Copyright copy 2014

Hadorsquoopsrsquo

or

Hadrsquooopsrsquo 1

Wersquore Hiringrocketfuelcomcareers

Kishore Kumar YellamrajuAbhijit Pol


The Web Is Monetized By Advertising


Delivery Methods

raquoDisplayraquoVideoraquoMobileraquoSocial


6 Ad Served

User

Segment

s

3 Bid

Reques

t

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp

Ad

User Engagement

s

Data Partners

Advertisers

Browser

Some Exchange Partners

Ad Exchange

Optimize

Rocket Fuel Platform

Real-time BidderAutomated Decisions

Models

Refresh

learning

Data

Store

Ads amp

Budget

Model

ScoresEvents

5 RocketfuelWinning Ad


125$211$126$278

$1256$1809$242125

$211$126$278

$0586$2009

125$211$126$278$156

$000

[ + ][ + ]

SitePageGeoWeatherTime of DayBrand AffinityUser

Always buying the best impressions amp serving the best ad

Real Time Bidding and Serving


GoalLeadsamp sales

GoalCoupondownloads

GoalBrandawareness

SitePageGeoWeatherTime of DayBrand AffinityDemo

Impression Scorecard

DemoBrand AffinityTime of DayGeoWeatherSitePageAd PositionIn-marketBehavior

Response


DemoBrand AffinityTime of DayGeoWeatherSitePageAd PositionIn-MarketBehavior

Response X



Response

+100+40-20+20+15+10+40+35

+97

+40-70-20+10+15-25-40-18

+07

+10-10-20+20+10-35-25+10

+14


X


6 Ad Served

User

Segment

s

3 Bid

Reques

t

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp

Ad

User Engagement

s

Data Partners

Advertisers

Browser


Ad Exchange

Optimize



Models

Refresh

learning

Data

Store

Ads amp

Budget

Model

ScoresEvents



5 B

6 B

45 B

Facebook likes

Searches on Google

Bid Requests Considered by Rocketfuel

Requests per day

Throughput


400

100

20

2

Blink of an eye

SF to Tokyo network round trip

One beat of a hummindbirds wing

Look up in Blackbird

Time (ms)

Latency


Architecture and Scale

raquoDatacenters

raquoScale

raquoGrowth

raquoArchitecture


Data Center Expansion

raquoabc


Data Center Design

bull Racks custom built at Rocket Fuel

bull Leased spacebandwidth in colocation facilities

Hadoop Server20 2U servers (85kW)

Bidders40 2-U Twin 2 servers (17kW)


Rocket Fuel Scale

raquo34474 CPU processor cores

ndash2655 servers

ndash1874 Teraflops of computing

raquo188 Terabytes of memory

ndash13X the memory of IBM computer Watson that played Jeopardy

raquo42PB Petabytes of storage

ndash106X the data volume of the entire Library of Congress


Hadoop at Rocket Fuel

raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots

raquo 12K daily MR jobs


200 Servers 1400 Servers

5 PB

41 PB

8x

Growth


Data Architecture 30


Hadoop Setup

QJM ZK Quorum

raquo 6x2TB Disksraquo 2x6 coreraquo 196 GB RAMraquo 2x1G NIC

raquo 12x3TB Disksraquo 2x6 coreraquo 64 GB RAMraquo 10G NIC

raquo same as DNrsquosraquo Dedicated disk

to ZK or JN

JT

Standby NN

ZKFCZKFC

Active NN

DN

TTDN

TT

DN

TT

DN

TTDN

TT

DN

TT


Operations

raquo Maintenance

raquo Performance Tuning

raquo Monitoring

raquo BCP

raquo YARN


Puppet

+

Infradb

Automation is key

Maintenance is Not Easy


Puppet and Infradb

raquo Automate as much as you can

raquo Adding a slave node to Hadoop cluster lt 120 seconds

raquo Bringing up a new Hadoop cluster lt 500 seconds

raquo MR slots are automatically determined based on hardware config

Isnrsquot it cool

Just define once


No issues when cluster is small Problems starts when it grows

Performance Tuning


dfsnamenodehandlercount

dfsimagetransfertimeout

mapredreduceparallelcopies

mapredjobtrackerhandlercount

iosortmbiosortfactor

maxClientCnxns

ZK

HDFS

MR

IMP MAPREDUCE-2026

-XX+UseConcMarkSweepGC

-XXCMSFullGCsBeforeCompaction=1

-XXCMSInitiatingOccupancyFraction=60

ha-timeoutms

JVM

Performance Tuning

mapreducereduceshuffleparallelcopies


MAPREDUCE-5351

MAPREDUCE-5508

keepfailedtaskfiles=true

We Have an Issue


instances of JobInProgressrdquo class = no of users submitted jobs Xmapredjobtrackercompleteuserjobsmaximum

mapredjobtrackercompleteuserjobsmaximum mapredjobtrackerretirejobinterval

mapredjobtrackerretiredjobscachesize

JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring

hadoopnamenodeCallQueueLength hadoopjobtrackerjvmmemheapusedm

Donrsquot fly blind you will crash


MR Workload Monitoring


Network Monitoring

Donrsquot blame network instead monitor it Network Mesh can be mess


Alerting

Monitoring is not enough need better Alerting


Alerts

httphostnameportjmx

qry=Hadoopservice=NameNodename=NameNodeInfo

gtgt Checking whether NN and JT are up is a no brainer gtgt Reduce alert noise by having summaryaggregate alertsgtgt We heavily rely on custom scripts that query jmx for NN and JT

qry=hadoopservice=JobTrackername=JobTrackerInfo

NameDirStatuses DeadNodes NumberOfMissingBlocks

qry=Hadoopservice=NameNodename=FSNamesystemState

FSState CapacityRemaining NumDeadDataNodes UnderReplicatedBlocks

Blacklisted TTrsquos jobs slots_used ThreadCount

qry=javalangtype=Memory

Used jvm free jvm etc


MR Workload Alerting

raquo Monitoring MR workload and alertndash In-house tool that use ldquohoudahrdquo ruby gem monitorsndash Long running jobs jobs with more map tasks blacklisted

TTrsquos with more failure counts etchellip

raquo Collect details and auto-restart blacklisted TTrsquosraquo Parse the JT logfile for rouge jobsraquo Parse the JT log and collects all Job related inforaquo White-elephant or hraven could helpraquo Parse the scheduler html page or use metrics page

httpltJT-hostnamegt50030scheduleradvancedhttpltJT-hostnamegt50030metrics


Modeling

OPS

ETL

Ad-hoc

Multi Tenancy


No Scheduler is perfect unless you understand and tune it properly

Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP

raquo BCP Business Continuity Plan

raquo Near real time reporting over 15+ TB of daily data

raquo Freshness of models trained over petabytes of data


Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster

USEUHK Serving Clusters

Research

Ad-hoc Queries

Processed Data


YARN

raquo Resource Manager- Global resource scheduler- Hierarchical queues- Application management

raquo Node Manager- Per-machine agent- Manages life cycle of container- Container resource monitoring

raquo Application Master- Per-application- Manages application scheduling and

task execution


YARN at Rocket FueI

raquo Yarn is in production

raquo 700+ nodes

raquo 31TB RAM 8500 disks 8500 cores

raquo Primary use case Map-Reduce

raquo No more static slots

raquo Tez Spark Storm are in race

YAY


Obligatory ldquowe are hiringrdquo slide

httprocketfuelcomcareers


THANKS

kishorerocketfuelcom

apolrocketfuelcom


The Web Is Monetized By Advertising


Delivery Methods



6 Ad Served

User

Segment

s

3 Bid

Reques

t

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp

Ad

User Engagement

s

Data Partners

Advertisers

Browser


Ad Exchange

Optimize



Models

Refresh

learning

Data

Store

Ads amp

Budget

Model

ScoresEvents



125$211$126$278

$1256$1809$242125

$211$126$278

$0586$2009

125$211$126$278$156

$000

[ + ][ + ]





GoalLeadsamp sales

GoalCoupondownloads

GoalBrandawareness




Response



Response X



Response

+100+40-20+20+15+10+40+35

+97

+40-70-20+10+15-25-40-18

+07

+10-10-20+20+10-35-25+10

+14


X


6 Ad Served

User

Segment

s

3 Bid

Reques

t

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp

Ad

User Engagement

s

Data Partners

Advertisers

Browser


Ad Exchange

Optimize



Models

Refresh

learning

Data

Store

Ads amp

Budget

Model

ScoresEvents



5 B

6 B

45 B

Facebook likes

Searches on Google


Requests per day

Throughput


400

100

20

2

Blink of an eye




Time (ms)

Latency



raquoDatacenters

raquoScale

raquoGrowth

raquoArchitecture



raquoabc


Data Center Design






Rocket Fuel Scale


ndash2655 servers








raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots




5 PB

41 PB

8x

Growth




Hadoop Setup

QJM ZK Quorum




to ZK or JN

JT

Standby NN

ZKFCZKFC

Active NN

DN

TTDN

TT

DN

TT

DN

TTDN

TT

DN

TT


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Puppet

+

Infradb

Automation is key



Puppet and Infradb





Isnrsquot it cool

Just define once



Performance Tuning







maxClientCnxns

ZK

HDFS

MR

IMP MAPREDUCE-2026




ha-timeoutms

JVM

Performance Tuning



MAPREDUCE-5351

MAPREDUCE-5508


We Have an Issue





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


Delivery Methods



6 Ad Served

User

Segment

s

3 Bid

Reques

t

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp

Ad

User Engagement

s

Data Partners

Advertisers

Browser


Ad Exchange

Optimize



Models

Refresh

learning

Data

Store

Ads amp

Budget

Model

ScoresEvents



125$211$126$278

$1256$1809$242125

$211$126$278

$0586$2009

125$211$126$278$156

$000

[ + ][ + ]





GoalLeadsamp sales

GoalCoupondownloads

GoalBrandawareness




Response



Response X



Response

+100+40-20+20+15+10+40+35

+97

+40-70-20+10+15-25-40-18

+07

+10-10-20+20+10-35-25+10

+14


X


6 Ad Served

User

Segment

s

3 Bid

Reques

t

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp

Ad

User Engagement

s

Data Partners

Advertisers

Browser


Ad Exchange

Optimize



Models

Refresh

learning

Data

Store

Ads amp

Budget

Model

ScoresEvents



5 B

6 B

45 B

Facebook likes

Searches on Google


Requests per day

Throughput


400

100

20

2

Blink of an eye




Time (ms)

Latency



raquoDatacenters

raquoScale

raquoGrowth

raquoArchitecture



raquoabc


Data Center Design






Rocket Fuel Scale


ndash2655 servers








raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots




5 PB

41 PB

8x

Growth




Hadoop Setup

QJM ZK Quorum




to ZK or JN

JT

Standby NN

ZKFCZKFC

Active NN

DN

TTDN

TT

DN

TT

DN

TTDN

TT

DN

TT


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Puppet

+

Infradb

Automation is key



Puppet and Infradb





Isnrsquot it cool

Just define once



Performance Tuning







maxClientCnxns

ZK

HDFS

MR

IMP MAPREDUCE-2026




ha-timeoutms

JVM

Performance Tuning



MAPREDUCE-5351

MAPREDUCE-5508


We Have an Issue





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


6 Ad Served

User

Segment

s

3 Bid

Reques

t

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp

Ad

User Engagement

s

Data Partners

Advertisers

Browser


Ad Exchange

Optimize



Models

Refresh

learning

Data

Store

Ads amp

Budget

Model

ScoresEvents



125$211$126$278

$1256$1809$242125

$211$126$278

$0586$2009

125$211$126$278$156

$000

[ + ][ + ]





GoalLeadsamp sales

GoalCoupondownloads

GoalBrandawareness




Response



Response X



Response

+100+40-20+20+15+10+40+35

+97

+40-70-20+10+15-25-40-18

+07

+10-10-20+20+10-35-25+10

+14


X


6 Ad Served

User

Segment

s

3 Bid

Reques

t

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp

Ad

User Engagement

s

Data Partners

Advertisers

Browser


Ad Exchange

Optimize



Models

Refresh

learning

Data

Store

Ads amp

Budget

Model

ScoresEvents



5 B

6 B

45 B

Facebook likes

Searches on Google


Requests per day

Throughput


400

100

20

2

Blink of an eye




Time (ms)

Latency



raquoDatacenters

raquoScale

raquoGrowth

raquoArchitecture



raquoabc


Data Center Design






Rocket Fuel Scale


ndash2655 servers








raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots




5 PB

41 PB

8x

Growth




Hadoop Setup

QJM ZK Quorum




to ZK or JN

JT

Standby NN

ZKFCZKFC

Active NN

DN

TTDN

TT

DN

TT

DN

TTDN

TT

DN

TT


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Puppet

+

Infradb

Automation is key



Puppet and Infradb





Isnrsquot it cool

Just define once



Performance Tuning







maxClientCnxns

ZK

HDFS

MR

IMP MAPREDUCE-2026




ha-timeoutms

JVM

Performance Tuning



MAPREDUCE-5351

MAPREDUCE-5508


We Have an Issue





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


125$211$126$278

$1256$1809$242125

$211$126$278

$0586$2009

125$211$126$278$156

$000

[ + ][ + ]





GoalLeadsamp sales

GoalCoupondownloads

GoalBrandawareness




Response



Response X



Response

+100+40-20+20+15+10+40+35

+97

+40-70-20+10+15-25-40-18

+07

+10-10-20+20+10-35-25+10

+14


X


6 Ad Served

User

Segment

s

3 Bid

Reques

t

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp

Ad

User Engagement

s

Data Partners

Advertisers

Browser


Ad Exchange

Optimize



Models

Refresh

learning

Data

Store

Ads amp

Budget

Model

ScoresEvents



5 B

6 B

45 B

Facebook likes

Searches on Google


Requests per day

Throughput


400

100

20

2

Blink of an eye




Time (ms)

Latency



raquoDatacenters

raquoScale

raquoGrowth

raquoArchitecture



raquoabc


Data Center Design






Rocket Fuel Scale


ndash2655 servers








raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots




5 PB

41 PB

8x

Growth




Hadoop Setup

QJM ZK Quorum




to ZK or JN

JT

Standby NN

ZKFCZKFC

Active NN

DN

TTDN

TT

DN

TT

DN

TTDN

TT

DN

TT


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Puppet

+

Infradb

Automation is key



Puppet and Infradb





Isnrsquot it cool

Just define once



Performance Tuning







maxClientCnxns

ZK

HDFS

MR

IMP MAPREDUCE-2026




ha-timeoutms

JVM

Performance Tuning



MAPREDUCE-5351

MAPREDUCE-5508


We Have an Issue





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


GoalLeadsamp sales

GoalCoupondownloads

GoalBrandawareness




Response



Response X



Response

+100+40-20+20+15+10+40+35

+97

+40-70-20+10+15-25-40-18

+07

+10-10-20+20+10-35-25+10

+14


X


6 Ad Served

User

Segment

s

3 Bid

Reques

t

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp

Ad

User Engagement

s

Data Partners

Advertisers

Browser


Ad Exchange

Optimize



Models

Refresh

learning

Data

Store

Ads amp

Budget

Model

ScoresEvents



5 B

6 B

45 B

Facebook likes

Searches on Google


Requests per day

Throughput


400

100

20

2

Blink of an eye




Time (ms)

Latency



raquoDatacenters

raquoScale

raquoGrowth

raquoArchitecture



raquoabc


Data Center Design






Rocket Fuel Scale


ndash2655 servers








raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots




5 PB

41 PB

8x

Growth




Hadoop Setup

QJM ZK Quorum




to ZK or JN

JT

Standby NN

ZKFCZKFC

Active NN

DN

TTDN

TT

DN

TT

DN

TTDN

TT

DN

TT


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Puppet

+

Infradb

Automation is key



Puppet and Infradb





Isnrsquot it cool

Just define once



Performance Tuning







maxClientCnxns

ZK

HDFS

MR

IMP MAPREDUCE-2026




ha-timeoutms

JVM

Performance Tuning



MAPREDUCE-5351

MAPREDUCE-5508


We Have an Issue





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


6 Ad Served

User

Segment

s

3 Bid

Reques

t

Overview

Publishers

2 Ad Request

1 Page Request

4 Bid amp

Ad

User Engagement

s

Data Partners

Advertisers

Browser


Ad Exchange

Optimize



Models

Refresh

learning

Data

Store

Ads amp

Budget

Model

ScoresEvents



5 B

6 B

45 B

Facebook likes

Searches on Google


Requests per day

Throughput


400

100

20

2

Blink of an eye




Time (ms)

Latency



raquoDatacenters

raquoScale

raquoGrowth

raquoArchitecture



raquoabc


Data Center Design






Rocket Fuel Scale


ndash2655 servers








raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots




5 PB

41 PB

8x

Growth




Hadoop Setup

QJM ZK Quorum




to ZK or JN

JT

Standby NN

ZKFCZKFC

Active NN

DN

TTDN

TT

DN

TT

DN

TTDN

TT

DN

TT


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Puppet

+

Infradb

Automation is key



Puppet and Infradb





Isnrsquot it cool

Just define once



Performance Tuning







maxClientCnxns

ZK

HDFS

MR

IMP MAPREDUCE-2026




ha-timeoutms

JVM

Performance Tuning



MAPREDUCE-5351

MAPREDUCE-5508


We Have an Issue





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


5 B

6 B

45 B

Facebook likes

Searches on Google


Requests per day

Throughput


400

100

20

2

Blink of an eye




Time (ms)

Latency



raquoDatacenters

raquoScale

raquoGrowth

raquoArchitecture



raquoabc


Data Center Design






Rocket Fuel Scale


ndash2655 servers








raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots




5 PB

41 PB

8x

Growth




Hadoop Setup

QJM ZK Quorum




to ZK or JN

JT

Standby NN

ZKFCZKFC

Active NN

DN

TTDN

TT

DN

TT

DN

TTDN

TT

DN

TT


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Puppet

+

Infradb

Automation is key



Puppet and Infradb





Isnrsquot it cool

Just define once



Performance Tuning







maxClientCnxns

ZK

HDFS

MR

IMP MAPREDUCE-2026




ha-timeoutms

JVM

Performance Tuning



MAPREDUCE-5351

MAPREDUCE-5508


We Have an Issue





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


400

100

20

2

Blink of an eye




Time (ms)

Latency



raquoDatacenters

raquoScale

raquoGrowth

raquoArchitecture



raquoabc


Data Center Design






Rocket Fuel Scale


ndash2655 servers








raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots




5 PB

41 PB

8x

Growth




Hadoop Setup

QJM ZK Quorum




to ZK or JN

JT

Standby NN

ZKFCZKFC

Active NN

DN

TTDN

TT

DN

TT

DN

TTDN

TT

DN

TT


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Puppet

+

Infradb

Automation is key



Puppet and Infradb





Isnrsquot it cool

Just define once



Performance Tuning







maxClientCnxns

ZK

HDFS

MR

IMP MAPREDUCE-2026




ha-timeoutms

JVM

Performance Tuning



MAPREDUCE-5351

MAPREDUCE-5508


We Have an Issue





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom



raquoDatacenters

raquoScale

raquoGrowth

raquoArchitecture



raquoabc


Data Center Design






Rocket Fuel Scale


ndash2655 servers








raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots




5 PB

41 PB

8x

Growth




Hadoop Setup

QJM ZK Quorum




to ZK or JN

JT

Standby NN

ZKFCZKFC

Active NN

DN

TTDN

TT

DN

TT

DN

TTDN

TT

DN

TT


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Puppet

+

Infradb

Automation is key



Puppet and Infradb





Isnrsquot it cool

Just define once



Performance Tuning







maxClientCnxns

ZK

HDFS

MR

IMP MAPREDUCE-2026




ha-timeoutms

JVM

Performance Tuning



MAPREDUCE-5351

MAPREDUCE-5508


We Have an Issue





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom



raquoabc


Data Center Design






Rocket Fuel Scale


ndash2655 servers








raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots




5 PB

41 PB

8x

Growth




Hadoop Setup

QJM ZK Quorum




to ZK or JN

JT

Standby NN

ZKFCZKFC

Active NN

DN

TTDN

TT

DN

TT

DN

TTDN

TT

DN

TT


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Puppet

+

Infradb

Automation is key



Puppet and Infradb





Isnrsquot it cool

Just define once



Performance Tuning







maxClientCnxns

ZK

HDFS

MR

IMP MAPREDUCE-2026




ha-timeoutms

JVM

Performance Tuning



MAPREDUCE-5351

MAPREDUCE-5508


We Have an Issue





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


Data Center Design






Rocket Fuel Scale


ndash2655 servers








raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots




5 PB

41 PB

8x

Growth




Hadoop Setup

QJM ZK Quorum




to ZK or JN

JT

Standby NN

ZKFCZKFC

Active NN

DN

TTDN

TT

DN

TT

DN

TTDN

TT

DN

TT


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Puppet

+

Infradb

Automation is key



Puppet and Infradb





Isnrsquot it cool

Just define once



Performance Tuning







maxClientCnxns

ZK

HDFS

MR

IMP MAPREDUCE-2026




ha-timeoutms

JVM

Performance Tuning



MAPREDUCE-5351

MAPREDUCE-5508


We Have an Issue





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


Rocket Fuel Scale


ndash2655 servers








raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots




5 PB

41 PB

8x

Growth




Hadoop Setup

QJM ZK Quorum




to ZK or JN

JT

Standby NN

ZKFCZKFC

Active NN

DN

TTDN

TT

DN

TT

DN

TTDN

TT

DN

TT


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Puppet

+

Infradb

Automation is key



Puppet and Infradb





Isnrsquot it cool

Just define once



Performance Tuning







maxClientCnxns

ZK

HDFS

MR

IMP MAPREDUCE-2026




ha-timeoutms

JVM

Performance Tuning



MAPREDUCE-5351

MAPREDUCE-5508


We Have an Issue





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom



raquo 1400 servers

raquo 15K Disks

raquo 15K Cores

raquo 90 TB

raquo 30K MR slots




5 PB

41 PB

8x

Growth




Hadoop Setup

QJM ZK Quorum




to ZK or JN

JT

Standby NN

ZKFCZKFC

Active NN

DN

TTDN

TT

DN

TT

DN

TTDN

TT

DN

TT


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Puppet

+

Infradb

Automation is key



Puppet and Infradb





Isnrsquot it cool

Just define once



Performance Tuning







maxClientCnxns

ZK

HDFS

MR

IMP MAPREDUCE-2026




ha-timeoutms

JVM

Performance Tuning



MAPREDUCE-5351

MAPREDUCE-5508


We Have an Issue





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom



5 PB

41 PB

8x

Growth




Hadoop Setup

QJM ZK Quorum




to ZK or JN

JT

Standby NN

ZKFCZKFC

Active NN

DN

TTDN

TT

DN

TT

DN

TTDN

TT

DN

TT


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Puppet

+

Infradb

Automation is key



Puppet and Infradb





Isnrsquot it cool

Just define once



Performance Tuning







maxClientCnxns

ZK

HDFS

MR

IMP MAPREDUCE-2026




ha-timeoutms

JVM

Performance Tuning



MAPREDUCE-5351

MAPREDUCE-5508


We Have an Issue





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom




Hadoop Setup

QJM ZK Quorum




to ZK or JN

JT

Standby NN

ZKFCZKFC

Active NN

DN

TTDN

TT

DN

TT

DN

TTDN

TT

DN

TT


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Puppet

+

Infradb

Automation is key



Puppet and Infradb





Isnrsquot it cool

Just define once



Performance Tuning







maxClientCnxns

ZK

HDFS

MR

IMP MAPREDUCE-2026




ha-timeoutms

JVM

Performance Tuning



MAPREDUCE-5351

MAPREDUCE-5508


We Have an Issue





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Puppet

+

Infradb

Automation is key



Puppet and Infradb





Isnrsquot it cool

Just define once



Performance Tuning







maxClientCnxns

ZK

HDFS

MR

IMP MAPREDUCE-2026




ha-timeoutms

JVM

Performance Tuning



MAPREDUCE-5351

MAPREDUCE-5508


We Have an Issue





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


Puppet

+

Infradb

Automation is key



Puppet and Infradb





Isnrsquot it cool

Just define once



Performance Tuning







maxClientCnxns

ZK

HDFS

MR

IMP MAPREDUCE-2026




ha-timeoutms

JVM

Performance Tuning



MAPREDUCE-5351

MAPREDUCE-5508


We Have an Issue





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


Puppet and Infradb





Isnrsquot it cool

Just define once



Performance Tuning







maxClientCnxns

ZK

HDFS

MR

IMP MAPREDUCE-2026




ha-timeoutms

JVM

Performance Tuning



MAPREDUCE-5351

MAPREDUCE-5508


We Have an Issue





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom



Performance Tuning







maxClientCnxns

ZK

HDFS

MR

IMP MAPREDUCE-2026




ha-timeoutms

JVM

Performance Tuning



MAPREDUCE-5351

MAPREDUCE-5508


We Have an Issue





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom







maxClientCnxns

ZK

HDFS

MR

IMP MAPREDUCE-2026




ha-timeoutms

JVM

Performance Tuning



MAPREDUCE-5351

MAPREDUCE-5508


We Have an Issue





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


MAPREDUCE-5351

MAPREDUCE-5508


We Have an Issue





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom





JT OOM


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


Monitoring

Wall of Ops


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


Monitoring






Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom




Network Monitoring



Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


Alerting



Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


Alerts

httphostnameportjmx

















Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom








Modeling

OPS

ETL

Ad-hoc

Multi Tenancy



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom



Scheduling


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


Operations

raquo Maintenance


raquo Monitoring

raquo BCP

raquo YARN


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


BCP





Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


Data BCP Cluster

INW Data

Cluster

US Serving Clusters

EU Serving Clusters

HK Serving Clusters

Modeling

Reporting

User Queries

Amazon Backup

LSV Data

Cluster


Research

Ad-hoc Queries

Processed Data


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


YARN




task execution


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom


YARN at Rocket FueI


raquo 700+ nodes





YAY





THANKS


apolrocketfuelcom





THANKS


apolrocketfuelcom

hado "ops" or had "oops"

Technology

kwproprietary confidential

ad request1

right ad

ad serveduser segments3

design of data centers

real time auctionselecting

best adreal time bidding

bid requestoverviewpublishers