managing capacity @ linkedin · linkedin espresso hadoop collection ingestion processing / store...

Managing Capacity @ LinkedIn

Anuprita Harkare

Site Reliability Engineer, LinkedIn

Agenda

To edit the table….

• Right click your mouse, select “Insert” or ”Delete” on drop down menu

Linkedin’s data footprint

The Data pipeline

Underlying Components

Tools & Metrics

Q&A

Entire Production Footprint

1.6MWInstalled power

2.5KServers

Equinix Chicago1.2K Servers | 800kW

Equinix Los Angeles 1.2K Servers | 800kW

2010

Production Application Footprint

29MWInstalled capacity

95KServers

Richardson, TX (LTX1)30K Servers | 7.2MW

Ashburn, VA (LVA1)34K Servers | 9.2MW

Hillsboro, OR (LOR1)20K Servers | 8MW

Singapore (LSG1)11K Servers | 4.2MW

2017

Storage By the Numbers

170PB

HDFS Storage

4.2PB

MySQL Storage

140TB 1.3PB

Oracle StorageEspresso Storage

1600Kafka Brokers

2T

Kafka messages per day

Linkedin’s pipeline

Gobblin

LumosEspresso

3rd Party Services

Collection

Oracle DB

Tracking

Lumos Landing

Zone

ODS Teradata

Ingestion Processing / Store Reporting

Proprietary tools

Tableau

MicroStrategy

Linkedin

Hadoop

Linkedin’s pipeline

Gobblin

Lumos

3rd Party Services

Oracle DB

Tracking

Lumos Landing

Zone

ODS Teradata Proprietary tools

Collection Ingestion Processing / Store Reporting

MicroStrategy

Tableau

LinkedinEspresso

Hadoop

Linkedin’s pipeline

Gobblin

Lumos

3rd Party Services

Oracle DB

Tracking

Lumos Landing

Zone

ODS Teradata Proprietary tools

LinkedinEspresso

Hadoop

Collection Ingestion Processing / Store Reporting

MicroStrategy

Tableau

EspressoLumos

Landing Zone

Tracking

Linkedin’s pipeline

Gobblin

Lumos

3rd Party Services

Oracle DB

Tracking

Lumos Landing

Zone

ODS Teradata Proprietary tools

LinkedinEspresso

Hadoop

Collection Ingestion Processing / Store Reporting

MicroStrategy

Tableau

EspressoLumos

Landing Zone

Tracking

Linkedin’s pipeline

Gobblin

Lumos

3rd Party Services

Oracle DB

Tracking

Lumos Landing

Zone

ODS Teradata Proprietary tools

LinkedinEspresso

Hadoop

Collection Ingestion Processing / Store Reporting

MicroStrategy

Tableau

EspressoLumos

Landing Zone

Tracking

Capacity planning for Major Components

Espresso

EspressoKey Value SOT

● Distributed Document Store

● Strongly Consistent

● Provide features between NoSQL and RDBMS

Stores LinkedIn’s Member Profile Data.

Espresso

Espresso

EspressoKey Value SOT

Espresso Footprint

237

Databases

4.2PB

Storage Capacity

Espresso Capacity Planning

Organic Growth New Projects/ DBs Unplanned Scenarios

Capacity model : Organic growth

InGraph (Disk Usage)

● Disk utilisation is not always linear

● Anomalies to be ignored

● Not max but percentile based calculation (~98th percentile)

● Quarterly projections are derived over weekly growth trend in previous quarter

● Buffer over projected capacity

● Distributed streaming platform

● Backbone of Linkedin’s data pipeline

● Metrics, member activity data, change data capture and much more

Kafka Footprint

Kafka Servers

2T

Kafka Events per Day

1600

Kafka Capacity Planning

● Disk usage bound

● Cap disk utilisation at 60%

● Horizontally scalable

● Configurable retention

● Kafka rebalances to keep utilization under 40%

Kcap - Kafka Capacity Auditor

● Distributed storage and processing framework

● It can store any kind of data

● Write once read many

Hadoop

Hadoop Footprint

170PB10K

Hadoop Nodes HDFS Storage

3.6K

Hadoop Users

145K

YARN jobs/per day

Hadoop Capacity Planning

Disk Utilization Memory Utilization CPU Utilization

Hadoop capacity management tools

Thank You

Q&A