data infrastructure at linkedin
DESCRIPTION
This talk was given by Kapil Surlaker (Staff Software Engineer @ LinkedIn) at the 28th IEEE International Conference on Data Engineering (ICDE 2012).TRANSCRIPT
Data Infrastructure at LinkedIn Kapil Surlaker http://www.linkedin.com/in/kapilsurlaker @kapilsurlaker
1
Outline
LinkedIn Products Data Ecosystem LinkedIn Data Infrastructure Solutions Next Play
2
LinkedIn By The Numbers
150M + users* ~ 4.2B People Searches in 2011** >2M companies with LinkedIn Company Pages** 16 languages 75% of Fortune 100 Companies use LinkedIn to hire***
* As of February 9th 2012 ** As of December 31st 2011
*** As of September 30th 2011
3
Broad Range of Products & Services
4
5
User Profiles Large dataset Medium writes Very high reads Freshness <1s
6
Communications Large dataset
High writes High reads
Freshness <1s
People You May Know
7
Large dataset Compute intensive
High reads Freshness ~hrs
LinkedIn Today
8
Moving dataset High writes High reads
Freshness ~mins
Outline
LinkedIn Products Data Ecosystem LinkedIn Data Infrastructure Solutions Next Play
9
Three Paradigms : Simplifying the Data Continuum
• Member Profiles
• Company Profiles • Connections • Communications
Online
• Linkedin Today
• Profile Standardization • News • Recommendations • Search • Communications
Nearline
• People You May Know
• Connection Strength • News • Recommendations • Next best idea
Offline
10
Activity that should be reflected immediately
Activity that should be reflected soon
Activity that can be reflected later
LinkedIn Product Architecture
11
LinkedIn Product Architecture
12
LinkedIn Product Architecture
13
Databus : Timeline-Consistent Change Data Capture
LinkedIn Data Infrastructure Solutions
14
Databus at LinkedIn
15
DB
Bootstrap
Capture Changes
On-line Changes
On-line Changes
DB
Consistent Snapshot at U
Consumer 1 Consumer n
Client
Dat
abus
C
lient
Lib
Consumer 1 Consumer n
Dat
abus
C
lient
Lib
Client
Relay Event Win
Databus at LinkedIn
16
DB
Bootstrap
Capture Changes
On-line Changes
On-line Changes
DB
Consistent Snapshot at U
Transport independent of data source: Oracle, MySQL, …
Transactional semantics In order, at least once delivery
Tens of relays Hundreds of sources Low latency - milliseconds
Consumer 1 Consumer n
Client
Dat
abus
C
lient
Lib
Consumer 1 Consumer n
Dat
abus
C
lient
Lib
Client
Relay Event Win
LinkedIn Product Architecture
17
LinkedIn Product Architecture
18
Voldemort: Highly-Available Distributed KV Store
LinkedIn Data Infrastructure Solutions
19
• Pluggable components • Tunable consistency /
availability • Key/value model,
server side “views”
• 10 clusters, 100+ nodes • Largest cluster – 10K+ qps • Avg latency: 3ms • Hundreds of Stores • Largest store – 2.8TB+
Voldemort: Architecture
LinkedIn Product Architecture
21
Kafka: High-Volume Low-Latency Messaging System
LinkedIn Data Infrastructure Solutions
22
LinkedIn Product Architecture
23
Kafka: Architecture
24
WebTier
Topic 1
Broker Tier
Push Events Topic 2
Topic N
Zookeeper Offset Management
Topic, Partition Ownership
Sequential write sendfile
Kaf
ka
Clie
nt L
ib
Consumers
Pull Events Iterator 1
Iterator n
Topic Offset
100 MB/sec 200 MB/sec
Kafka: Architecture
25
WebTier
Topic 1
Broker Tier
Push Events Topic 2
Topic N
Zookeeper Offset Management
Topic, Partition Ownership
Sequential write sendfile
Kaf
ka
Clie
nt L
ib
Consumers
Pull Events Iterator 1
Iterator n
Topic Offset
100 MB/sec 200 MB/sec
Billions of Events, TBs per day 50K+ per sec at peak Inter and Intra-cluster replication End-to-end latency: few seconds
At least once delivery Very high throughput Low latency Durability
LinkedIn Product Architecture
26
Espresso: Indexed Timeline-Consistent Distributed Data Store
LinkedIn Data Infrastructure Solutions
27
Application View
28
Hierarchical data model
Rich functionality on resources Conditional updates Partial updates Atomic counters
Rich functionality within resource groups
Transactions Secondary index Text search
Partitioning
29
Node 3
Node 2
Espresso Partition Layout: Master, Slave
Cluster Manager
Partition: P.1 Node: 1 … Partition: P.12 Node: 3
Database
Node: 1 M: P.1 – Active … S: P.5 – Active …
Cluster Node 1
P.1 P.2
P.4
P.3
P.5 P.6
P.9 P.10
P.5 P.6
P.8
P.7
P.1 P.2
P.11 P.12
P.9 P.10
P.12
P.11
P.3 P.4
P.7 P.8 Master
Slave
3 Storage Engine nodes, 2 way replication
Espresso: System Components
31
Generic Cluster Manager: Helix
• Generic Distributed State Model • Centralized Config Management • Automatic Load Balancing • Fault tolerance • Health monitoring • Cluster expansion and
rebalancing • Espresso, Databus and Search • Open Source Apr 2012 • https://github.com/linkedin/helix
32
Espresso@Linkedin
Launched first application Oct 2011 Open source 2012 Future
– Multi-Datacenter support – Global secondary indexes – Time-partitioned data
33
LinkedIn Product Architecture
34
Acknowledgments
Siddharth Anand, Aditya Auradkar, Chavdar Botev, Vinoth Chandar, Shirshanka Das, Dave DeMaagd, Alex Feinberg, John Fung, Phanindra Ganti, Mihir Gandhi, Lei Gao, Bhaskar Ghosh, Kishore Gopalakrishna, Brendan Harris, Rajappa Iyer, Swaroop Jagadish, Joel Koshy, Kevin Krawez, Jay Kreps, Shi Lu, Sunil Nagaraj, Neha Narkhede, Sasha Pachev, Igor Perisic, Lin Qiao, Tom Quiggle, Jun Rao, Bob Schulman, Abraham Sebastian, Oliver Seeliger, Adam Silberstein, Boris Shkolnik, Chinmay Soman, Subbu Subramaniam, Roshan Sumbaly, Kapil Surlaker, Sajid Topiwala, Cuong Tran, Balaji Varadarajan, Jemiah Westerman, Zach White, Victor Ye, David Zhang, and Jason Zhang
35
Questions?
36