lessons learned: scaling a social network

52
Lessons Learned Scaling a Social Network [email protected] senior data architect O’Reilly MySQL Conference & Expo 2011 2011–04–13

Upload: oleksiy-kovyrin

Post on 14-Oct-2014

102 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Lessons Learned: Scaling a Social Network

Lessons Learned

Scaling a Social [email protected]

senior data architect

O’Reilly MySQL Conference & Expo 20112011–04–13

Page 2: Lessons Learned: Scaling a Social Network

Who?

Page 3: Lessons Learned: Scaling a Social Network

日本

Page 4: Lessons Learned: Scaling a Social Network

UTF-8

Page 5: Lessons Learned: Scaling a Social Network

PostgreSQL

Page 6: Lessons Learned: Scaling a Social Network

community

Page 7: Lessons Learned: Scaling a Social Network

Codd

Page 8: Lessons Learned: Scaling a Social Network

Why?

Page 9: Lessons Learned: Scaling a Social Network

by genderby gender

52% female

48% male

by ageby ageby age

37% 13–17

27% 18–24

13% 25–34

12% 35–44

11% 45+

myYearbook.comcasual social network

founded in 2006

Google Analytics

Page 10: Lessons Learned: Scaling a Social Network

myYearbook Teens and Twitter Survey (August 2009, ~%)myYearbook Teens and Twitter Survey (August 2009, ~%)myYearbook Teens and Twitter Survey (August 2009, ~%)myYearbook Teens and Twitter Survey (August 2009, ~%)

myYearbookmyYearbook FacebookFacebook

70% meet people 80% keep in touch with friends

50% play games 35% meet people

40% keep in touch with friends 30% share photos

35% flirt/date 25% play games

Page 11: Lessons Learned: Scaling a Social Network

comScore Teens Category (April 2009)comScore Teens Category (April 2009)comScore Teens Category (April 2009)comScore Teens Category (April 2009)comScore Teens Category (April 2009)comScore Teens Category (April 2009)

rank site visits (K) uniques (K) minutes (M) page views (M)

1 myYearbook 55,808 4,604 851 1,630

2 MEEZ 8,629 1,407 226 469

3 Zwinky 11,558 3,691 153 108

4Hearst Teen Network

6,940 2,314 53 117

5 Quizilla 7,924 2,058 75 75

Page 12: Lessons Learned: Scaling a Social Network

comScore page viewsJuly 2009

comScore page viewsJuly 2009

comScore page viewsJuly 2009

rank site views (M)

20 GaiaOnline 1,105

21 Chase 1,056

22 ESPN 984

23 myYearbook 953

24 Wikipedia 903

25 Onemanga 840

26 Mapquest 833

27 Foxsports 815

comScore time spentJuly 2009

comScore time spentJuly 2009

comScore time spentJuly 2009

rank site minutes (M)

20 Amazon 806

21 CNN 744

22 GaiaOnline 709

23 Bing 707

24 MSNBC 691

25 myYearbook 678

26 Iwin 670

27 NickJr 665

Page 13: Lessons Learned: Scaling a Social Network

2007100M HTTP requests/month

1 PostgreSQL server

1 Apache/PHP server

static content servers

1 outage per night

Page 14: Lessons Learned: Scaling a Social Network

20082.5G page views

30 PostgreSQL servers

120 Apache/PHP servers

8 ActiveMQ message brokers

99.84% uptime

Page 15: Lessons Learned: Scaling a Social Network

20102.5G HTTP requests/month

Database serversconnection poolers, routers

Web serversApache/PHP, Tornado/Python, lighthttpd

Message brokersActiveMQ, RabbitMQ

Monitoring, R&D, other servers

Page 16: Lessons Learned: Scaling a Social Network

rapid growth

Page 17: Lessons Learned: Scaling a Social Network

resources

Page 18: Lessons Learned: Scaling a Social Network

1

Page 19: Lessons Learned: Scaling a Social Network

API

Page 20: Lessons Learned: Scaling a Social Network

bottleneck

Page 21: Lessons Learned: Scaling a Social Network

time

Page 22: Lessons Learned: Scaling a Social Network

1 CPU Register

1–2 L1 cache

6–10 L2 cache

25–50 main memory (RAM)

10,000,000 hard disk

100,000,000 LAN

1,000,000,000 WAN

Relative data access latencies

Page 23: Lessons Learned: Scaling a Social Network

fsync

Page 24: Lessons Learned: Scaling a Social Network

memory

Page 25: Lessons Learned: Scaling a Social Network

more activity? views/month 2007 100M ➙ 2009 1.5G

more TPSmore serversmore connectionsmore configurationmore pain!

Page 26: Lessons Learned: Scaling a Social Network

reduce TPS!

➙ memcached

1.2 TB

get 140K/s, set 15K/s

pgfouine

Page 27: Lessons Learned: Scaling a Social Network

SkypePL/Proxy

pgBouncer

Page 28: Lessons Learned: Scaling a Social Network

app-server 1

postgres 1

app-server 2 app-server 3

postgres 2

PL/Proxy

less configuration

fewer connections

Page 29: Lessons Learned: Scaling a Social Network

app-server 1

postgres 1

app-server 2 app-server 3

postgres 2 postgres 3 postgres 4

fewer connections!

Page 30: Lessons Learned: Scaling a Social Network

app-server 1

pgbouncer 1

app-server 2 app-server 3

postgres 1

postgres 2 postgres 3 postgres 4

Page 31: Lessons Learned: Scaling a Social Network

app-server 1

pgbouncer 1

app-server 2 app-server 3

postgres 1

internal pgbouncer

postgres 2 postgres 3 postgres 4

Page 32: Lessons Learned: Scaling a Social Network

app-server 1

pgbouncer 1

app-server 2 app-server 3

postgres 1 postgres 2 postgres 3 postgres 4

pgproxy

internal pgbouncer

reduce TPS/server

less configuration

Page 33: Lessons Learned: Scaling a Social Network

app-server 1 app-server 2 app-server 2 app-server n

external pgbouncer tier

app-server 1

app-pgbouncer 1

load balancer

app-server 2

app-pgbouncer 2

app-server 3

app-pgbouncer 3

app-server n

app-pgbouncer n

pgbouncer 3pgbouncer 2pgbouncer 1

pgproxy

internal pgbouncer

postgres 1 postgres 2 postgres 3 postgres 4 postgres n

Page 34: Lessons Learned: Scaling a Social Network

app-server 1 app-server 2 app-server 2 app-server n

external pgbouncer tier

internal pgbouncer tier

app-server 1

app-pgbouncer 1

load balancer

app-server 2

app-pgbouncer 2

app-server 3

app-pgbouncer 3

app-server n

app-pgbouncer n

pgbouncer 3pgbouncer 2pgbouncer 1

pgproxy

load balancer

pgbouncer 4

postgres 1 postgres 2 postgres 3 postgres n

pgbouncer 5

Page 35: Lessons Learned: Scaling a Social Network

app-server 1 app-server 2 app-server 3 app-server n

external pgbouncer tier

pgproxy 2

internal pgbouncer tier

app-server 1

app-pgbouncer 1

load balancer

app-server 2

app-pgbouncer 2

app-server 3

app-pgbouncer 3

app-server n

app-pgbouncer n

load balancer

pgbouncer 3 pgbouncer 2pgbouncer 1 pgbouncer

pgbouncer 4pgbouncer 5

pgproxy pgproxy

postgres 1 postgres 2 postgres 3 postgres n

Page 36: Lessons Learned: Scaling a Social Network

app-server 1 app-server 2 app-server 3 app-server n

external pgbouncer tier

pgproxy 3pgproxy 2pgproxy 1

internal pgbouncer tier

app-server 1

app-pgbouncer 1

load balancer

app-server 2

app-pgbouncer 2

app-server 3

app-pgbouncer 3

app-server n

app-pgbouncer n

pgbouncerpgbouncerpgbouncer

pgproxy

load balancer

pgproxypgproxy

pgbouncer 4pgbouncer 5

postgres 1 postgres 2 postgres 3 postgres n

Page 37: Lessons Learned: Scaling a Social Network

1

Page 38: Lessons Learned: Scaling a Social Network

SOAREST

DB API

Page 39: Lessons Learned: Scaling a Social Network

death

Page 40: Lessons Learned: Scaling a Social Network

failure isa choice

Page 41: Lessons Learned: Scaling a Social Network

network

Page 42: Lessons Learned: Scaling a Social Network

measure

Page 44: Lessons Learned: Scaling a Social Network

learn

Page 45: Lessons Learned: Scaling a Social Network

logicalversus

physical

Page 46: Lessons Learned: Scaling a Social Network

thoughtworddeed

Page 47: Lessons Learned: Scaling a Social Network

Fusion-io

Postgres buffer

kernel buffer

CPU

Fusion-io card

CPU

application buffer

Fusion-io card

CPU

direct IO buffered IO

Page 48: Lessons Learned: Scaling a Social Network

improve

Page 49: Lessons Learned: Scaling a Social Network

automate

Page 50: Lessons Learned: Scaling a Social Network

test

Page 51: Lessons Learned: Scaling a Social Network

Lessons Learned

Scaling a Social [email protected]

Senior Data Architect

O’Reilly MySQL Conference & Expo 20112011–04–13

Page 52: Lessons Learned: Scaling a Social Network

partitionsharding

inter and intra server

roll off old data