PHP at 5000 Requests / SecHootsuite’s Scaling Story

Lead Technical Engineer - Platform@bmonkman

Bill Monkman

Overview - Selected Current Architecture

Users lb1 lb2 lb3 ... Nginx Load balancers

web1 web2 web3 ... Nginx web serversPHP-FPM PHP-FPM PHP-FPM PHP-FPM

mem1 ...

Memcached cluster

master slave

Mysql cluster

master slave

MongoDB cluster

slavemaster

shard1

shard2

geard1 geard2

Gearman cluster

worker1 ... ...

Services

...

• Apache

• PHP

• MySQL

Technologies - at first

Then...

It’s hard to scale MySQL horizontally

Problem

Memcached.

● Distributed cache, cluster of boxes with lots of RAM, trivial to scale

● Cache as much as possible, invalidate only when necessary

● Use cache instead of DB

● No joins - decouple entities (collection caching)

● Twemproxy!

Solution - Caching

• Phil Karlton

“There are only two hard things in Computer Science: cache invalidation and naming things.”

Solution - Caching

SocialNetwork

MvcModelMysql

MvcModelBase

MvcModelBaseCaching

SELECT * FROM member WHERE org_id=888

set individual cache records

set collection cache

Automatic invalidation of collection cache

Solution - Caching

member_1 {data}

member_5 {data}

member_9 {data}

member_org_888 [1,5,9]

It’s hard to scale MySQL horizontally

Now:

● No need to scale MySQL

● Able to serve the whole site on 1 MySQL server

● 500 MySQL SELECTs per second. 50,000 Memcached GETs.

● 99+% hit rate

Solution - Caching

Then...

Need a way to perform asynchronous, distributed tasks using a single-threaded language.

Problem

Gearman.

● Distribute work to other servers to handle (workers also using

PHP, same codebase)

● Precursor to SOA where everything is truly distributed

● Many other solutions, queueing systems.

Solution - Gearman

Solution - Gearman

geard1 geard2

gearworker1 gearworker2 gearworker6

Need a way to perform asynchronous, distributed tasks using a single-threaded language.

Now:● Moved key tasks to Gearman

● Another cluster, scalable separately from web

● Discrete tasks, callable sync or async

Solution - Gearman

Then...

Need to store data with the potential to grow too big to handle effectively with MySQL.

Problem

MongoDB.

● Certain data did not need to be highly relational

● NoSQL DB, many other solutions these days

● Mongo can be a pain, lots of moving parts

● Had to make our own sequencer where auto-incremented ids were

necessary

Solution - MongoDB

Need to store data with the potential to grow too big to handle effectively with MySQL.

Now:● Multiple clusters containing amounts of data that likely would

have crushed MySQL● Billions of rows per collection, many TB of data on disk

Solution - MongoDB

• Apache

• PHP

• MySQL

• Memcached

• Gearman

• MongoDB

Technologies

Then...

With a codebase and an engineering team increasing in size, how do we keep up the pace of development and maintain control of the system?(SVN, big branches, merge hell)

Problem

Dark Launching.

● Wrap code in block with a specific name

● That name will appear in a management page

● Can control whether or not that block is executed by modifying it’s value

● Boolean , random percentage, session-based, member list, organization

list, etc.

Solution - Dark Launching

if (In_Feature::isEnabled(‘TWITTER_ADS’)) {

// execute new code

} else {

// execute old code

}

Solution - Dark Launching

• Control your code

• Limit risk -> raise confidence -> speed up pace of releases

• “Branching in Production”

• Learning happens in Production

Dark Launching - Reasons

With a codebase and an engineering team increasing in size, how do we keep up the pace of development and maintain control of the system?

Now:● Work fast with more confidence

● Huge amount of control over production systems

● Typically 10+ code releases to production per day

● Push-based distribution with Consul

Solution - Dark Launching

Then...

With a rapidly increasing codebase and amount of users / traffic how do we keep visibility into the performance of the code?

Problem

Statsd / Graphite.Logstash / Elasticsearch / Kibana. Sensu

● Statsd for metrics

● Logstash for log events

● Sensu for monitoring / alerting

Solution - Monitoring

Statsd::timing('apiCall.facebookGraph', microtime(true) - $startTime);

Solution - Monitoring

Logger::event('user liked from in-stream', In_Log::CATEGORY_UX, $logData);

Solution - Monitoring

• Visibility into the performance and behaviour of your application

• Iterate upon your code, measure results

• Pairs well with dark launching

• Also systems like New Relic

Solution - Monitoring

With a rapidly increasing codebase and amount of users / traffic how do we keep visibility into the performance of the code?

Now:● Able to watch performance / behaviour in real time.● Able to view important events both in the aggregate or very

granular● Able to control the system and watch the effect of changes

Solution - Monitoring

Optimizations

• Things expand beyond their initial scope

• Case in point: Translations

Optimizations

• Within reason, push work up to users

• Make your users into a distributed processing grid

• e.g. Stream rendering

Optimizations - Push work to users

• Performance is more important than clean code, business reqts (in the instances where they may be mutually exclusive)

• Fine line between future proofing and premature optimization

• Don’t add burdensome processes, but make it easy for your team to do things the right way

• Know your weak spots, protect against abuse

Optimizations - Performance / Risks

Technologies

LinuxNginx

MySQL

PHP-FPM

Jenkins

Scala

MongoDB

GearmanRedis

Akka

Python

Memcached

HAProxy

jQuery

ZeroMQ

RabbitMQBackbone

EC2Zend

Docker

Cloudfront CDN

VarnishElasticSearch

Logstash

Zookeeper

Kibana

Statsd/Graphite

Packer

Vagrant

Nagios

VirtualBox

Sensu

SymfonyRiak

Websockets

Comet

Hadoop

Spark/SharkAnsible

Consul

Composer

Git

RedshiftWebpack

With a huge and growing monolithic codebase and over 80 engineers, how to keep scaling in a manageable way?

Problem

SOA.

● Split up the system into independent services which communicate only via APIs

● Teams can work on their own services with encapsulated business logic and have their own

deployment schedules.

● We chose to use Scala/Akka for services, communicating via ZeroMQ

● SOA transition made easier by the “no joins” philosophy

● Tons of work

Solution - SOA

SOM.

● “Service Oriented Monolith”

● When splitting up a monolithic codebase, dependencies are what kill you

● Fulfill dependencies by writing interim services using existing PHP code

● Maintain the contract and future scala services will be drop-in replacements

Solution - SOA

With a huge and growing monolithic codebase and over 130 engineers, how to keep scaling in a manageable way?

Today:● Transitioning to Scala SOA● PHP will still be used as the Façade, a thin layer built on top of

the business logic of the services it interacts with.

Solution - SOA

Conclusion

Thank You!Bill Monkman@bmonkman

More Info:code.hootsuite.com