scaling marketplace to 10,000 add-ons - arun bhalla

#atlassian

ARUN BHALLA • DEVELOPMENT TEAM LEAD • ATLASSIAN

Scaling Marketplace to 10,000 Add-Ons

Marketplace roles

• Add-on search• Add-on listings• Ratings and reviews• Email notifications• Update checks• Adding new releases

• Hosting binaries• Pricing• License reporting• Sales reporting• Vendor management• Add-on approvals

24,938,198Requests per day

13,462Requests per second at peak

Who’s this guy?

In the beginning

Website

Server

Database

httpdTomcat

Groovy on Grails

Java/J2EE Hibernate

PostgreSQL

Evolution

Website

Server

Database

httpdTomcat

Groovy/Grails

Java/J2EE Hibernate

PostgreSQL

Website

Database

httpd

Scala

MongoDB

I’ve got 99 problems, but a null ain’t one.

The adventure begins

Website

MongoDB

httpd

MongoDB

Crowd

The homepage is slow!

You can’t improve what you don’t

measure.

Profiling with YourKit

Optimize data retrieval for cases.

Cache in HTTP

varnishlog

varnishreplay

varnishtop

varnishstat

Apply the Varnish

Website

MongoDB

httpd

MongoDB

Crowd

Varnish

Edge Side Includes (ESI)

Use a CDN

CDN to the rescue

Enter CloudFront

MongoDB MongoDB

Website

httpd

Crowd

VarnishCloudFront

Flat response time

Monitor everything

Pingdom

New Relic

MongoDB Management Service

Nagios + Cacti

The site is slow!

Add more app servers

Double down

MongoDB MongoDB

Website

nginx

Crowd

VarnishCloudFront

Website

Write-through memcached

MongoDB MongoDB

Website

httpd

Crowd

VarnishCloudFront

Website

memcached

Add more database servers

Three’s company

MDB MDB

Website

nginx

Crowd

VarnishCloudFront

Website

memcached

MDB

The site is unavailable during

deploys!

One node…

App

Two nodes…

App1

App2

Three nodes!

App1

App2

App3

Three’s company

MDB MDB

Web

nginx

Crowd

VarnishCloudFront

Web

memcached

MDB

Web

HAProxy

VarnishHAProxy

Bad timing

App1

App2

App3

Coordinated deploys

App1

App2

App3

Deployment Engine

We’re in AWS

MDB MDB

Web

nginx

VarnishCloudFront

Web

memcached

MDB

Web

HAProxy

Varnish

HAProxy

MDB MDB

WebWeb

HAProxy

#atlassianKey takeaways:

• All you need for DevOps is a brain, heart, and courage.• Open-source tools will get you pretty far.• Build your systems with scalability in mind.• Look to the cloud for flexibility.

ARUN BHALLA • DEVELOPMENT TEAM LEAD • ATLASSIAN

Thank you!