scaling marketplace to 10,000 add-ons - arun bhalla
DESCRIPTION
In two years, we've transformed Atlassian Marketplace from a simple Rails application to a highly available SaaS service with thousands of add-ons and millions of dollars in transactions. Marketplace team lead Arun Bhalla will guide you through lessons learned building Scala web applications that scale.TRANSCRIPT
#atlassian
ARUN BHALLA • DEVELOPMENT TEAM LEAD • ATLASSIAN
Scaling Marketplace to 10,000 Add-Ons
Marketplace roles
• Add-on search• Add-on listings• Ratings and reviews• Email notifications• Update checks• Adding new releases
• Hosting binaries• Pricing• License reporting• Sales reporting• Vendor management• Add-on approvals
24,938,198Requests per day
13,462Requests per second at peak
Who’s this guy?
In the beginning
Website
Server
Database
httpdTomcat
Groovy on Grails
Java/J2EE Hibernate
PostgreSQL
Evolution
Website
Server
Database
httpdTomcat
Groovy/Grails
Java/J2EE Hibernate
PostgreSQL
Website
Database
httpd
Scala
MongoDB
I’ve got 99 problems, but a null ain’t one.
The adventure begins
Website
MongoDB
httpd
MongoDB
Crowd
The homepage is slow!
You can’t improve what you don’t
measure.
Profiling with YourKit
Optimize data retrieval for cases.
Cache in HTTP
varnishlog
varnishreplay
varnishtop
varnishstat
Apply the Varnish
Website
MongoDB
httpd
MongoDB
Crowd
Varnish
Edge Side Includes (ESI)
Use a CDN
CDN to the rescue
Enter CloudFront
MongoDB MongoDB
Website
httpd
Crowd
VarnishCloudFront
Flat response time
Monitor everything
Pingdom
New Relic
MongoDB Management Service
Nagios + Cacti
The site is slow!
Add more app servers
Double down
MongoDB MongoDB
Website
nginx
Crowd
VarnishCloudFront
Website
Write-through memcached
MongoDB MongoDB
Website
httpd
Crowd
VarnishCloudFront
Website
memcached
Add more database servers
Three’s company
MDB MDB
Website
nginx
Crowd
VarnishCloudFront
Website
memcached
MDB
The site is unavailable during
deploys!
One node…
App
Two nodes…
App1
App2
Three nodes!
App1
App2
App3
Three’s company
MDB MDB
Web
nginx
Crowd
VarnishCloudFront
Web
memcached
MDB
Web
HAProxy
VarnishHAProxy
Bad timing
App1
App2
App3
Coordinated deploys
App1
App2
App3
Deployment Engine
We’re in AWS
MDB MDB
Web
nginx
VarnishCloudFront
Web
memcached
MDB
Web
HAProxy
Varnish
HAProxy
MDB MDB
WebWeb
HAProxy
#atlassianKey takeaways:
• All you need for DevOps is a brain, heart, and courage.• Open-source tools will get you pretty far.• Build your systems with scalability in mind.• Look to the cloud for flexibility.
ARUN BHALLA • DEVELOPMENT TEAM LEAD • ATLASSIAN
Thank you!