efficiency blast with containers at paypal · o long running task o marathon scheduler ©2015...
TRANSCRIPT
Efficiency Blast with Containers at PayPal
Manivannan Selvaraj and Soma Shekar Oruganti
© 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 2
At PayPal, we take CI seriously
©2015 PayPal Inc. Confidential and proprietary. 3
o One Ginormous Jenkins Instance, customized for PayPal
o 40,000 runnable jobs
o 24,000 builds run each day on Single Instance
o Stack: Java, Node, Python, C++
o CD : Unit Test, Static analysis and code coverage, & deployments
o One Jenkins master with a pool of a hundred Jenkins slaves
o Custom build system for 50 MLOC in <1 Min
Single Jenkins instance
One Giant Jenkins Butler!
©2015 PayPal Inc. Confidential and proprietary. 4
o SPOF
o Scalability
o Change Management
o No freedom for users
o Inefficient resource management
o Plugins don’t scale
Limitations with single Jenkins instance
Too much load!
©2015 PayPal Inc. Confidential and proprietary. 5
o 1:1 model
o Individual VMs running Jenkins for each Application
o 2,500+ VMs in use
o Each Jenkins instance would have 2 executor on master to perform builds
o Freedom for users
Users loved it!
Dedicated VM for each Jenkins Instance
Jenkins on Mesos
o Light weight
o Long running task
o Marathon scheduler
©2015 PayPal Inc. Confidential and proprietary. 6
Jenkins Master
o Provisioned on demand with Jenkins Mesos plugin
o Flavors of slaves based on the application stack
Jenkins slave
©2015 PayPal Inc. Confidential and proprietary. 7
Optimized Resource Utilization with Mesos
©2015 PayPal Inc. Confidential and proprietary. 8
Operating cost reduced 10X times
After Mesos
o ~2400 CPU’s,
o 18 TB,
o 7.2 TB RAM
Before Mesos
o ~ 24,000 CPU’s,
o 180 TB Disk,
o 54 TB RAM
©2015 PayPal Inc. Confidential and proprietary. 9
After:
~2400CPU’s,18TB,7.2TBRAM
~300K
2.7millionsavingcouldbuy:
- ApersonalJetforGPI
10X Savings
Before:
~24,000CPU’s,180TBDisk,54TBRAM
~3Million
Recovery was slow without Mesos
©2015 PayPal Inc. Confidential and proprietary. 10
10
After:
~2400CPU’s,18TB,7.2TBRAM
~300K
2.7millionsavingcouldbuy:
- ApersonalJetforGPI
10X Savings
Before:
~24,000CPU’s,180TBDisk,54TBRAM
~3Million
Instant Recovery with Mesos
©2015 PayPal Inc. Confidential and proprietary. 11
After:
~2400CPU’s,18TB,7.2TBRAM
~300K
2.7millionsavingcouldbuy:
- ApersonalJetforGPI
10X Savings
Before:
~24,000CPU’s,180TBDisk,54TBRAM
~3Million
Our workload was too much for Marathon
©2015 PayPal Inc. Confidential and proprietary. 12
o Zero outages and no customer impact.
o Aurora was more stable
o Aurora was able to scale to more than 5,000 jobs
Replaced Marathon with Aurora
©2015 PayPal Inc. Confidential and proprietary. 13
After:
~2400CPU’s,18TB,7.2TBRAM
~300K
2.7millionsavingcouldbuy:
- ApersonalJetforGPI
10X Savings
Before:
~24,000CPU’s,180TBDisk,54TBRAM
~3Million13
CIAPI
Aurora
Mesos Master
Mesos Cluster
Swift / Object Storage
Zookeeper
Jenkins Master Jenkins Slave
Nginx
Highly available setup
Challenges
©2015 PayPal Inc. Confidential and proprietary. 14
o Inconsistent build environments between user
workstation and CI build system
o Tasks stepping onto each other
o Providing conflicting tool set for different technology
stacks
o Different DevOps teams/applications had their own
Mesos cluster
o Slower rollout of new tools to the build environment
o Inconsistent build environments within the CI build
system
Why Docker?
o Task isolation
o Cloud Native
o Eliminates the dependency on the host
o Reproducibility
o Portability
o Immutability
o Homogenous cluster
©2015 PayPal Inc. Confidential and proprietary. 15
©2015 PayPal Inc. Confidential and proprietary. 16
10X Savings
Before:
~24,000CPU’s,180TBDisk,54TBRAM
~3Million
Dockerized Jenkins Masters
o Immutable containers for consistency
o Eliminates the need to have tools installed on slave
o Provides task isolation
o Other work loads can co-exist in Mesos cluster
o Easy to cater to conflicting OS / toolset requirements in a polyglot environment
o Easy to roll out updates
o Other work loads can co-exist in Mesoscluster
o Users can bake their own build environment
Dockerized Jenkins Slaves
Jenkins with Docker
©2015 PayPal Inc. Confidential and proprietary. 17
10X Savings
Before:
~24,000CPU’s,180TBDisk,54TBRAM
~3Million
Adoption to Docker containers
BC [ Before Containers ] AD [ After Docker ]
SSH key / Tokens / Maven credentials
Stored in local disk Injected only during build time and rotating them
Toolsets like JDK, Maven etc.
Installed in host Installed within Docker container
©2015 PayPal Inc. Confidential and proprietary. 18
10X Savings
Before:
~24,000CPU’s,180TBDisk,54TBRAM
~3Million
Use case - Maven build
Total build time - 4 minutesPersisting workspace for subsequent builds
Decreased build time
Source code [Clone]
Artifacts
Time taken 0m27.618s 2m56.000s
Number of files 10570 4456
Total Size 158 M 173 M
©2015 PayPal Inc. Confidential and proprietary. 19
10X Savings
Before:
~24,000CPU’s,180TBDisk,54TBRAM
~3Million
CIAAS with Docker Architecture
©2015 PayPal Inc. Confidential and proprietary. 20
After:
~2400CPU’s,18TB,7.2TBRAM
~300K
2.7millionsavingcouldbuy:
- ApersonalJetforGPI
10X Savings
Before:
~24,000CPU’s,180TBDisk,54TBRAM
~3Million
DEMO
©2015 PayPal Inc. Confidential and proprietary. 21
After:
~2400CPU’s,18TB,7.2TBRAM
~300K
2.7millionsavingcouldbuy:
- ApersonalJetforGPI
Before:
~24,000CPU’s,180TBDisk,54TBRAM
~3Million