paradigm shift in ci at paypal with docker and mesos

23
©2015 PayPal Inc. Confidential and proprietary. 15 th February 2017 Running CI at scale with containers at PayPal Sathiya Narayanan

Upload: sathiya-narayanan

Post on 10-Apr-2017

62 views

Category:

Software


2 download

TRANSCRIPT

Page 1: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary.

15th February 2017

Running CI at scale with containers at PayPal

Sathiya Narayanan

Page 2: Paradigm shift in CI at PayPal with Docker and Mesos

© 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 2

At PayPal, we take CI seriously

Page 3: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary. 3

o One Ginormous Jenkins Instance, customized for PayPal

o 40,000 runnable jobso 24,000 builds run each day on Single

Instanceo Stack: Java, Node, Python, C++ o CD : Unit Test, Static analysis and code

coverage, & deployments o One Jenkins master with a pool of a

hundred Jenkins slaveso Custom build system for 50 MLOC in <1

Min

Single Jenkins instance

One Giant Jenkins Butler!

Page 4: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary. 4

o SPOFo Scalability o Change Managemento No freedom for userso Inefficient resource managemento Plugins don’t scale

Limitations with single Jenkins instance

Too much load!

Page 5: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary. 5

o 1:1 modelo Individual VMs running Jenkins for each

Applicationo 2,500+ VMs in useo Each Jenkins instance would have 2

executor on master to perform buildso Freedom for users

Users loved it!

Dedicated VM for each Jenkins Instance

Page 6: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary. 6

Jenkins on Mesos

o Light weight o Long running tasko Marathon scheduler

Jenkins Master

o Provisioned on demand with Jenkins Mesos plugin

o Flavors of slaves based on the application stack

Jenkins slave

Page 7: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary. 7

Optimized Resource Utilization with Mesos

Page 8: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary. 8

Operating cost reduced 10X times

After Mesoso ~2400 CPU’s, o 18 TB, o 7.2 TB RAM

Before Mesoso ~ 24,000 CPU’s, o 180 TB Disk, o 54 TB RAM

Page 9: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary. 9

After:

~2400 CPU’s, 18 TB, 7.2 TB RAM

~300K

2.7 million saving could buy:- A personal Jet for GPI

10X Savings

Before:

~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM

~3 Million

Recovery was slow without Mesos

Page 10: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary. 10

10

After:

~2400 CPU’s, 18 TB, 7.2 TB RAM

~300K

2.7 million saving could buy:- A personal Jet for GPI

10X Savings

Before:

~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM

~3 Million

Instant Recovery with Mesos

Page 11: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary. 11

After:

~2400 CPU’s, 18 TB, 7.2 TB RAM

~300K

2.7 million saving could buy:- A personal Jet for GPI

10X Savings

Before:

~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM

~3 Million

Our workload was too much for Marathon

Page 12: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary. 12

o Zero outages and no customer impact.o Aurora was more stableo Aurora was able to scale to more than

5,000 jobs

Replaced Marathon with Aurora

Page 13: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary. 13

After:

~2400 CPU’s, 18 TB, 7.2 TB RAM

~300K

2.7 million saving could buy:- A personal Jet for GPI

10X Savings

Before:

~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM

~3 Million13

CIAPI

Aurora

Mesos Master

Mesos Cluster

Swift / Object Storage

Zookeeper

Jenkins Master Jenkins Slave

Nginx

Highly available setup

Devxdash

Page 14: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary. 14

Challengeso Inconsistent build environments between user

workstation and CI build system

o Tasks stepping onto each other

o Providing conflicting tool set for different technology stacks

o Different DevOps teams/applications had their own Mesos cluster

o Slower rollout of new tools to the build environment

o Inconsistent build environments within the CI build system

Page 15: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary. 15

Why Docker?

o Task isolation

o Cloud Native

o Eliminates the dependency on the host

o Reproducibility

o Portability

o Immutability

o Homogenous cluster

Page 16: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary. 16

10X Savings

Before:

~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM

~3 Million

Dockerized Jenkins Masters

o Immutable containers for consistency

o Eliminates the need to have tools installed on slave

o Provides task isolation

o Other work loads can co-exist in Mesos cluster

o Easy to cater to conflicting OS / toolset requirements in a polyglot environment

o Easy to roll out updates

o Other work loads can co-exist in Mesos cluster

o Users can bake their own build environment

Dockerized Jenkins Slaves

Jenkins with Docker

Page 17: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary. 17

10X Savings

Before:

~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM

~3 Million

Adoption to Docker containers

BC [ Before Containers ]

AD [ After Docker ]

SSH key / Tokens / Maven credentials

Stored in local disk Injected only during build time and rotating them

Toolsets like JDK, Maven etc.

Installed in host Installed within Docker container

Page 18: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary. 18

Before:

~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM

~3 Million

What is state for Jenkins master ?

o Jenkins war

o Plugins

o Global configuration

o Job configuration

o Build historyo Console outputo Reports like code coverage, test result trend etc.o Archived artifacts

Page 19: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary. 19

Before:

~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM

~3 Million

Backing up Jenkins state

Supervisord

Archive the

directory

Encrypt the data

Store in cloud

storage

Page 20: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary. 20

Before:

~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM

~3 Million

When Jenkins Master container moves to another VM

Untar the files

into Jenkins home

Decrypt the

backup

Check if a backup exists

in cloud storage

Provisioning script [Entry

point for Docker]

Page 21: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary. 21

10X SavingsBefore:

~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM

~3 Million

Use case - Maven build

Total build time - 4 minutesPersisting workspace for subsequent builds

Decreased build timeSource code [Clone]

Artifacts

Time taken 0m27.618s

2m56.000sNumber of files 10570 4456

Total Size 158 M 173 M

Page 22: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary. 22

10X Savings

Before:

~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM

~3 Million

CIAAS with Docker ArchitecturePublic cloud

Private cloud

Page 23: Paradigm shift in CI at PayPal with Docker and Mesos

©2015 PayPal Inc. Confidential and proprietary. 23

After:

~2400 CPU’s, 18 TB, 7.2 TB RAM

~300K

2.7 million saving could buy:- A personal Jet for GPI

Before:

~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM

~3 Million