ci and cd at scale: scaling jenkins with docker and apache mesos
TRANSCRIPT
CI AND CD AT SCALE
SCALING JENKINS WITH DOCKERAND APACHE MESOS
Carlos Sanchez
@csanchez csanchez.org
Watch online at carlossg.github.io/presentations
ABOUT MESenior So�ware Engineer @ CloudBees
Contributor to the Jenkins Mesos plugin and the JavaMarathon client
Author of Jenkins Kubernetes plugin
Long time OSS contributor at Apache, Eclipse, Puppet,…
OUR USE CASE
Scaling JenkinsYour mileage may vary
SCALING JENKINSTwo options:
More build agents per masterMore masters
SCALING JENKINS: MORE BUILDAGENTS
Pros
Multiple plugins to add more agents, even dynamically
Cons
The master is still a SPOFHandling multiple configurations, plugin versions,...There is a limit on how many build agents can beattached
SCALING JENKINS: MORE MASTERSPros
Different sub-organizations can self service and operateindependently
Cons
Single Sign-OnCentralized configuration and operation
CLOUDBEES JENKINS ENTERPRISE EDITIONCloudBees Jenkins Operations Center
CLOUDBEES JENKINS PLATFORM - PRIVATESAAS EDITION
The best of both worlds
CloudBees Jenkins Operations Center with multiple masters
Dynamic build agent creation in each master
ElasticSearch for Jenkins metrics and Logstash
BUT IT IS NOT TRIVIAL
A 2000 JENKINS MASTERS CLUSTER
A 2000 JENKINS MASTERS CLUSTER3 Mesos masters (m3.xlarge: 4 vCPU, 15GB, 2x40 SSD)317 Mesos slaves (c3.2xlarge, m3.xlarge, m4.4xlarge)7 Mesos slaves dedicated to ElasticSearch: (c3.8xlarge: 32vCPU, 60GB)
12.5 TB - 3748 CPU
Running 2000 masters and ~8000 concurrent jobs
ARCHITECTUREDocker Docker Docker
Isolated Jenkins masters
Isolated build agents and jobs
Memory and CPU limits
How would you design your infrastructure ifyou couldn't login? Ever.
Kelsey Hightower
EMBRACE FAILURE!
CLUSTER SCHEDULINGRunning in public cloud, private cloud, VMs or bare metal
Starting with AWS and OpenStackHA and fault tolerantWith Docker support of course
APACHE MESOS
A distributed systems kernel
ALTERNATIVES
Docker Swarm / Kubernetes
MESOSPHERE MARATHON
For long running Jenkins masters
<1.4 does not scale with the number of apps
App definitions hit the ZooKeeper node limit
TERRAFORM
TERRAFORMresource "aws_instance" "worker" { count = 1 instance_type = "m3.large" ami = "ami-xxxxxx" key_name = "tiger-csanchez" security_groups = ["sg-61bc8c18"] subnet_id = "subnet-xxxxxx" associate_public_ip_address = true tags { Name = "tiger-csanchez-worker-1" "cloudbees:pse:cluster" = "tiger-csanchez" "cloudbees:pse:type" = "worker" } root_block_device { volume_size = 50 } }
TERRAFORMState is managedRuns are idempotentterraform apply
Sometimes it is too automaticChanging image id will restart all instances
Had to fix a number of bugs, ie. retry AWS calls
Preinstall packages: Mesos, Marathon, DockerCached docker imagesOther drivers: XFS, NFS,...Enhanced networking driver (AWS)
MESOS FRAMEWORKStarted with Jenkins Mesos plugin
Means one framework per Jenkins master, does not scale
If master is restarted all jobs running get killed
OUR NEW MESOS FRAMEWORKUsing Netflix Fenzo
Runs under Marathon, exposes REST API that Jenkinsmasters call
Reduce number of frameworksFaster to spawn new build agents because framework isnot startedPipeline durable builds, can survive a restart of the masterDedicated workers for buildsAffinity
STORAGEHandling distributed storage
Servers can start in any host of the cluster
And they can move when they are restarted
Jenkins masters need persistent storage, agents (typically)don't
Supporting EBS (AWS) and external NFS
SIDEKICK CONTAINERA privileged container that manages mounting for other
containers
Can execute commands in the host and other containers
SIDEKICK CONTAINER CASTLERunning in Marathon in each host
"constraints": [ [ "hostname", "UNIQUE" ] ]
A lot of magic happening with nsenter
both in host and other containers
Jenkins master container requests data on startup usingentrypoint
REST call to CastleCastle checks authenticationCreates necessary storage in the backend
EBS volumes from snapshotsDirectories in NFS backend
Mounts storage in requesting containerEBS is mounted to host, then bind mounted intocontainerNFS is mounted directly in container
Listens to Docker event stream for killed containers
CASTLE: BACKUPS AND CLEANUPPeriodically takes snapshots from EBS volumes in AWS
Cleanups happening at different stages and periodically
EMBRACE FAILURE!
PERMISSIONSContainers should not run as root
Container user id != host user id
i.e. jenkins user in container is always 1000 but matchesubuntu user in host
CAVEATSOnly a limited number of EBS volumes can be mounted
Docs say /dev/sd[f-p], but /dev/sd[q-z] seem towork too
Sometimes the device gets corrupt and no more EBSvolumes can be mounted there
NFS users must be centralized and match in cluster and NFSserver
MEMORYScheduler needs to account for container memory
requirements and host available memory
Prevent containers for using more memory than allowed
Memory constrains translate to Docker --memory
WHAT DO YOU THINK HAPPENSWHEN?
Your container goes over memory quota?
WHAT ABOUT THE JVM?
WHAT ABOUT THE CHILDPROCESSES?
CPUScheduler needs to account for container CPU requirements
and host available CPUs
WHAT DO YOU THINK HAPPENSWHEN?
Your container tries to access more than one CPU
Your container goes over CPU limits
Totally different from memory
CPU translates into Docker \-\-cpu-shares
OTHERCONSIDERATIONS
ZOMBIE REAPING PROBLEM
ZOMBIE REAPING PROBLEM
Zombie processes are processes that have terminated buthave not (yet) been waited for by their parent processes.
The init process -- PID 1 -- task is to "adopt" orphaned childprocesses
source
THIS IS A PROBLEM IN DOCKERJenkins build agent run multiple processes
But Jenkins masters too, and they are long running
TINISystemd or SysV init is too heavyweight for containers
All Tini does is spawn a single child (Tini ismeant to be run in a container), and waitfor it to exit all the while reaping zombies
and performing signal forwarding.
PROCESS REAPINGDocker 1.9 gave us trouble at scale, rolled back to 1.8
Lots of defunct processes
NETWORKINGJenkins masters open several ports
HTTPJNLP Build agentSSH server (Jenkins CLI type operations)
NETWORKING: HTTPWe use a simple nginx reverse proxy for
MesosMarathonElasticSearchCJOCJenkins masters
Gets destination host and port from Marathon
NETWORKING: HTTPDoing both
domain based routing master1.pse.example.compath based routing pse.example.com/master1
because not everybody can touch the DNS or get awildcard SSL certificate
NETWORKING: JNLPBuild agents started dynamically in Mesos cluster can
connect to masters internally
Build agents manually started outside cluster get host andport destination from HTTP, then connect directly
NETWORKING: SSHSSH Gateway Service
Tunnel SSH requests to the correct host
Simple configuration needed in clientHost=*.ci.cloudbees.com ProxyCommand=ssh -q -p 22 ssh.ci.cloudbees.com tunnel %h
allows to runssh master1.ci.cloudbees.com
TERRAFORM AWSInstancesKeypairsSecurity GroupsS3 bucketsELBVPCs
AWSResource limits: VPCs, S3 snapshots, some instance sizes
Rate limits: affect the whole account
Retrying is your friend, but with exponential backoff
AWSRunning with a patched Terraform to overcome timeouts
and AWS eventual consistency<?xml version="1.0" encoding="UTF-8"?> <DescribeVpcsResponse xmlns="http://ec2.amazonaws.com/doc/2015-10-01/" <requestId>8f855bob-3421-4cff-8c36-4b517eb0456c</requestld> <vpcSet> <item> <vpcId>vpc-30136159</vpcId> <state>available</state> <cidrBlock>10.16.0.0/16</cidrBlock> ... </DescribeVpcsResponse> 2016/05/18 12:55:57 [DEBUG] [aws-sdk-go] DEBUG: Response ec2/DescribeVpcAttribute Details:--[ RESPONSE] ------------------------------------ HTTP/1.1 400 Bad Request<Response><Errors><Error><Code>InvalidVpcID.NotFound</Code><Message> The vpc ID 'vpc-30136159‘ does not exist</Message></Error></Errors>
TERRAFORM OPENSTACKInstancesKeypairsSecurity GroupsLoad BalancerNetworks
OPENSTACKCustom flavors
Custom images
Different CLI commands
There are not two OpenStack installations that are the same
GRACIAScsanchez.org
csanchez
carlossg