![Page 1: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/1.jpg)
FROM MONOLITH TO DOCKERDISTRIBUTED APPLICATIONS
Carlos Sanchez
@csanchez csanchez.org
![Page 2: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/2.jpg)
ABOUT MESenior So�ware Engineer @ CloudBees
Author of Jenkins Kubernetes plugin
Long time OSS contributor at Apache Maven, Eclipse,Puppet,…
![Page 3: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/3.jpg)
DOCKER DOCKER DOCKER
![Page 4: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/4.jpg)
![Page 5: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/5.jpg)
OUR USE CASE
Scaling JenkinsYour mileage may vary
![Page 6: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/6.jpg)
![Page 7: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/7.jpg)
![Page 8: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/8.jpg)
![Page 9: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/9.jpg)
![Page 10: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/10.jpg)
A 2000 JENKINS MASTERS CLUSTER3 Mesos masters (m3.xlarge: 4 vCPU, 15GB, 2x40 SSD)317 Mesos slaves (c3.2xlarge, m3.xlarge, m4.4xlarge)7 Mesos slaves dedicated to ElasticSearch: (c3.8xlarge: 32vCPU, 60GB)
12.5 TB - 3748 CPU
Running 2000 masters and ~8000 concurrent jobs
![Page 11: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/11.jpg)
ARCHITECTURE
![Page 12: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/12.jpg)
![Page 13: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/13.jpg)
Isolated Jenkins masters
Isolated build agents and jobs
Memory and CPU limits
![Page 14: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/14.jpg)
![Page 15: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/15.jpg)
![Page 16: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/16.jpg)
CLUSTER SCHEDULINGDistribute tasks across a cluster of hosts
Running in public cloud, private cloud, VMs or bare metal
HA and fault tolerant
With Docker support of course
![Page 17: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/17.jpg)
APACHE MESOS
A distributed systems kernel
![Page 18: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/18.jpg)
ALTERNATIVES
Docker Swarm / Kubernetes
![Page 19: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/19.jpg)
MESOSPHERE MARATHON
![Page 20: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/20.jpg)
TERRAFORM
![Page 21: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/21.jpg)
TERRAFORMresource "aws_instance" "worker" { count = 1 instance_type = "m3.large" ami = "ami-xxxxxx" key_name = "tiger-csanchez" security_groups = ["sg-61bc8c18"] subnet_id = "subnet-xxxxxx" associate_public_ip_address = true tags { Name = "tiger-csanchez-worker-1" "cloudbees:pse:cluster" = "tiger-csanchez" "cloudbees:pse:type" = "worker" } root_block_device { volume_size = 50 } }
![Page 22: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/22.jpg)
TERRAFORMState is managedRuns are idempotentterraform apply
Sometimes it is too automaticChanging image id will restart all instances
![Page 23: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/23.jpg)
![Page 24: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/24.jpg)
IF YOU HAVEN'T AUTOMATICALLYDESTROYED SOMETHING BY
MISTAKE,YOU ARE NOT AUTOMATING ENOUGH
![Page 25: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/25.jpg)
STORAGEHandling distributed storage
Servers can start in any host of the cluster
And they can move when they are restarted
![Page 26: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/26.jpg)
DOCKER VOLUME PLUGINSFlockerGlusterFSNFSEBS
![Page 27: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/27.jpg)
KUBERNETESGCE disksFlockerGlusterFSNFSEBS
![Page 28: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/28.jpg)
PERMISSIONSContainers should not run as root
Container user id != host user id
i.e. jenkins user in container is always 1000 but matchesubuntu user in host
![Page 29: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/29.jpg)
MEMORYScheduler needs to account for container memory
requirements and host available memory
Prevent containers for using more memory than allowed
Memory constrains translate to Docker --memory
![Page 30: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/30.jpg)
WHAT DO YOU THINK HAPPENSWHEN?
Your container goes over memory quota?
![Page 31: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/31.jpg)
![Page 32: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/32.jpg)
WHAT ABOUT THE JVM?WHAT ABOUT THE CHILD
PROCESSES?
![Page 33: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/33.jpg)
CPUScheduler needs to account for container CPU requirements
and host available CPUs
![Page 34: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/34.jpg)
WHAT DO YOU THINK HAPPENSWHEN?
Your container tries to access more than one CPU
Your container goes over CPU limits
![Page 35: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/35.jpg)
Totally different from memory
Mesos/Kubernetes CPU translates into Docker --cpu-shares
![Page 36: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/36.jpg)
NETWORKINGMultiple services running in the same ports
Must redirect from random ports in the host
Services running in one host need to access services in otherhosts
![Page 37: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/37.jpg)
NETWORKING: SOFTWARE DEFINEDNETWORKS
Create new custom networks on top of physical networks
Allow grouping containers in subnets
![Page 38: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/38.jpg)
NETWORKING: SOFTWARE DEFINEDNETWORKS
Battlefield: Calico, Flannel, Weave and Docker OverlayNetwork
http://chunqi.li/2015/11/15/Battlefield-Calico-Flannel-Weave-and-Docker-Overlay-Network/
![Page 39: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/39.jpg)
SCALINGNew and interesting problems
![Page 40: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/40.jpg)
AWSResource limits: VPCs, S3 snapshots, some instance sizes
Rate limits: affect the whole account
Retrying is your friend, but with exponential backoff
![Page 41: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/41.jpg)
EMBRACE FAILURE!
![Page 42: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/42.jpg)
JENKINS PLUGINS
![Page 43: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/43.jpg)
JENKINS DOCKER PLUGINSDynamic Jenkins agents with Docker plugin or Yet AnotherDocker Plugin
No support yet for Docker 1.12 Swarm modeAgent image needs to include Java, downloads slave jarfrom Jenkins masterMultiple plugins for different tasks
Docker build and publishDocker build step pluginCloudBees Docker Hub/Registry NotificationCloudBees Docker Traceability
Great pipeline support
![Page 44: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/44.jpg)
![Page 45: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/45.jpg)
![Page 46: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/46.jpg)
![Page 47: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/47.jpg)
![Page 48: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/48.jpg)
JENKINS DOCKER PIPELINEdef maven = docker.image('maven:3.3.9-jdk-8');
stage 'Mirror' maven.pull() docker.withRegistry('https://secure-registry/', 'docker-registry-login'
stage 'Build' maven.inside { sh "mvn -B clean package" }
stage 'Bake Docker image' def pcImg = docker.build("examplecorp/spring-petclinic:${env.BUILD_TAG}"
pcImg.push(); }
![Page 49: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/49.jpg)
JENKINS MESOS PLUGINDynamic Jenkins agents, both Docker and isolatedprocessesAgent image needs to include Java, grabs slave jar fromMesos sandboxCan run Docker commands on the host, outside of Mesos
![Page 50: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/50.jpg)
![Page 51: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/51.jpg)
![Page 52: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/52.jpg)
![Page 53: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/53.jpg)
![Page 54: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/54.jpg)
![Page 55: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/55.jpg)
JENKINS MESOS PLUGINCan use Docker pipelines with some tricks
Need Docker client installedShared docker.sock from hostMount the workspace in the host, visible under same dir
![Page 56: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/56.jpg)
MESOS PLUGIN AND PIPELINEnode('docker') { docker.image('golang:1.6').inside {
stage 'Get sources' git url: 'https://github.com/hashicorp/terraform.git', tag: "v0.6.15"
stage 'Build' sh """#!/bin/bash -e mkdir -p /go/src/github.com/hashicorp ln -s `pwd` /go/src/github.com/hashicorp/terraform pushd /go/src/github.com/hashicorp/terraform make core-dev plugin-dev PLUGIN=provider-aws popd cp /go/bin/terraform-provider-aws . """
stage 'Archive' archive "terraform-provider-aws" } }
![Page 57: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/57.jpg)
JENKINS KUBERNETES PLUGINDynamic Jenkins agents, running as PodsMultiple container support
One jnlp image, others customPipeline support for both agent Pod definition andexecution will be in next version
![Page 58: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/58.jpg)
JENKINS KUBERNETES PIPELINEpodTemplate(label: 'mypod', containers: [ [name: 'jnlp', image: 'jenkinsci/jnlp-slave:alpine', args: '${computer.jnlpmac} ${computer.name}' [name: 'maven', image: 'maven:3-jdk-8', ttyEnabled: true, command: [name: 'golang', image: 'golang:1.6', ttyEnabled: true, command: ]) {
node ('mypod') { stage 'Get a Maven project' git 'https://github.com/jenkinsci/kubernetes-plugin.git' container('maven') { stage 'Build a Maven project' sh 'mvn clean install' }
stage 'Get a Golang project' git url: 'https://github.com/hashicorp/terraform.git' container('golang') { stage 'Build a Go project' sh """ mkdir -p /go/src/github.com/hashicorp ln -s `pwd` /go/src/github.com/hashicorp/terraform cd /go/src/github.com/hashicorp/terraform && make core-dev """ } }
![Page 59: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/59.jpg)
JENKINS PLUGINS RECAPDynamic Jenkins agent creationUsing JNLP slave jar
In complex environments need to use the tunneloption to connect internally
Using the Cloud APINot ideal for containerized workloadAgents take > 1 min to start provision and are keptaroundAgents can provide more than one executor
![Page 60: F R OM MON OL I T H TO DOC K E R DIST R I B U T E D A P P ......CLUSTER SCHEDULING Distribute tasks across a cluster of hosts Running in public cloud, priv ate cloud, VMs or bare metal](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec54fd690768c486a546aca/html5/thumbnails/60.jpg)
СПАСИБОcsanchez.org
csanchez
carlossg