kubecon eu 2016: leveraging ephemeral namespaces in a ci/cd pipeline
TRANSCRIPT
![Page 1: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/1.jpg)
LEVERAGINGLEVERAGING
EPHEMERAL NAMESPACESEPHEMERAL NAMESPACES
IN A CI/CD PIPELINEIN A CI/CD PIPELINE
Can Yücel (@canthefason)
Senior Software EngineerKubeCon EU 2016
![Page 2: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/2.jpg)
HighlightsHighlights
Fundamentals of namespaces
Breaking the idea of having separate clusters
Ephemeral namespaces
Talk about some Kubernetes early stage features
Running every single piece as Kubernetes components
![Page 3: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/3.jpg)
NamespacesNamespaces
“ A namespace is a mechanism to partition resourcescreated by users into a logically named group.
~ Kubernetes Docs
![Page 4: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/4.jpg)
Isolation on Different LevelsIsolation on Different Levels
Network level isolation
Access policies
Resource control
![Page 5: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/5.jpg)
Network Level IsolationNetwork Level IsolationLeveraging subdomainsLeveraging subdomains
![Page 6: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/6.jpg)
Access PoliciesAccess Policies{"user":"admin"}{"user":"scheduler", "readonly": true, "resource": "pods"}{"user":"scheduler", "resource": "bindings"}{"user":"proxy", "resource": "services"}{"user":"proxy", "resource": "endpoints"}{"user":"kubelet", "resource": "pods"}{"user":"kubelet", "resource": "nodes"}{"user":"kubelet", "readonly": true, "resource": "services"}{"user":"kubelet", "readonly": true, "resource": "endpoints"}{"user":"kubelet", "resource": "events"}{"user":"bob", "readonly": true, "namespace": "prod"}{"user":"alice", "namespace": "prod"}
policy.jsonlABAC provides much more granularity on policy
management
![Page 7: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/7.jpg)
Resource ControlResource Control
apiVersion: v1kind: ResourceQuotametadata: name: quotaspec: - hard: memory: "1Gi" cpu: 20 pods: 15 services: 5 replicationcontrollers: 10 resourcequotas: 1
Cluster:32 GB RAM, and 16 cores
Team A:20 GB RAM, and 10 cores
Team B:10 GB RAM, and 4 cores
![Page 8: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/8.jpg)
How to Partition?How to Partition?
Environment based partitioningEnvironment based partitioning
qa, stage, production...
System / team based partitioning System / team based partitioning
kube-system, devops, bots
Project based partitioningProject based partitioning
example.com, better-example.com
![Page 9: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/9.jpg)
A Day of a CI/CD PipelineA Day of a CI/CD Pipeline
Provision separate machines for every build
Run your tests on isolated clusters
When all tests are successful tear down the cluster
If it fails keep the cluster up for a while for debugging
Ephemeral Namespaces!Ephemeral Namespaces!
namespaces
namespaces
namespace
![Page 10: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/10.jpg)
Ephemeral Namespaces AreEphemeral Namespaces Are
Isolated environments that are running different versionsof services on top of it
The environments where we run our integrations/e2e tests,and gets dumped when we get the end results
![Page 11: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/11.jpg)
Namespaces with Benefits!Namespaces with Benefits!
Time effective provisioning
Efficient resource utilization
In a CI/CD pipeline, namespaces provide:In a CI/CD pipeline, namespaces provide:
![Page 12: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/12.jpg)
Time Effective ProvisioningTime Effective Provisioning
It takes only a couple of seconds to create all
services
![Page 13: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/13.jpg)
Efficient Resource UtilizationEfficient Resource Utilization
Let your scheduler decide on whichhost you will run your test
instances
![Page 14: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/14.jpg)
Deployment ProcessDeployment Process
1. Run your unit tests
2. Build Docker Image
3. Deploy to sandbox
4. Provision services that you will run your
tests against
5. Run your integration/e2e tests
6. Delete namespace
7. Deploy updated services to staging/prod
Happy Path!
![Page 15: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/15.jpg)
Provisioning Test EnvironmentsProvisioning Test Environments
Identical environments with different versions!
![Page 16: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/16.jpg)
Pods From Different NamespacesPods From Different Namespaces
➜ kubectl get po --namespace=e2e-1NAME READY STATUS RESTARTS AGEmongo-oij3f 1/1 Running 0 10mnginx-44k6p 1/1 Running 0 10mselenium-9bcfc 1/1 Running 0 10mtodo-service-phgrb 1/1 Running 0 10mtodo-service-rbrjl 1/1 Running 0 10m
➜ kubectl get po --namespace=e2e-2NAME READY STATUS RESTARTS AGEmongo-p6g8c 1/1 Running 0 5mnginx-mgdzz 1/1 Running 0 5mselenium-9l81p 1/1 Running 0 5mtodo-service-mt9gh 1/1 Running 0 5mtodo-service-yxo9v 1/1 Running 0 5m
➜ kubectl get po --namespace=e2e-3NAME READY STATUS RESTARTS AGEmongo-llm3x 1/1 Running 0 1mnginx-vvov6 1/1 Running 0 1mnightwatch 1/1 Running 0 34sselenium-g2g1i 1/1 Running 0 1mtodo-service-1k8vc 1/1 Running 0 1mtodo-service-ddfjw 1/1 Running 0 1m
![Page 17: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/17.jpg)
Adding E2E Components as PodsAdding E2E Components as Pods
Selenium serverNightwatch.js scripts
![Page 18: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/18.jpg)
All Tests Passed All Tests Passed ✓✓$ kubectl delete namespace e2e-10
It will dump every Kubernetes componentwithin that namespace!
![Page 19: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/19.jpg)
Test Gets Failed Test Gets Failed 😞
Find a way to connect to the Selenium Server fordebugging
Expose VNC Port 5900
kubectl port-forward selenium :5900 --namespace=e2e-1
![Page 20: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/20.jpg)
Live In ActionLive In Action
![Page 21: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/21.jpg)
How We Use GoCDHow We Use GoCD
Idempotent pipeline stages
Dependency management is handled with fan-inresolution
![Page 22: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/22.jpg)
Evaluating DependenciesEvaluating Dependencies
Text
GO_DEPENDENCY_LABEL_E2E=5.2eedd92GO_DEPENDENCY_LOCATOR_E2E=e2e-tests/5/buildImage/1GO_DEPENDENCY_LABEL_TODO=35.86ca86cGO_DEPENDENCY_LOCATOR_TODO=todo-service/35/deployK8s/1GO_DEPENDENCY_LABEL_NGINX=12.4288a7cGO_DEPENDENCY_LOCATOR_NGINX=nginx/12/deployK8s/1
Each GO_DEPENDENCY variable hasdependant pipeline information
For Provisioning Test Environments: Create all dependencies
For Deployment: Compare versions and call create/rollingupdate
![Page 23: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/23.jpg)
Running Every Piece in PodsRunning Every Piece in PodsNightwatch scripts
kubectl run -i -tty nightwatch --image=canthefason/e2e-tests:$E2E_IMAGE_TAG \--restart=Never --namespace=e2e-$GO_PIPELINE_LABEL
state=$(kubectl get -o template po nightwatch $kubeargs \--template={{.status.phase}})
while [ "$state" == "Running" ]; do sleep 5 echo "waiting for the state" state=$(kubectl get -o template po nightwatch $kubeargs \ --template={{.status.phase}})done
echo "State: $state"if [ "$state" == "Failed" ]; then exit 1fi
![Page 24: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/24.jpg)
Running Every Piece in PodsRunning Every Piece in PodsSelenium manifest
apiVersion: v1kind: ReplicationControllermetadata: name: seleniumspec: replicas: 1 selector: app: selenium template: metadata: name: selenium labels: app: selenium spec: volumes: - name: shm hostPath: path: /dev/shm containers: - name: selenium image: selenium/standalone-chrome-debug:2.52.0 ports: - containerPort: 4444 - containerPort: 5900 imagePullPolicy: Always volumeMounts: - name: shm mountPath: /dev/shm
![Page 25: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/25.jpg)
Health CheckersHealth Checkers
Text
curl -k --retry 10 --retry-delay 5 -v \ https://$KUBE_HOST/api/v1/proxy/namespaces/sandbox/services/todo/ping
curl -k --silent --output /dev/stderr --write-out "%{http_code}" -v \https://$KUBE_HOST/api/v1/proxy/namespaces/sandbox/services/todo/ping
if [ "$STATUSCODE" -ne "200" ]; then if [ "$rcExist" != "ReplicationController" ]; then kubectl delete -f scripts/rc.yml $kubeargs fi exit 1fi
![Page 26: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/26.jpg)
Future WorkFuture Work
Scale down the pods when the namespace is idle
Automatically delete namespaces that are olderthan certain age
Build a Selenium Grid infrastructure and utilizeSelenium Agents among the namespaces
![Page 27: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/27.jpg)
TakeawaysTakeaways
Never ever expose your Apiserver 8080 port!
Think twice before defining your ssh keys assecrets!
Make sure that you properly setup kubeletgarbage collectors
--maximum-dead-containers=100 --maximum-dead-containers-per-container=2 --minimum-container-ttl-duration=1m0s
![Page 28: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/28.jpg)
LinksLinks
http://github.com/canthefason/kubecon
https://github.com/kubernetes/contrib
![Page 29: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/29.jpg)
Thanks ToThanks To
Kubernetes Team
LaunchPad Central
Quest Henkart
UK Consulate in NY...
![Page 30: KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline](https://reader034.vdocuments.us/reader034/viewer/2022051710/58f12cfd1a28aba7338b45b7/html5/thumbnails/30.jpg)
Q & AQ & A
Twitter: @canthefason
GitHub: /canthefason