Using Docker to build and test in laptop and Jenkins

Micael Gallegomicael.gallego@urjc.es

@micael_gallegohttps://goo.gl/syUwZp

Download slides

@micael_gallego

micael.gallego@urjc.es

@micaelgallego

About me

developer

University professor

Trainer & Consultor

Consultancy / Training

Cloud ComputingWeb Technologies

Extreme ProgrammingTesting / Git / Jenkins

Software ArchitectureConcurrent Programming

Open source elastic platform for end to end testing

http://codeurjc.es http://elastest.io

Advanced log managementTest orchestration

Test execution comparisonWeb and Mobile testing

Security testingIoT testing

Performance testing

Virtualizacion and Containers

● Developers want to reduce the differences between local, continuous integration and production environments

● Avoiding “It works in my machine” type of problems

Virtualization Containers

Virtualizacion and Containers

● Virtualization– Full fledged Virtual Machine (VirtualBox)

– Developer friendly managed VM (Vagrant)

● Containers– Docker

VirtualBox

● Developed by Oracle (was owned by Sun Microsystems)

● Mostly open source, several free (but closed source) modules

● Windows, Linux and Mac versions

● Advanced Desktop Virtualization– Shared folders host / guest

– Keyboard and mouse advanced integration

– Graphic 3D acceleration

– Webcam

https://www.virtualbox.org/

VirtualBox

VirtualBox

● Manual– Create an empty virtual machine– Connect to a ISO (simulating real CD

devide)– Install a full fledged Operating System– It is time consuming and it is not easy

to share VM between developers

VirtualBox

VirtualBox

Vagrant

● It is a command line utility to manage VMs

● It makes very easy to download and start a new VM (only with a command)

● Allows to provisioning the new VM with command provisioning tools (script, chef, puppet, ansible…)

● VM configuration is specified in a text file, allowing to share it in the git repository

https://www.vagrantup.com/

Vagrant

● How to create a new VM with ubuntu Xenial

● Vagrant manages certificates and networking to make easy to connect to the new VM

● By default, working dir is shared with VM

$ vagrant init ubuntu/xenial64$ vagrant up$ vagrant ssh

containers

https://www.docker.com/

Docker

● With VMs you can have the production environment in your laptop

● But…– VMs takes minutes to start up

– VMs use (waste?) a lot of resources (memory and disk space)

Docker

● In a first look, containers can be considered as “lightweight VMs”

– They contain an isolated environment to run apps

– Start in milliseconds

– They use only the resources it needs

– A container doesn't have a full fledged operating system, only the minimal software to execute apps

Docker

Docker

● Containers and VMs are very different

Virtual Machines ContainersHeavier LighterExecute several processes per Virtual Machine

Usually execute only one process per container

Ssh connection Direct execution in the container (rarely needed)

More isolated using hypervisor Less isolated because are executed using kernel features

Can virtualize Windows over Linux

Linux containers must be executed in linux hosts*

* More on that later

Docker

● To install an application in a linux system you need all dependencies installed

● Can be incompatibilities between applications that need different version of the same dependency

● Docker include in a container all needed software isolated to the rest of the system

Docker

Docker

● Docker containers SO support

– Linux containers● Very mature technology● It can be used in any* linux distribution

– Windows containers● Preliminary technology● It only can be used in a very recent**

Windows Server version

* Kernel version 3.10 or greather. Published in June 2013** Windows Server 2016 (Core and with Desktop Experience), Nano Server, and Windows 10 Professional and Enterprise (Anniversary Edition).

Docker

● You can execute linux containers in any operating system

● It uses virtualization (under the covers) in Mac and Windows

Docker

● Docker Toolbox for Mac and Windows

– It uses VirtualBox as virtualization

– It is not the same development experience than in linux

● Docker for Mac and Windows– Uses native virtualization technology in each

operating system– Only available in new versions of that SOs

DockerDocker concepts

•Docker Image – Basic template for a container (hard disk of

VM)– It contains SO (ubuntu), libs (Java) and app

(webapp.jar)– A container always is started from an image– If you want to start a new container from an

image that is not in your system, it is automatically downloaded from Internet

Docker concepts

•Docker Registry ● Remote service used to store and retrive docker

images● It can hold several versions of the same image ● All versions of the same image are located in the

same repository (like in git)● Docker Hub is a public registry managed by Docker

Inc.● You can buy private repositories in Docker Hub● You can also operate your own private registry

Docker concepts

•Popular repositories in Docker Hub

Docker concepts

•Docker Container – It is the “equivalent” of a Virtual Machine– A container is created from a docker image– When a file is wrote, the image it is not

modified, the container is modified– It can be started, paused or

stopped

Docker concepts

•Docker Engine– Local service used to control docker– Manages images (download, create, pull,

push…)– Manages containers (start, stop, commit...)– It can be used with the docker client or

using its REST API

Docker concepts

•Docker client– Command line interface (CLI) tool to

control docker engine– It is available when docker is installed in a

system to connect to their local docker engine

Docker concepts

•Official documentation– https://docs.docker.com/

•Advanced tutorials– https://docs.docker.com/engine/tutorials/dockerizing/– https://docs.docker.com/engine/tutorials/usingdocker/– https://docs.docker.com/engine/tutorials/dockerimages/– https://docs.docker.com/engine/tutorials/networkingcontainers/– https://docs.docker.com/engine/tutorials/dockervolumes/– https://docs.docker.com/engine/tutorials/dockerrepos/

● Cheat Sheet– https://github.com/wsargent/docker-cheat-sheet/blob/master/README.md

Docker documentation

First steps with docker

Install Docker– Windows:

● Microsoft Windows 10 Professional or Enterprise 64-bit: https://store.docker.com/editions/community/docker-ce-desktop-windows

● Other Windows versions: https://www.docker.com/products/docker-toolbox

– Linux:● Ubuntu: https://store.docker.com/editions/community/docker-ce-server-ubuntu ● Fedora: https://store.docker.com/editions/community/docker-ce-server-fedora ● Debian: https://store.docker.com/editions/community/docker-ce-server-debian ● CentOS: https://store.docker.com/editions/community/docker-ce-server-centos

– Mac:● Apple Mac OS Yosemite 10.10.3 or above: https://store.docker.com/editions/community/docker-ce-desktop-mac

● Older Mac: https://www.docker.com/products/docker-toolbox

First steps with docker

Hands on…

https://github.com/docker/labs/tree/master/beginner

First steps with docker

Testing if docker is correctly installed

$ docker run hello-worldUnable to find image 'hello-world:latest' locallylatest: Pulling from library/hello-world03f4658f8b78: Pull completea3ed95caeb02: Pull completeDigest: sha256:8be990ef2aeb16dbcb9271ddfe2610fa6658d13f6dfb8bc72074cc1ca36966a7Status: Downloaded newer image for hello-world:latest

Hello from Docker.This message shows that your installation appears to be working correctly....

First steps with docker

Running your first container

$ docker run alpine ls -ltotal 48drwxr-xr-x 2 root root 4096 Mar 2 16:20 bindrwxr-xr-x 5 root root 360 Mar 18 09:47 devdrwxr-xr-x 13 root root 4096 Mar 18 09:47 etcdrwxr-xr-x 2 root root 4096 Mar 2 16:20 homedrwxr-xr-x 5 root root 4096 Mar 2 16:20 lib............

First steps with docker

Running your first container

$ docker run alpine ls -l

Command “run”

Creates a new container and start it

First steps with docker

Running your first container

$ docker run alpine ls -l

Image name

alpine is a minimal linux system (4.8Mb). The image is downloaded if

not stored in local machine

First steps with docker

Running your first container

$ docker run alpine ls -l

Command “ls -l”

This command will be executed inside the running container

First steps with docker

Inspecting the downloaded images

$ docker imagesREPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZEalpine latest c51f86c28340 4 weeks ago 1.109 MBhello-world latest 690ed74de00f 5 months ago 960 B

List all images stored in the system

First steps with docker

Executing a container

$ docker run alpine echo "hello from alpine"hello from alpine

Execute the command “echo” inside the container

First steps with docker

Inspecting containers (executing)

$ docker ps -aCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESa6a9d46d0b2f alpine "echo 'hello from alp" 6 minutes ago Exited (0) 6 minutes ago lonely_kilbyff0a5c3750b9 alpine "ls -l" 8 minutes ago Exited (0) 8 minutes ago elated_ramanujanc317d0a9e3d2 hello-world "/hello" 34 seconds ago Exited (0) 12 minutes ago stupefied_mcclintock

It shows containers in the system. All of them has STATUS Exited. These containers are

not currently executing (but using disk space)

First steps with docker

Interactive commands in containers

$ docker run -it alpine /bin/sh/ # lsbin dev etc home lib linuxrc media mnt proc root run sbin sys tmp usr var/ # uname -aLinux 97916e8cb5dc 4.4.27-moby #1 SMP Wed Oct 26 14:01:48 UTC 2016 x86_64 Linux/ # exit$

To execute an interactive command it is necessary to use the option “-it” to connect the console to the

container command

First steps with docker

● Interactive commands in containers

– When you execute a /bin/sh command in a container it offers a “similar” experience than a ssh connection

– Buy there are no ssh server neither ssh client

– It is executing a shell inside the container

First steps with docker

● Managing containers lifecycle

$ docker run -d seqvence/static-site

Option “-d”

Executes the container in background

First steps with docker

● Managing containers lifecycle$ docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESa7a0e504ca3e seqvence/static-site "/bin/sh -c 'cd /usr/" 28 seconds ago Up 26 seconds

Container id is a7a0e504ca3e

This id is used to refer to this container

STATUS is UP

First steps with docker

● Managing containers lifecycle

– Stop running container

– Delete files of the stopped container

$ docker stop a7a0e504ca3e

$ docker rm a7a0e504ca3e

Net services with docker

● Start container exposing a port

docker run --name static-site \ -e AUTHOR="Your Name" -d \ -p 9000:80 seqvence/static-site

Net services with docker

● Start container exposing a port

docker run --name static-site \ -e AUTHOR="Your Name" -d \ -p 9000:80 seqvence/static-site

--name static-site

Specifies a unique name for the container

Net services with docker

docker run --name static-site \ -e AUTHOR="Your Name" -d \ -p 9000:80 seqvence/static-site

-e AUTHOR="Your Name"

Set the environment variable AUTHOR to value “Your Name”

● Start container exposing a port

Net services with docker

docker run --name static-site \ -e AUTHOR="Your Name" -d \ -p 9000:80 seqvence/static-site

-d

Execute container as deamon

● Start container exposing a port

Net services with docker

docker run --name static-site \ -e AUTHOR="Your Name" -d \ -p 9000:80 seqvence/static-site

-p 9000:80

Connects the host port 9000 to the port 80 in the container

● Start container exposing a port

Net services with docker

● Use the service– Open http://127.0.0.1:9000 in a browser in

your host to access 80 port in container

Net services with docker

● Use the service– If you are using Docker Toolbox for Mac or

Windows you can’t use 127.0.0.1 IP

– Then you have to open http://192.168.99.100:9000/ in the browser

$ docker-machine ip default192.168.99.100

Net services with docker

● Container management– Stop and remove the container

– Stop and remove a running container

– Remove all running containers

$ docker rm -f static-site

$ docker stop static-site$ docker rm static-site

$ docker rm -f $(docker ps -a -q)

Managing docker images

● List images in host

Tag is like “version”. Latest is… the latest ;)

$ docker imagesREPOSITORY TAG IMAGE ID CREATED SIZEseqvence/static-site latest 92a386b6e686 2 hours ago 190.5 MBnginx latest af4b3d7d5401 3 hours ago 190.5 MBpython 2.7 1c32174fd534 14 hours ago 676.8 MBpostgres 9.4 88d845ac7a88 14 hours ago 263.6 MBContainous/traefik latest 27b4e0c6b2fd 4 days ago 20.75 MB...

Managing docker images

● Managing versions

– Download a concrete version

– Download latest version

$ docker pull ubuntu:12.04

$ docker pull ubuntu

Managing docker images

● Searching images in DockerHub

Managing docker images

● Searching images in Docker Store

Managing docker images

● Image types

– Base images● Images without a parent image● Examples: Ubuntu, debian, alpine…● Used by Operating Systems

– Child images● Base images plus some additional software● Examples: Nginx, Apache, MySQL...

Managing docker images

● Official vs User images

– Official images● Images created by trusted companies or communities

– User images● Any user can create an account and upload her own images

Managing docker images

● Create your first image– We will create a web application for display

random cat pics using Python

– Create a folder called flask-app

– Download all files in this URL to the folder

https://github.com/docker/labs/tree/master/beginner/flask-app

Managing docker images

● Create your first image– You have all source files for the web

application– But you need Python and Flask to execute

the app– To execute the web application, you will

create a new image with dependencies (Python and Flask) and your application code

– Then you can create a new container to execute your application

Managing docker images

● Dockerfile– File used to describe a new image– Specifies

● Base image● Commands to execute in the image● Files to include in the image from the

project folder● Open ports● Command to execute when start the image

# Select base imageFROM alpine:latest

# Install python and pipRUN apk add ­­update py­pipRUN pip install ­­upgrade pip

# copy app filesCOPY app.py /usr/src/app/COPY templates/index.html /usr/src/app/templates/COPY requirements.txt /usr/src/app/

# install Python modules needed by the appRUN pip install ­­no­cache­dir ­r     /usr/src/app/requirements.txt

# tell the port number the container should exposeEXPOSE 5000

# run the applicationCMD ["python", "/usr/src/app/app.py"]

Managing docker images

● Dockerfile– FROM: Base image

– COPY: Copy files from Dockerfile folder

– RUN execute commands

– EXPOSE: Public ports

– CMD: Command to execute when container is started

https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/

Managing docker images

● Build the image

– In the folder with a Dockerfile execute

– Executed actions● Create a new container with base image● Execute commands and copy app files● Create a new container with the result

$ docker build -t myfirstimage .

Managing docker images

● Run the new image

– Open http://127.0.0.1:9000/ in the browser

– Windows and Mac users with Toolbox use the IP

$ docker run -p 9000:5000 myfirstimage * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)

Managing docker images

● Build the image again

– Change some HTML in templates\index.html

– Create the image again

– The Dockerfile steps without changes are not re-executed (are reused from previous execution)

– The image is created very quickly because only the files copy is perfomed

$ docker build -t myfirstimage .

Volumes

● Volumes– Allow sharing files between host and container– Execute a container to show an nonexistent file

– Create a text file

$ docker run alpine cat /data/file.txtcat: can't open '/data/file.txt': No such file or directory

$ echo "My file" >> file.txt

Volumes

Volumes● Mount a host folder inside a container folder● Host contents replace container contents of

that folder

● Containers can write files in volumes to be available in the host

$ sudo docker run -v $PWD:/data alpine cat /data/file.txtMy file

Volumes

● Volumes– Docker images use volumes to read files

from host

– Official NGINX container can serve host files using http

● Serving current folder files ($PWD)

● Go to http://127.0.0.1:9000/file.txt

https://hub.docker.com/_/nginx/

$ docker run -p 9000:80 -v \ $PWD:/usr/share/nginx/html:ro -d nginx

Volumes

● Volumes– Docker Toolbox for Win or

Mac only allow folders inside user folder to be used as volume

– You can use other folders but have to configure shared folders in VirtualBox

https://hub.docker.com/_/nginx/

•Containers main use cases– Net service

● Executed in background long time...● Used through network● Ex: Databases, web servers...

– Command● Execute a single command and stop● Read and write files from host with volumes● Ex: Java Compiler, jekyll, ffmpeg...

Docker container usage

•Docker for building software– A container can have all needed environment to

execute a developer tool– For example, you can have the compiler and the

test dependencies in a container– You can clone a git repository and execute the

(dockerized) compiler without install any software in your host

Docker for software developers

•Dockerized Java Maven– Clone a maven repo

– Compile and exec tests

Docker for software developers

$ git clone \ https://github.com/jglick/simple-maven-project-with-tests.git

$ cd simple-maven-project-with-tests

$ docker run --rm -v $PWD:/data -w /data maven mvn package

•Dockerized Java Maven

Docker for software developers

$ docker run --rm -v $PWD:/data -w /data maven mvn package

https://hub.docker.com/_/maven/

--rm

Remove container when execution finish

•Dockerized Java Maven

Docker for software developers

$ docker run --rm -v $PWD:/data -w /data maven mvn package

https://hub.docker.com/_/maven/

--w

Working dir for the command

•Dockerized Java Maven

Docker for software developers

$ docker run --rm -v $PWD:/data -w /data maven mvn package

https://hub.docker.com/_/maven/

maven

Official Maven image

•Dockerized Java Maven

Docker for software developers

$ docker run --rm -v $PWD:/data -w /data maven mvn package

https://hub.docker.com/_/maven/

mvn package

Maven command

•Dockerized Java Maven– Jar package is generated in /target folder in host

– As container command is executed as root user (by default), generated files are owned by root.

– Change to your user

Docker for software developers

simple-maven-project-with-tests-1.0-SNAPSHOT.jar

https://hub.docker.com/_/maven/

sudo chown -R username:group target

•Advantages of dockerized dev tools– Avoid several developers having different

versions of such tools– It is very easy to test the same code in different

versions (Java 7, Java 8...)– Reduce tools configuration problems. You can

compile and execute a project easily– The same tools can be executed in development

laptops and also in CI environment

Docker for software developers

Docker for software developers

•Docker in Continuous Integration– If you execute dev tools in containers, it is very

easy to compile, test and package in CI environment

– Only have to execute the same command in laptop and CI environment

– If a tool changes, only have to change the command, it is not necessary to install anything

Docker in CI servers

https://jenkins.io/

•Docker in Continuous Integration

Docker in CI servers

● Jenkins installation

– You need Java

– Go to https://jenkins.io/

– Download

– Download LTS Release

– Generic Java Package (.war)

Docker in CI servers

Jenkins installation

Jenkins installation

New admin account

Jenkins installation

Jenkins installation

● Create new Jenkins job– Create a job with pipeline– Pipeline:

● Clone git repository● Compile, test and package Java project● Copy test results to Jenkins

Jenkins Job

Creamos una nueva tarea para descargar el proyecto y ejecutar los

tests

Jenkins Job

Jenkins Job

Change command to execute

dockerized maven command

node { // Mark the code checkout 'stage'.... stage 'Checkout'

// Get some code from a GitHub repository git url: 'https://github.com/jglick/simple-maven-project-with-tests.git'

// Mark the code build 'stage'.... stage 'Build' // Run the maven build sh "docker run --rm -v \$PWD:/data -w /data maven mvn package" step([$class: 'JUnitResultArchiver', testResults: '**/target/surefire-reports/TEST-*.xml'])}

Pipeline

Jenkins Job

Execute the new job

Jenkins Job

“Checkout” stage

Jenkins Job

Compilation and test stage

Jenkins Job

Successful finished job build (test passed)

Jenkins Job

Clic on build to see details

Jenkins Job

See console output

Jenkins Job

Repository clone and dockerized

maven execution

Jenkins Job

● Advantages of using docker in CI– CI server just need docker installed, nothing

more

– All tools needed by devs are containerized

– Tools are downloaded (and cached) automatically when needed

– Several languages/stacks/dependencies can be used in the same CI server without conflicts

– Sysadmins do not need to give access to CI server to developers (enforcing security)

Docker in CI servers

● Testing different languages with Docker– Testing Node apps with mocha

● https://dzone.com/articles/testing-nodejs-application-using-mocha-and-docker

– Testing C++ apps with Gtest● https://github.com/yutakakinjyo/gtest-cmake-example

– Testing Angular apps ● https://jaxenter.com/build-and-test-angular-apps-using

-docker-132371.html

Docker in CI servers

● Some issues of using docker in CI– Issue: By default project dependencies have to

be downloaded in every build● Solution: Use a host folder as cache

– Issue: Old docker images waste HD space● Solution: Use docker garbage collector (as you

can download images when needed)

– Issue: Window tools can’t be dockerized in linux containers

● Solution: Use portable tools as much as possible ;)

Docker in CI servers

● Docker plugins for Jenkins– Docker Plugin

● https://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin

– Docker build step plugin● https://wiki.jenkins-ci.org/display/JENKINS/Docker+build+step+plugin

– CloudBees Docker Custom Build Environment Plugin● https://wiki.jenkins-ci.org/display/JENKINS/CloudBees+Docker+Custom+Buil

d+Environment+Plugin

– Docker Slaves Plugin● https://wiki.jenkins-ci.org/display/JENKINS/Docker+Slaves+Plugin

– Yet Another Docker Plugin: ● https://wiki.jenkins-ci.org/display/JENKINS/Yet+Another+Docker+Plugin

Docker in CI servers

● Testing tools based on docker– TestContainers

● Define testing dependencies in your JUnit test● https://www.testcontainers.org/

– Dockunit● Test your code in several environments● https://www.npmjs.com/package/dockunit

– Muchas más...

Docker in CI servers

● Conclusions– Docker containers are changing the way we

develop, build, test and ship software

– Containers allow developers to use the same dev tools and execute the project in the same environment

– Containers ease the configuration and share of CI servers

– If Continuous Integration is easier to use, more projects will use it and more test will be executed

Docker in CI servers

Thanks!Do you have any

question?