always be shipping
TRANSCRIPT
Always be shippingMicroservice application Continuous Deployment with Docker
Linux Day Roma 2015
Oct 24th 2015
David Rossi e Francesco Uliana
demo code: github.com/francescou/docker-continuous-deployment1 / 66
Who we areFrancesco UlianaTechnologist at CNR - Reti e Sistemi Informativi
Java dev, FP languages, DevOps, IoT
@warrior10111 | www.uliana.it/francesco | github.com/francescou
David RossiSysadmin at CNR - Reti e Sistemi Informativi
GNU\Linux & FOSS evangelist
2 / 66
1 / 66
2 / 66
Software deployment maturity
3 / 66
3 / 66
Software deployment maturity - level 0build on developer machine
little/no SCM usage (e.g. git, svn)
deploy over FTP
4 / 66
4 / 66
Software deployment maturity- level 1building automation tool (e.g. GNU Makefile, Ant, Maven,Rake)
proficient SCM usage (branches, tags)
5 / 66
5 / 66
Software deployment maturity- level 2Unit testing (e.g. jUnit, TestNG)
Continuous Integration (e.g. Jenkins, Bamboo, CircleCI)
Artifact Repository
6 / 66
6 / 66
Software deployment maturity- level 3externalized (and versioned) application configuration
extensive (and up to date) documentation about the releaseprocess
how to scale?
how to diff instances?
7 / 66
7 / 66
Continuous Deployment
8 / 66
8 / 66
Continuous Deploymentbuild software in such a way that it can be released toproduction at any time. (Martin Fowler)
short cycles
software can be reliably released at any time
9 / 66
9 / 66
Key pointsautomated testing
Continuous Integration (CI)
rapidly, reliably and repeatedly push outenhancements/bugfixes at low risk and with minimal manualoverhead
10 / 66
10 / 66
Benefits1. Accelerated Time to Market
2. Improved Productivity and Efficiency
3. Reliable Releases
4. Improved Product Quality
11 / 66
11 / 66
Requirements and constraints1. good knowledge of several tools
2. less "agility"
3. team maturity
12 / 66
12 / 66
Tools of the trade
13 / 66
13 / 66
14 / 66
14 / 66
lightweight
Linux
containers
15 / 66
15 / 66
Diagram source: Docker Inc.
16 / 66
16 / 66
Dockerdeployment of applications inside software containers
layer of abstraction and automation of operating-system-level virtualization
> 25k stars on github
> 1100 contributors
RESTful API
17 / 66
17 / 66
Inside a Docker Container1. your application code
2. runtime
3. system tools
4. system libraries
5. configuration
18 / 66
18 / 66
Hands on
19 / 66
19 / 66
Steps1. Create a Dockerfile
2. Build container image
3. Run the image
20 / 66
20 / 66
Docker Build (1/3)francesco ~/consul-template-nginx $ ls -ltotal 12-rw-r--r-- 1 francesco francesco 498 6 ago 15.19 Dockerfile-rw-r--r-- 1 francesco francesco 507 6 ago 15.43 nginx.conf.tpl-rw-r--r-- 1 francesco francesco 93 6 ago 15.19 README.md
21 / 66
21 / 66
Docker Build (2/3)francesco ~/consul-template-nginx $ cat Dockerfile
FROM nginx:1.9.3
WORKDIR /opt/consul-template-nginx
RUN apt-get update && apt-get install -y curl
RUN curl --location-trusted https://github.com/hashicorp/consul-template/.../consul-template_....tar.gz | tar -xz
COPY nginx.conf.tpl ./
CMD service nginx start && ./consul-template/consul-template -consul consul:8500 -template './nginx.conf.tpl:/etc/nginx/nginx.conf:service nginx reload'
22 / 66
22 / 66
Docker Build (3/3)francesco ~/consul-template-nginx $ docker build -t myapplication:1.0 .
23 / 66
23 / 66
Docker Rundocker run --name my-nginx -p 8080:80 myapplication:1.0
24 / 66
24 / 66
Just a single container?
25 / 66
25 / 66
usually...multiple services
26 / 66
26 / 66
multiple processes
27 / 66
27 / 66
multiple containers!
28 / 66
28 / 66
Case study: web application4 Containers:
1. web frontend (e.g. apache, nginx)
2. web backend
3. cache (e.g. redis)
4. Database (e.g. mysql, postgres)
29 / 66
29 / 66
Docker Compose
30 / 66
30 / 66
31 / 66
31 / 66
docker-compose.ymlweb: image: node volumes: - ./app:/opt/ links: - db ports: - "3000:3000" command: node /opt/app/index.jsdb: image: mongo ports: - "27017:27017"
32 / 66
32 / 66
More complex docker-compose.ymlpostgresql: image: sameersbn/postgresql:9.4-3 environment: - DB_USER=gitlab - DB_PASS=password - DB_NAME=gitlabhq_production volumes: - /srv/docker/gitlab/postgresql:/var/lib/postgresqlgitlab: image: sameersbn/gitlab:8.0.4 links: - redis:redisio - postgresql:postgresql ports: - "10080:80" - "10022:22" environment: - TZ=Asia/Kolkata - GITLAB_TIMEZONE=Kolkata - .............. volumes: - /srv/docker/gitlab/gitlab:/home/git/dataredis: image: sameersbn/redis:latest volumes: - /srv/docker/gitlab/redis:/var/lib/redis
33 / 66
33 / 66
Manual Docker Compose deployment
docker-compose up -d
34 / 66
34 / 66
...but(few seconds) downtime!
you still need manual intervention
35 / 66
35 / 66
Automated application Deploymentcontinuous integration tool (e.g. Jenkins) should performdocker-compose up -d on each "release" build
36 / 66
36 / 66
Docker Compose UI
37 / 66
37 / 66
38 / 66
38 / 66
Docker Compose UIgithub.com/francescou/docker-compose-ui
docker-compose REST API wrapper
open source (MIT)
93 stars, 12 forks
39 / 66
39 / 66
Continuous Deployment Lifecycle(dev perspective)git clone myrepo.gitgit co -b awesomefeatureimplement featuremerge requestmerge request acceptance triggers Jenkins Build
mvn clean installdocker build -t application:1.1curl -X POST http://docker:5000/api/v1/projects --data'{"id":"myapp"}' -H'Content-type: application/json'
40 / 66
40 / 66
Zero downtime deployments
41 / 66
41 / 66
42 / 66
42 / 66
Live Demosee github.com/francescou/docker-continuous-deployment
43 / 66
43 / 66
Once in productionmonitoring
alerting
log analysis
44 / 66
44 / 66
HA, a philosophy...
45 / 66
45 / 66
by xkcd.com
# Last step...yum erase openssh-server
46 / 66
46 / 66
Habits are changing...Sysadmin scope of interest becomes broader
OS is the last part, like installing java
Sysadmin needs developers procedural skills
Keyword: Automation and Planning
47 / 66
47 / 66
Production1. Provisioning Bare Metal
2. Hardware Lifecycle management
The sysadmin must support different vendors andheterogeneous tecnologies
Use abstraction from hardware (VMs help a lot!)
Avoid lock-in tecnologies !!!
48 / 66
48 / 66
Service must be Continuous1. Automation and planning
2. Monitoring and alerting
3. Backup & Restore
4. Failure (Hardware & Software) solving
49 / 66
49 / 66
Context Separation1. Production, Staging, Development, ...
For the sysadmin... all is CRITICALDevelopers does't feel happy when their builds fail
50 / 66
50 / 66
Planning1. Scheduled to... NOW!
Time is a thing you cannot buy in black market!
Infrastructure automation gives flexibility shortening timeto production
Dev and Ops can plan time for a beer... :)
51 / 66
51 / 66
52 / 66
52 / 66
Opensource tools:1. Provisioning and configuration management service
+
2. Hardware inventory service
53 / 66
53 / 66
Opensource tools:3. Discovery, monitoring, metrics database and alertingservice
+
4. Log service and log parsing dashboard
+
54 / 66
54 / 66
55 / 66
55 / 66
ELK
56 / 66
56 / 66
Zabbix: Cassandra nodes storage
57 / 66
57 / 66
Zabbix: Apache metrics
58 / 66
58 / 66
Machine Lifecycle1. Foreman builds machine (metal or VMs) with dynamictemplates
2. High monitoring automation with Zabbix low discoveryrules and templates
3. Metrics exposed to third party tools (like Grafana)
4. Production: log server configured for docker containers
59 / 66
59 / 66
Enterprise
60 / 66
60 / 66
Habits to avoid (-)(-) Server is a holy place but Ssh is allowed
(-) Environment separation only at physical level (when planned)
(-) Lack of strategy for log services.
(-) Limited sharing of the know-how between teams. Cooperation is limitedto problem solving
(-) Vertical scaling. Expensive hardware. Complexity increases exponentialy
61 / 66
61 / 66
Habits to acquire (+)(+) Ssh is not to be used in production
(+) Containers permits isolation, icc off
(+) Use version control system and build environment for infrastructure
(+) Common tecnologies permit coral planning and better relationshipsbetween Ops and Dev
(+) Horizontal scaling. Commodity hardware. Complexity increases linearly
62 / 66
62 / 66
Google commodity server
63 / 66
63 / 66
Links:Syslog-ng server and compose ui
http://bitfieldconsulting.com/category/tags/devops
http://www.slideshare.net/albertspijkers/linux-containers-and-redhat-7pdf
64 / 66
64 / 66
TakeawaysImmutable containers are awesome
Automate all the things
Abstract from Hardware
65 / 66
65 / 66
Q&A time
66 / 66