BASLE BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA
HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
Docker BestPracticesfor MicroServices
Thomas Brö[email protected]
@tvdtb
Agenda
Docker Best Practices for MicroServices - Oracle Code2 04.04.2019
1. Docker Best Practices
2. MicroServices in Docker
3. Security
4. Pitfalls
Docker Basics: The two ways to use docker
Docker Best Practices for MicroServices - Oracle Code5 04.04.2019
docker run -d ...
Use Docker as a daemon
Docker isolates resources
Easy reset
docker run --rm -ti \
-v ${PWD}:/work -w /work ...
Use Docker as CLI tool wrapper
Don‘t install software any more, use images
Run everything in docker!
^P^Q to detach (if started using -ti)
^C to stop
Dockerfile: CMD Dockerfile: ENTRYPOINT
Single Responsibility
Docker Best Practices for MicroServices - Oracle Code6 04.04.2019
Docker container
daemon1
1=
Docker Hub https://hub.docker.com
Docker Best Practices for MicroServices - Oracle Code7 04.04.2019
Docker Hub is a great starting point
Most of these sources can be found at github
„docker inspect <image>“ and
„docker history --no-trunc <image>“
show the details of the image
Best Practices
Important images should
controlled by the project team
– everything which is running in
production
– base images
Docker filesystem
Docker Best Practices for MicroServices - Oracle Code8 04.04.2019
Base Linux
5-100 MB Base distribution
Java Installation
90MB JDK 12
Base Image
50MB Runtime/Libraries
Application Image
1-x MB Application
Container
0-x MB Log files ...
Frozen
Image
Layers
Any change goes here
Docker filesystem
Docker Best Practices for MicroServices - Oracle Code9 04.04.2019
Changes apply to the container only
Changes are copy-on write
Opening the file for write access is
sufficient
Initial write to large files seem to block
the container
Best Practices:
Don‘t change image contents at
runtime
➢No software updates
➢Container contains what you expect
➢Less disk usage
Create well-prepared images
➢Better startup time
➢Less disk usage
Docker versions
Docker Best Practices for MicroServices - Oracle Code10 04.04.2019
Docker allows implicit versioning
:latest if no version is given
Hard to determine what‘s in the image
if :latest tag changed
Best Practices:
Use latest only if you don‘t care
Explicitly version your images
Correlate versions to SCM
➢Create tags
➢Use SCM commit id
➢Git history „distance“
FROM nginx
FROM nginx:1.15.5
Docker history (demo)
Docker Best Practices for MicroServices - Oracle Code11 04.04.2019
Handling a 7.74 MB file
IMAGE CREATED CREATED BY SIZE
COMMENT
abf4100168ab About an hour ago /bin/sh -c #(nop) ENV ENTRY2=two 0B
593d5b0661f0 About an hour ago /bin/sh -c #(nop) ENV ENTRY1=one 0B
cc23f0b1af4a About an hour ago /bin/sh -c rm /test2.zip 0B
6867ccb1d83b About an hour ago /bin/sh -c mv /test.zip /test2.zip 7.74MB
56eb7570f1e3 About an hour ago /bin/sh -c #(nop) COPY file:41e18487354e1a3f… 7.74MB
5cb3aa00f899 3 weeks ago /bin/sh -c #(nop) CMD ["/bin/sh"] 0B
<missing> 3 weeks ago /bin/sh -c #(nop) ADD file:88875982b0512a9d0… 5.53MB
Docker images
Docker Best Practices for MicroServices - Oracle Code12 04.04.2019
Frozen
Image
Layers
Base Linux
5-100 MB Base distribution
Java Installation
90MB JDK 12
Base Image
50MB Runtime/Libraries
Application Image
1-x MB Application
Container
0-x MB Log files ...
ADD - 50 MB
RUN – 5MB
RUN – 1kB
ENV- 0 MB
Docker images – command chaining
Docker Best Practices for MicroServices - Oracle Code13 04.04.2019
ENV chaining
RUN multiple commands
RUN echo "my first command" && \
echo "second command"
ENV JAVA_HOME=/opt/jdk12 && \
PATH=/opt/jdk12/bin:$PATH
Docker images – Build Cache
Docker Best Practices for MicroServices - Oracle Code14 04.04.2019
Docker builds bottom up
Each Dockerfile command (ADD,
COPY, ENV, RUN, ...) results in an
additional layer
Docker uses the build cache
The lowest layer which changes
causes a rebuild of all subsequent
layers
Best Practices:
Correct ordering of layers
➢Add frequently changing layers the
latest possible
Use (and don‘t clear) the build cache
➢ It saves time and disk space
Check caching behaviour
➢Only create new images if contents
change
Build Cache - Java
Docker Best Practices for MicroServices - Oracle Code15 04.04.2019
JAR plugin
copy-dependencies plugin
WAR plugin
FAT Jar Plugins
Best Practices:
copy dependencies
– no fat jars
create extracted WAR directories
– Do not use .war file
Jlink
– Reduces JDK size even when using
all modules
– MultiStage Build
Smiley Images from pixabay.com, user Pixaline
Docker software installation
Docker Best Practices for MicroServices - Oracle Code16 04.04.2019
Centos – don‘t yum update!
Always install & clean! – useless without chaining!
Debian – don‘t apt-get upgrade!
RUN apt-get update && \
apt-get install –y ...... && \
apt-get clean
RUN yum install –y ..... && \
yum clean all
Docker images – how to install software
Docker Best Practices for MicroServices - Oracle Code17 04.04.2019
Task:
Install software from files (non-
package manager)
– Archives
– Downloaded packages
– Installation files not in image
Best Practices:
Install software using trusted
repositories
Use all options of Docker commands
– ADD (extracts archives)
– COPY (e.g. chown)
Docker – install via download
Docker Best Practices for MicroServices - Oracle Code18 04.04.2019
Simple Web-Server:
Installation via Download (curl) and delete installation file!
RUN curl -s -o /tmp/oracle-instantclient.rpm "http://... " && \
yum -y install /tmp/oracle-instantclient.rpm && \
yum clean all && \
rm -f /tmp/oracle-instantclient.rpm && \
echo done installing software
docker run -d --name downloadserver -p 80:80 \
-v /path/to/your/downloads:/usr/share/nginx/html/ \
nginx:1.15.5
Docker – install - using MultiStage Build
Docker Best Practices for MicroServices - Oracle Code19 04.04.2019
Finally build (in the same Dockerfile) the final image
Create a temporary build image ...
FROM my-image:version as builder
ADD ...
RUN ...
RUN ...
...
FROM my-image:version
COPY --from=builder --chown=tomcat:tomcat \
/opt/tomcat $CATALINA_HOME
Why small images
Docker Best Practices for MicroServices - Oracle Code20 04.04.2019
Keep in mind
Small images might not be reusable
Many small images are bad as well
Making small images can be hard
Reduce disk storage usage for CI/CD
– Faster builds
– Less push & pull network transfer
Deployments run faster
– Shared files on the host
– Shared Layers for less effort
Less cleanup
Docker is daemon software
Docker Best Practices for MicroServices - Oracle Code21 04.04.2019
„docker build“ creates a build context (tar file) which is sent to the docker daemon
Everything is included - unless .dockerignore hides it
All arguments to launch a container refer to the docker host (normall local)
Docker CLI uses Docker REST api
Everything refers to the host
target/*
my-temp-file
*.tmp
temp?.tmp
Docker & MicroServices
Docker Best Practices for MicroServices - Oracle Code23 04.04.2019
Split your domain into Bounded
Contexts
Run multiple services
One image for each service
– Run on multiple stages and
machines
Best Practices
It‘s all about automation & tests
– Keep it simple
– Everything as code
Use Docker in development
– Test database
– Mock services
Build in Docker (agent)
Test in Docker
– Docker-Compose
– Load Tests
Docker MicroServices
Docker Best Practices for MicroServices - Oracle Code24 04.04.2019
Baseimage Java
MicroService
MicroService
MicroService
MicroService
Baseimage Java
Baseimage Java
Baseimage Java
>60MB
>60MB
>60MB
>60MB
Docker MicroServices
Docker Best Practices for MicroServices - Oracle Code25 04.04.2019
Baseimage Java
MicroService
MicroService
MicroService
MicroService
60MB
Baseimage Java
Baseimage Java
Baseimage Java
<1 MB
60MB
Dependencies
Dependencies
Dependencies
Dependencies
>60MB
Docker layout for MicroServices
Docker Best Practices for MicroServices - Oracle Code26 04.04.2019
Baseimage Java Dependencies
>60MB90MB70MB
Baseimage Java Dependencies
>60MB90MB70MBMicroService
MicroService
MicroService
MicroService
<1 MB
Docker layout for MicroServices
Docker Best Practices for MicroServices - Oracle Code27 04.04.2019
Baseimage Java Dependencies
Larger size
Higher frequency of change
60MB90MB70MB
MicroService
MicroService
MicroService
MicroService
<1 MB
MicroServices Development
Docker Best Practices for MicroServices - Oracle Code28 04.04.2019
Everything in Docker
Multi-Branch Development
Every „git push“ creates an image
Deploy automatically
Images are selected manually and
promoted to „production“ registry
– Including tagging the sources
Test & deploy to production
Image types
Backend - Java (in MicroService
layout)
Frontend – HTML5 (nginx)
Build Agents
Database including test data
Mock Containers (nginx)
Load Test Containers
Infrastructure (Login, Loadbalancer, ..)
CI / CD environment
Container and security?
Docker Best Practices for MicroServices - Oracle Code30 04.04.2019
“Gartner asserts that applications deployed in
containers are more secure than applications
deployed on the bare OS [...] as long as a kernel
privilege escalation vulnerability does not exist on
the host OS”
(Joerg Fritsch, Research Director, Gartner, 2016)
Container and security?
Docker Best Practices for MicroServices - Oracle Code31 04.04.2019
Attacking process
Vulnerability in exposed service
Attack
– Privilege escalation
– Attack network
Common OS kernel
Docker Best Practices for MicroServices - Oracle Code32 04.04.2019
Host and container share the kernel
Vulnerabilities in the host kernel apply
to all containers
Best Practices:
Hardened systems
– Minimum software
Keep your docker host up-to-date
Update your docker base images from
a trusted source
Dangerous defaults & obvious leaks
Docker Best Practices for MicroServices - Oracle Code33 04.04.2019
Use secrets, no cleartext passwords
– No ENV entries
Use non-root users for execution, limit
privileges
Use cgroups to limit resource access (DoS)
Use only official images, better:
– Build your own, updated images
Minimize toolset – use different containers for
MultiStage build
Do not open your docker daemon
Build your own images!
Docker Best Practices for MicroServices - Oracle Code34 04.04.2019
Starting with the base image
– https://docs.docker.com/develop/develop-images/baseimages/
– Centos:
rpm --root <new-root> -ivh <centos-release>.rpm
rpm --root <new-root> \
--import $centos_root/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
yum -y --installroot=<new-root> --setopt=... install \
<place your package names here>
Build your own images!
Docker Best Practices for MicroServices - Oracle Code35 04.04.2019
Dockerfile
Docker Images are just directory contents
tar -C <new-root> cf centos.tar .
FROM scratch
ADD centos.tar
Docker MicroServices updates
Docker Best Practices for MicroServices - Oracle Code36 04.04.2019
Baseimage Java
MicroService
MicroService
MicroService
MicroService
Dependencies
Higher probability of known vulnerabilities
Java release cadence
8 releases/year
git push
frequency of non-security relevant changes
HOST
Docker – Pitfalls Terminal
Docker Best Practices for MicroServices - Oracle Code38 04.04.2019
Interactive session in installation
– „docker build“ runs without terminal connection
– Add „-y“ option to commands if available
„docker run“ without „-ti“
– Command silently exits
– No input / no output
„-ti“ but no terminal
– E.g. In cron jobs – results in an error
Pitfalls and BestPractices: Docker install
Docker Best Practices for MicroServices - Oracle Code39 04.04.2019
Pitfalls
Build Caches:
Dockerfile command is the build cache
key for RUN
Command chaining may result in large
and costly builds
Best Practices
Use unique names which change
when the contents change
No git clone without tag, Curl urls with
version identifier
Clear unnecessary files as soon as
possible
Use all options of ADD and COPY
Automate everything – no manual
steps
Docker – Pitfalls Java
Docker Best Practices for MicroServices - Oracle Code40 04.04.2019
Until Java 8u131 you have to specify maximum heap size
– Java unaware of cgroups limits
– Container will be killed by host OS due to memory overload
– Worse with more memory
– Worse with more containers
Especially critical when using RESTART=always
https://blogs.oracle.com/java-platform-group/java-se-support-for-docker-cpu-and-
memory-limits
Docker – Pitfall 8.8.8.8
Docker Best Practices for MicroServices - Oracle Code41 04.04.2019
About: --dns
“…If the container cannot reach any of the IP addresses
you specify, Google’s public DNS server 8.8.8.8 is added,
so that your container can resolve internet domains.…”
Source:
https://docs.docker.com/config/containers/container-networking/#dns-services
Docker – Pitfall 8.8.8.8
Docker Best Practices for MicroServices - Oracle Code42 04.04.2019
Simple DNS:
Provide a DNS service – remember DNS is „AP“ in „CAP theorem“
$cat /etc/docker/daemon.json
{
"dns": ["192.168.12.100"]
}
yum install –y dnsmasq
systemctl enable dnsmasq
systemctl start dnsmasq
Docker – Pitfalls bind mounts
Docker Best Practices for MicroServices - Oracle Code43 04.04.2019
Bind mounts to non-existing host files/directories will create them
– by default as a directory
Changes to bind-mounted files on the host might not be visible in the container
– Bind mounts refer to their inode
– Some editors (depending on their config) change the file‘s inode
• Nano: OK
• Vi(m): Not OK
• ~/.vimrc: set backupcopy=yes
Summary
Docker Best Practices for MicroServices - Oracle Code45 04.04.2019
Single Responsibility
Create well-prepared images
Command chaining
Build Cache
Small but reusable images
Install
– Packages
– Via download
– Using MultiStage build
MicroServices image layout
Security Updates – host & image
Trusted image source
– Build your own
Docker Best Practices for MicroServices - Oracle Code46 04.04.2019
Questions and answers
Thomas Bröll
Principal Consultant
Trivadis