solving real world production problems with docker

SOLVING REAL WORLD PRODUCTION PROBLEMS WITH DOCKER

OCTOBER 11TH, 2016

DOCKER MEETUP, LOS ANGELES

MARC CAMPBELL@mccode

GOALS

• Review Docker features that enable a more reliable, secure production environment

• Present a secure build-deliver-execute process that includes Docker in production

• Provide solutions you can start using today

“The only difference between a process in a container and a process not in a container is a few labels on top of a process that say ‘this is in container X’”

Jérôme Petazzoni, Docker July 06, 2015

SECURE DELIVERY PIPELINE

BUILD DELIVER EXECUTE

BUILD DELIVER EXECUTE

BUILD DELIVER EXECUTE

BUILD Choosing and creating container images that will run in your production environment.

THREE DIFFERENT ROLES, EQUALLY IMPORTANT

Operations

Development

Security

Does it work?

Can it be supported?

Can it be safely run?

Is it Elasticsearch? I don’t want “elasticsearch-like”, I want Elasticsearch and I want version 2.4.1.

Will it send alerts when it breaks? Does it support zero-downtime upgrades?

There are 2,532 Elasticsearch containers in DockerHub. Why this one?

BUILD DELIVER EXECUTE

• Development images do not have to be the same as production images

• Prefer library (official) images when possible

BUILD DELIVER EXECUTE

• Always look at the Dockerfile, regardless of pull count

• Be cautious when bind mounting the docker.sock file

BUILD DELIVER EXECUTE

Best practices- whitelist (or choose) base images - don’t trust “pull count” from DockerHub, find and read the dockerfile - use the most specific tag possible

redis:sha256@abc > redis:3.2.4 > redis:3.2 > redis:3 > redis:latest - adopt a tagging pattern for your own images - use security scanning (coreos clair or dockerhub) - use docker content trust

BUILD DELIVER EXECUTE

Monitor images with DockerHub Security Scanning or CoreOS Clair

The current nginx container on DockerHub has:

13 Critical CVEs 23 Major CVEs

Including 1 CRITICAL OpenSSL CVE

BUILD DELIVER EXECUTE

BUILD DELIVER EXECUTE

DELIVER Ensure the images you want to run are the images you are running

BUILD DELIVER EXECUTE

I typed `docker run redis` so now i’m running redis…right?…right???

BUILD DELIVER EXECUTE

What happens when you type `docker run redis`

DOCKER RUN REDIS

REDIS:LATEST IMAGE EXISTS?

CREATE REDIS CONTAINER

PULL REDIS:LATEST

START REDIS CONTAINER

NO

YES

BUILD DELIVER EXECUTE

BUILD DELIVER EXECUTE

DOCKER RUN REDISDOCKER CLI

DOCKER ENGINE

DOCKER HUB

CREATE NO IMAGE PULL

GET /V2

PARSE HEADER

Trust Boundaries

401 AUTH REQUIRED

POST /LOGIN

GET /V2/…/MANIFEST

GET /V2/…/LAYER

IMAGE COMPLETE CREATE START

Connect to a trusted host

Deliver the content over a secure channel

Sent the content you requested

Verify the author of the content

A.

B.

C.

D.

To securely download data from the Internet

BUILD DELIVER EXECUTE

HTTPS

TLS

Content Addressable IDs

Signed Images

The problems The solutions

Downloading and executing software from the Internet is dangerous

Don’t download from untrusted hosts. e.g.: `docker pull [—insecure-registry] 52.207.178.113/redis:latest` Don’t download on insecure channels. e.g.: `docker pull [—insecure-registry] registry.mycompany.com/redis:latest`

Don’t trust the remote server to look up the content. e.g.: `docker pull redis:latest`

Don’t trust content that isn’t signed by the publisher. e.g.: `docker pull --disable-content-trust redis:latest`

1.

2.

3.

4.

BUILD DELIVER EXECUTE

Docker Content Trust

“Content trust gives you the ability to verify both the integrity and the publisher of all the data received from a registry over any channel”

BUILD DELIVER EXECUTE

$ docker pull redis Using default tag: latest latest: Pulling from library/redis

6a5a5368e0c2: Pull complete <...> 2bcdfa1b63bf: Pull complete Digest: sha256:38e873a...912 Status: Downloaded newer image for redis:latest

WITHOUT TRUST: PULL BY TAG

BUILD DELIVER EXECUTE

$ export DOCKER_CONTENT_TRUST=1 $ docker pull redis Using default tag: latest Pull (1 of 1): redis:latest@sha256:c4365e...680 sha256:c4365ec...680: Pulling from library/redis

6a5a5368e0c2: Pull complete <...> 58e3d55f4ce5: Pull complete Digest: sha256:c4365e...680 Status: Downloaded newer image for redis@sha256:c4365e...680 Tagging redis@sha256:c4365e...680 as redis:latest

WITH TRUST: PULL BY SHA

BUILD DELIVER EXECUTE

BUILD DELIVER EXECUTE

DEMO

•Create a signed image •Run a signed image •Update the image from an untrusted source •Pull and run the new image

BUILD DELIVER EXECUTE

EXECUTE Provide a consistent, secure environment with continuous auditing

BUILD DELIVER EXECUTE

Center For Internet Security• Use AppArmor / SELinux • Enable Kernel Auditing • User namespaces • /var/lib/docker volume • Enable an authorization plugin • Use a centralized log driver • Prevent registry v1 access

https://benchmarks.cisecurity.org/tools2/docker/CIS_Docker_1.11.0_Benchmark_v1.0.0.pdf

BUILD DELIVER EXECUTE

Docker Bench for Securityhttps://dockerbench.com/

docker run -it --net host --pid host --cap-add audit_control \ -v /var/lib:/var/lib \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /usr/lib/systemd:/usr/lib/systemd \ -v /etc:/etc --label docker_bench_security \ docker/docker-bench-security

BUILD DELIVER EXECUTE

2.1 - Restrict network traffic between containers 2.2 - Set the logging level 2.3 - Allow Docker to make changes to iptables 2.4 - Do not use insecure registries 2.5 - Do not use the aufs storage driver 2.6 - Configure TLS authentication for Docker daemon * Docker daemon not listening on TCP 2.7 - Set default ulimit as appropriate * Default ulimit doesn't appear to be set 2.8 - Enable user namespace support 2.9 - Confirm default cgroup usage 2.10 - Do not change base device size until needed 2.11 - Use authorization plugin 2.12 - Configure centralized and remote logging 2.13 - Disable operations on legacy registry (v1)

[WARN] [PASS] [PASS] [PASS] [WARN] [INFO] [INFO] [INFO] [INFO] [WARN] [PASS] [PASS] [WARN] [WARN] [WARN]

BUILD DELIVER EXECUTE

Review☑ Choose images carefully ☑ Scan your Dockerfiles ☑ Enable Docker Content Trust ☑ Run Docker Benchmark for Security