puppetconf 2016: the challenges with container configuration – david lutterkort, puppet

The challenges ofcontainer configuration David Lutterkort @lutterkort [email protected]

mailto:[email protected]

Overview

● What is configuration ?

● Immutability

● Build vs Run

● Who configures the scheduler ?

● Conclusions

3

What is configuration ?

package/file/service

is only one instance of a more general problem

5

Configuration is any input into infrastructure

It needs to be managed

over time and at scale

6

Core configuration management features:

❏ describe system aspects in isolation

❏ combine aspects into whole

❏ common format for querying

❏ bridge across entire infrastructure

7

$ docker run -d \ -e MYSQL_HOST=mysql.example.com \ -e MYSQL_PORT=3306 \ --health-cmd /usr/bin/check \ webapp

Immutability

$ docker run \ --name example fedora:24 \ /bin/sh -c ‘while true; do \ cat /etc/system-release; \ sleep 1; \ done’

$ docker run …

Fedora release 24 (Twenty Four)Fedora release 24 (Twenty Four)Fedora release 24 (Twenty Four)Fedora release 24 (Twenty Four)Fedora release 24 (Twenty Four)Fedora release 24 (Twenty Four)Fedora release 24 (Twenty Four)Fedora release 24 (Twenty Four)

$ docker exec example /bin/sh -c \ ‘sed -i -e s/24/25/ /etc/system-release’

Fedora release 24 (Twenty Four)Fedora release 24 (Twenty Four)

Fedora release 25 (Twenty Four)Fedora release 25 (Twenty Four)Fedora release 25 (Twenty Four)Fedora release 25 (Twenty Four)Fedora release 25 (Twenty Four)Fedora release 25 (Twenty Four)

$ docker exec …

$ docker diff exampleC /runA /run/secretsC /etcC /etc/system-release

Containers are not immutable by defaultOnly as immutable as packages

15

$ docker run --read-only \ --name example fedora:24 \ /bin/sh -c ‘while true; do \ cat /etc/system-release; \ sleep 1; \ done’

$ docker exec example /bin/sh -c \ ‘sed -i -e s/24/25/ /etc/system-release’sed: couldn't open temporary file /etc/sed5OCs5t: Read-only file system

$ docker diff exampleC /runA /run/secrets

Suggestion

Enable --read-only whenever possible

19

require 'rubygems'require 'sinatra'require 'haml'

# Handle GET-request (Show the upload form)get "/upload" do haml :uploadend

# Handle POST-request (Receive and save the uploaded file)post "/upload" do File.open('uploads/' + params['myfile'][:filename], "w") do |f|

f.write(params['myfile'][:tempfile].read) end return "The file was successfully uploaded!"end

$ docker run -d --read-only lutter/lolcat



# Handle POST-request (Receive and save the uploaded file)post "/upload" do

File.open('uploads/' + params['myfile'][:filename], "w") do |f|f.write(params['myfile'][:tempfile].read)

end return "The file was successfully uploaded!"end

$ docker run -d --read-only \ -v /srv/lolcat/uploads:/app/uploads \ lutter/lolcat



# Handle POST-request (Receive and save the uploaded file)post "/upload" do File.open('uploads/' + params['myfile'][:filename], "w") do |f|

f.write(params['myfile'][:tempfile].read) end return "The file was successfully uploaded!"end

$ docker run -d --read-only \ -v /srv/lolcat/uploads:/app/uploads \ --tmpfs /tmp \ lutter/lolcat

Suggestion

Use --tmpfs where needed

26

Without technical controls you only have

social guarantees of immutability

27

How do you know the correct

invocation for an image ?

28

Build vs Run

Given an image

❏ What machine built this image ?

❏ How do you run this image ?

❏ Who supports this image ?

❏ Does the image contain malware ?

30

Given a container

❏ Who built it ?

❏ How was it built ?

❏ What software does it contain ?

❏ Is the software up-to-date ?

31

FROM fedora:24

RUN dnf update -y && \ dnf install -y ruby rubygem-bundler && \ dnf clean all

COPY . /app

RUN cd /app && bundle install --path vendor/bundle

WORKDIR /appVOLUME /app/uploadsEXPOSE 9292CMD ["/usr/bin/bundle", "exec", "rackup"]

FROM fedora:24


COPY . /app



Where did the base image come from ?

FROM fedora:24


COPY . /app



What repositories and what package versions ?

FROM fedora:24


COPY . /app



What was in this directory at build time ?

Time is your enemy

36

When do you rebuild images ?

37

Code changes and external factors

should trigger rebuild

38

Explain yourself with metadataDocker labels are a great way to do that

39

Name : glibcVersion : 2.23.1Release : 10.fc24Architecture: x86_64License : LGPLv2+ and LGPLv2+ with exceptions and GPLv2+Signature : RSA/SHA256, Thu 18 Aug 2016 09:27:43 AM PDT, Key ID 73bde98381b46521Source RPM : glibc-2.23.1-10.fc24.src.rpmBuild Date : Thu 18 Aug 2016 06:37:42 AM PDTBuild Host : buildvm-16.phx2.fedoraproject.orgPackager : Fedora ProjectVendor : Fedora ProjectSummary : The GNU libc libraries

$ docker inspect \ -f "{{json .Config.Volumes}}" lutter/lolcat{ "/app/uploads": {}}

$ docker inspect \ -f "{{json .Config.ExposedPorts}}" lutter/lolcat{ "9292/tcp": {}}

LABEL vendor=”ACME Incorporated” \ com.acme.release-status=”beta” \ com.acme.version=”0.1.0-beta” \ com.acme.git.sha=”f260653a”

$ docker inspect \ -f "{{json .Config.Labels}}" lutter/lolcat | jq{ "com.acme.git.sha": "f260653a", "com.acme.release-status": "beta", "com.acme.version": "0.1.0-beta", "vendor": "ACME Incorporated"}

Suggestion

Decide upon and enforcemetadata standards

45

LABEL com.acme.dockerfile=”/Dockerfile”

$ docker inspect \ -f "{{json .Config.Labels}}" lutter/alpine | jq{ "com.example.dockerfile": "/Dockerfile"}

$ docker run -it lutter/alpine cat /DockerfileFROM alpineRUN apk add --update bash && rm -rf /var/cache/apk/*COPY Dockerfile /LABEL com.example.dockerfile="/Dockerfile"

Suggestion

Embed your Dockerfile in the image

49

LABEL com.acme.cmd.packages=”apk info -vv”

$ docker run -it lutter/alpine apk info -vvmusl-1.1.14-r12 - the musl c library (libc)busybox-1.24.2-r11 - Size optimized toolbox of ...alpine-baselayout-3.0.3-r0 - Alpine base dir ...alpine-keys-1.1-r0 - Public keys for Alpine Linux ...zlib-1.2.8-r2 - A compression/decompression Librarybash-4.3.42-r3 - The GNU Bourne Again shell...

Suggestion

Make your images discoverable

52

puppetlabs/puppetlabs-image_build

class { 'nginx': }

nginx::resource::vhost { 'default': www_root => '/var/www/html',}

file { '/var/www/html/index.html': ensure => present, content => 'Hello Puppet and Docker',}

exec { 'Disable Nginx daemon mode': path => '/bin', command => 'echo "daemon off;" >> /etc/nginx/nginx.conf', unless => 'grep "daemon off" /etc/nginx/nginx.conf',}

# metadata.yamlcmd: nginxexpose: 80image_name: puppet/nginx

$ puppet docker build...

$ docker run -d -p 8080:80 acme/nginx-test83d5fbe370e84d424c71c1c038ad1f5892fec579d28b...

$ curl http://127.0.0.1:8080Hello Puppet and Docker

Who configures the scheduler ?

Schedulers/orchestrators isolate you from

❏ where individual containers run

❏ balancing due to new resources

❏ respawning due to failed resources

58

Schedulers operate on constraints

59

Decisions depend on accurate resource

information

60

$ docker daemon \ --label environment=production \ --label storage=ssd

$ docker run -d -P \ --label com.example.environment=production \ -e constraint:storage==ssd --name db mysql

template: metadata: labels: app: guestbook tier: frontend spec: containers: - name: php-redis image: gcr.io/google-samples/gb-frontend:v4 resources: requests: cpu: 100m memory: 100Mi env: - name: GET_HOSTS_FROM value: dns # If your cluster config does not include a dns service, then to # instead access environment variables to find service host # info, comment out the 'value: dns' line above, and uncomment the # line below. # value: env ports: - containerPort: 80

How do you manage properties

for all your hosts ?

64

Suggestion

Compute host properties dynamically

65

$ facter -y | head -n 20aio_agent_version: 1.7.0augeas: version: 1.4.0disks: sda:

model: SanDisk SDSSDA24size: 223.57 GiBsize_bytes: 240057409536vendor: ATA

...dmi: bios: ...memory:...

$ docker daemon \ --label os=$(facter os.family) \ --label kernel=$(facter kernelversion) \ --label memory=$(facter memory.system.total_bytes)

https://forge.puppet.com/puppetlabs/docker_platform

class { 'docker': labels => [ "os=${facts[os][family]", "kernel=${facts[kernelversion]}", "memory=${facts[memory][system][total_bytes]}" ],}

Schedulers introduce higher-level primitives

70

Docker networks

Kubernetes services and replication controllers

Chronos jobs

71

Many interfaces imperative not declarative

72

$ kubectl get pod mypod -o yaml \ | sed -e ‘s/$image:myimage$:.*$/\1:v4/’ \ | kubectl replace -f -

$ docker network create bobca7b185775966003d38ccbd9bba822fb570766e4bb

$ docker network create bobError response from daemon: network with name bob ...

docker_network { 'bob': ensure => present, driver => 'overlay', subnet => '192.168.1.0/24', gateway => '192.168.1.1', ip_range => '192.168.1.4/32',}

And everything is in YAML

76

“The language to represent the data should be a simple, data-only

format such as JSON or YAML, and programmatic modification of

this data should be done in a real programming language, where

there are well-understood semantics, as well as good tooling.

Borg, Omega, and Kubernetes, ACM Queue, Volume 14 Issue 1 | http://queue.acm.org/detail.cfm?id=2898444

77

Code plus data has advantages

over data alone

78

https://forge.puppet.com/garethr/kubernetes

kubernetes_pod { 'sample-pod': ensure => present, metadata => { namespace => 'default', }, spec => { containers => [{ name => 'container-name', image => 'nginx', }] },}

controller_service_pair { 'redis-master': app => 'redis', role => 'master', tier => 'backend', port => 6379,}

Conclusions

The difference between how you think a

system behaves and how it actually behaves

risks hard-to-debug production issues

83

Container use at scale and over time

requires meaningful abstraction

84

Configuration management as a discipline

provides tools to build those abstractions and

thereby minimize risk

85

86

Project Blueshift boothExhibition Hall

Docker, Mesos, Kubernetes and Puppet? Don't Panic !Deepak Giridharagopal, Thur, 4:45pm

Pulling the strings to containerize your lifeScott Coulton, Fri, 9:50am

Running Puppet software in Docker containersGareth Rushgrove, Fri, 1:30pm

puppetconf 2016: the challenges with container configuration – david lutterkort, puppet

Technology