Test Driven Infrastructure with Docker, Test Kitchen and Serverspec Yury Tsarev

Agenda

▸ Goal▸ Test kitchen▸ Docker driver▸ kitchen-puppet provisioner▸ Serverspec verifier▸ Shellmocking▸ Defining puppet type in kitchen▸ Create test-driven infrastructure change▸ Wrap-up

Goal

▸ Infrastructure code should be treated as any other code▸ Apply TDD for puppet▸ Grow regression test suite

Test kitchen

▸ http://kitchen.ci/ ▸ Test orchestrator▸ Originated in Chef community▸ Very pluggable on all levels▸ "Your infrastructure deserves tests too."▸ Book - Test-Driven Infrastructure with Chef

Test kitchen - high level process

1. Create VM/Container (docker driver)2. Run configuration management code there (puppet

provisioner)3. Test with verifier (serverspec)

Main configuration file - .kitchen.yml

▸ Driver: what type of VM/containerization/cloud to use▸ Provisioner: which configuration management tool to apply▸ Verifier: test automation type to verify with▸ Transport: mechanism to upload files into instance under

test

Docker driver

▸ https://github.com/portertech/kitchen-docker▸ kitchen driver to work with docker containers as machines

under testdriver:

name: docker

image: docker-registry.na.intgdc.com:80/gdc:R23

platform: rhel

use_sudo: false

provision_command:

yum clean all && yum makecache

kitchen-puppet provisioner

▸ https://github.com/neillturner/kitchen-puppet▸ Uploads puppet code into instance▸ Runs puppet there (converge)▸ Provides facts customization facility

custom_facts:

docker: 1

hostname: kitchen_test

ec2data_branch: develop # pulp repo branch, critical for gitflow repo

ec2data_gitbranch: develop # git branch

ec2data_freeipa_otp: test

ec2_public_ipv4: 127.0.0.1

ec2_local_ipv4: 127.0.0.1

has_mnt: 0

Serverspec test suite

▸ http://serverspec.org/ - RSpec based framework▸ http://serverspec.org/resource_types.html ▸ Independent of kitchen - can be used standalone▸ We keep test suite together with puppet code under

spec directory to have ability to create consistent PRsdescribe file('/var/log/httpd') do

it { should be_directory }

end

describe docker_image('busybox:latest') do

its(['Architecture']) { should eq 'amd64' }

end

Serverspec verifier: busser or shell

▸ Simple

https://github.com/test-kitchen/busser-serverspec

▸ Advanced

https://github.com/vincentbernat/serverspec-example

Serverspec verifier in GD

▸ Based on shell verifier and serverspec test suite▸ Runs a serverspec test suite against the configured

(converged) instance▸ Reports in shell and junit - jenkins ready

verifier:

name: shell

remote_exec: true

command: |

sudo -s <<SERVERSPEC

export SERVERSPEC_ENV=$EC2DATA_ENVIRONMENT

export SERVERSPEC_BACKEND=exec

serverspec junit=true tag=~skip_in_kitchen check:role:$EC2DATA_TYPE

SERVERSPEC

Transport

▸ https://github.com/coderanger/kitchen-sync▸ Replaced default scp with sftp transport▸ Reduced 1.5m to 5 sec for puppet and test suite upload!

Shellmocking

▸ To bypass external dependencies▸ To bypass docker specific limitations▸ Implementation located in puppet repo under

spec/shellmock▸ Simple ruby script▸ Wraps yum invocation▸ Mock is defined in simple yaml format:

package:

/path/to/executable: contents

Defining puppet type in kitchen

▸ Declare in platforms section of .kitchen.ymlplatforms:

- name: zuul

provisioner:

custom_facts:

ec2data_type: zuul

▸ kitchen converge▸ pray▸ shellmock▸ repeat

Create test-driven infrastructure change

▸ Write serverspec expectation for new code▸ kitchen verify <type>▸ Observe related test is red▸ Write puppet code▸ kitchen converge <type>▸ kitchen verify <type>▸ Observe related test is green▸ Commit the changes and create PR to puppet repo▸ DEMO

Wrap-up: Benefits

▸ Scratch environment▸ Test in isolation▸ Easy to test permutations▸ Resource efficiency▸ Test-first/test-driven approach for infrastructure code▸ Fast feedback - even before a commit▸ Naturally growing regression test suite

Wrap-up: OSS side of things

▸ Multiple open source projects combined▸ We contributed a lot into related kitchen projects and

serverspec▸ Nominated as kitchen-puppet core contributor

Wrap-up: What is next ?

▸ Define all types in kitchen▸ with every next adopted type it will be easier

▸ Create smart way to test only related types for code change, a.k.a puppet dependency tracking

Wrap-up: A note on agnostic approach

▸ As the kitchen driver is agnostic to virtualization solution/cloud

▸ As the kitchen provisioner is agnostic to configuration management solution

▸ As the serverspec is agnostic to way to configure servers at all

▸ We are capable to make brave movements in future!

Wrap-up: Try it!

▸ http://kitchen.ci/ ▸ https://github.com/portertech/kitchen-docker ▸ https://github.com/neillturner/kitchen-puppet ▸ http://serverspec.org/