W11 DevOps & Testing 5/4/16 13:45
Continuous Integration Testing Techniques to Improve Chef Cookbook
Quality
Presented by:
Glenn Buckholz
Coveros, Inc.
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888-268-8770 904-278-0524 [email protected] http://www.stareast.techwell.com/
Glenn Buckholz Coveros, Inc.
With fifteen years of industry experience, Glenn Buckholz leads continuous integration and deployment automation efforts at Coveros. His career began as a consultant implementing automated test frameworks and introducing the concept of change management to many, many projects. Glenn then decided to become a part of honest society and settled down at the Public Company Accounting Oversight Board as their full8 time enterprise change manager. Several years later, he joined Coveros, where he specializes in implementing agile practices and CI, and engineering configuration management instead of simply documenting it.
© Copyright 2016 Coveros, Inc. All rights reserved.
Continuous Integration Testing
Techniques to Improve Chef
Cookbook Quality
STAREAST – 04 May 2016
Glenn Buckholz
© Copyright 2016 Coveros, Inc. All rights reserved.
Agenda
•What did I do?
•What is problem?
•What are the needs?
•Demo
•Architecture
•Workflow
•Testing Techniques and Tools
•Metrics?
•Docker and Cloud
•Discussion
© Copyright 2016 Coveros, Inc. All rights reserved.
What did I do?
● Used a modern CI/CD toolchain to apply
continuous integration testing techniques to
Chef code (CI):
○ Static analysis
○ Chefspec testing
○ Automated Functional Testing
● Introduces the idea of automatically making
cookbooks available when they are ready
(CD).
○ There is still a human gate between
available and in use by way of version
pinning.
© Copyright 2016 Coveros, Inc. All rights reserved.
What is the problem?
● Infrastructure as Code is currently the wild
west with little formal testing methodology.
● Chef lacks coherent CM on its own.
○ Cookbook versions and freezing are not
enough.
● No quick way to advance a piece of
infrastructure from one state to another and
back for purposes of testing.
● No minimal standards and requirements on
cookbook functionality.
© Copyright 2016 Coveros, Inc. All rights reserved.
What are the needs?
YOU ARE CHANGING THE TIRE AT
55MPH AND CAN’T STOP!!! ● Find flaws early.
● Ability to test pieces of an architecture
without having to have a whole system.
○ Unit testing system components.
● Simulate the effect of a deploy on a running
system.
● Dashboarding so you can trend code
quality.
© Copyright 2016 Coveros, Inc. All rights reserved.
Definitions
● Hands Off Deploy - Deployment that is
completely driven by Chef or some other
infrastructure tool.
● Static Analysis - evaluation of the code
without execution.
● Feedback loop - The events between when
a developer commits their code and results
can be observed.
● Application Code - The custom or home
grown software the current IT organization
is curating.
© Copyright 2016 Coveros, Inc. All rights reserved.
Architecture
Local Git Branch
Chefspec
Test Kitchen
Food Critic
Developer Workstation
Docker
Container
Binary Repository
Docker
Repository
Local Testing
Docker
Container
Upload Cookbook
© Copyright 2016 Coveros, Inc. All rights reserved.
Workflow
1. Clone master branch on the developer
machine.
2. Make changes and run static analysis.
3. Run Chefspec and test kitchen locally on a
docker container.
4. Push changes to remote branch.
5. Jenkins detects changes and checks out
branch onto a slave node.
© Copyright 2016 Coveros, Inc. All rights reserved.
Workflow
6. Jenkins merges the code with master
locally.
7. Chef reruns the static analysis.
8. Chef reruns Chefspec and test kitchen.
9. Results and metrics are recorded.
10.Successful tests see code merged into
master.
11.Cookbooks are uploaded to the chef
server.
12.After the code is executed in production,
AMI and Docker containers are
automatically updated from production.
Steps 8-9
automate
integration
testing
© Copyright 2016 Coveros, Inc. All rights reserved.
Reuse of well known patterns
● Feature Branches
○ Small bite sized changes.
● Automated Merging
○ Let the Jenkins objectively merge the
code after certain minimum standards are
achieved by the developer.
● Local tests that mirror a golden standard
○ Give the developers a cheat sheet.
● Short feedback loops.
● Auto Promotion
○ Let Jenkins keep the working cookbooks
in sync with SCM
Smaller changes are
more likely to make it
through the pipeline.
© Copyright 2016 Coveros, Inc. All rights reserved.
Testing Techniques
● Tabula Rasa
○ Start with a base image
○ Apply Security Hardening
○ Install stack fresh
○ Deploy application code
○ Run tests
● Production Approximation
○ Start with a clone from production
○ Run cookbooks
■ Only changes are executed
○ Run test
© Copyright 2016 Coveros, Inc. All rights reserved.
Testing Techniques
● Test in parallel
○ Do not block on critical resources clone
them.
○ VMs and containers are cheap, create
one for every commit you want to test.
○ Make unit and integration test cases
isolated. (Script system state for each
test case do not rely on other tests)
● Test local
○ Very short feedback loop for developers.
○ Reduces most pipeline failures to
integration errors.
© Copyright 2016 Coveros, Inc. All rights reserved.
Testing Tools
● Unit Testing
○ chefspec - extension of Rspec
● Functional Testing
○ BATS or chefspec
● Static Analysis
○ Foodcritic - lint like
● Git hooks
○ Make certain static tests run as a git hook
making them prerequisites to commit or
push.
© Copyright 2016 Coveros, Inc. All rights reserved.
Testing Tools
● Git Branching
○ Let the branch name decide if the CI
stack will run the changes.
○ You may want to use a prefix CI-
branchname
© Copyright 2016 Coveros, Inc. All rights reserved.
Testing Tools - Jenkins Master
● Jenkins master only monitors git.
○ Container on slave in the cloud for quick
tests.
○ VMs in cloud for executing tests in a
production like system.
■ Template AMI matches target machine.
○ All “work” takes place off the master to allow
testing in parallel.
● Create Jenkins jobs to capture production
templates post deployment. (AMI or docker
image)
© Copyright 2016 Coveros, Inc. All rights reserved.
Testing Tools - Developer Workstation
● Provide an easy to install package that includes:
○ Tools - Chefspec, Foodcritic, etc…
○ Docker settings.
○ Access to a docker repository.
○ Templates to create new cookbooks
● Enough memory and CPU to run the tools.
● DO NOT ALLOW COOKBOOK UPLOAD PRIVILEGES!!!!!
● DO NOT ALLOW MERGE TO MASTER PRIVILEGES!!!!!
● The ability for a developer to find mistakes locally is the
most cost effective testing technique available to an
organization. (Short Feedback Loop)
© Copyright 2016 Coveros, Inc. All rights reserved.
Transitioning to the CI Approach
● From the previous slides we have the tools to create
an method for retrofitting a CI driven approach to an
existing Chef infrastructure.
● On a separate branch create cookbooks that install
a piece of the production stack on a server or
container.
● On a feature branch develop just the chef code
needed to deploy your custom application.
● Slowly merge the two once per iteration. Merge one
item from the stack branch and merge your install
code to master.
© Copyright 2016 Coveros, Inc. All rights reserved.
Transitioning to the CI Approach
● Do not try to boil the ocean.
● Start with installing the application code.
● Move up to controlling the application stack
one layer at a time.
● This maps to Agile sprints and iterations.
● Cookbooks should be idempotent.
Master (What is in Chef Server)
Install Code
(Feature
Branch)
Install Code
(Feature
Branch)
Install Code
(Feature
Branch)
Install Code
(Feature
Branch)
Install Code
(Feature
Branch)
Infrastructure or Application Stack Cookbooks (Apache, Tomcat, etc.)
Test
Here
Test
Here
Test
Here Test
Here
© Copyright 2016 Coveros, Inc. All rights reserved.
Metrics
● Failed Merges
○ How many times did a developer cause a
potential merge conflict?
● Static analysis thresholds
○ How many times did a commit fail?
● Failed rspec tests.
● Failed functional tests.
● How long is the component down during
upgrade.
© Copyright 2016 Coveros, Inc. All rights reserved.
Docker and Cloud
● Docker
○ Can be very fast
○ Need a local Docker repository.
● Cloud
○ You can use a mirror of production
environment.
○ Need to find a way to clone VMs without
downtime or cluster components.
● If production uses docker this is a moot
point, just use docker.
© Copyright 2016 Coveros, Inc. All rights reserved.
Discussion Points
● Tool chain can be complicated.
○ If you already invested in CI you have a sunk cost
with many of the tools already.
● Two major types of transitions:
○ Introducing CI to chef.
○ Introducing Chef and CI to a project.
● Start small, big bang automation is high risk.
● Make the quality gates automated and give the
developers the answer key, don’t change the test, and
let Jenkins proctor the exam.
● Allocate time to sync production changes to testing
images.
© Copyright 2016 Coveros, Inc. All rights reserved.
What Problems Did We Solve?
● Version management - The git cookbook version linkage
provides change traceability. This is enforced by Jenkins.
● Enforcing testing in general for Infrastructure - By ceeding
cookbook upload control to Jenkins we can have a
minimal set of quality standards for all deployments and
changes.
● Making Testing accessible to the developers - By putting a
framework at their fingertips. This makes it easier for
developers to follow the process than circumvent the
process.
● Provide testing metrics so cookbook quality can be
analyzed.
● Provided an automated testing framework to provide quick
feedback on integration errors.
● Disposable infrastructure for rapidly changing server state.
© Copyright 2016 Coveros, Inc. All rights reserved.
What Problems Didn’t We Solve?
● Adoption
○ Change it hard, things can be done to ease
the transition but it cannot be forced. You
still have to convince developers the value
proposition of writing the tests.
● Integration errors
○ While we can find them faster, there is no
tool I know of that can predict code
interaction accurately.