Continuous Integration of Puppet Code

@d2si

Who am I ?

2

Jack of all trades, I love challenges of any kind from motor sports to IT.

@fgouteroux

@d2si

Outline

Puppet Dev and TestsState of the artWhat's missing ?

Enters beakerPrinciples and common use casesExamples

Real use case testing with beakerSimulate a true puppet deploymentDetails and Setup

Integration in the dev workflow

@d2si

Tests

Why ?

Syntax error è Manifest does not compileDuplicate resource è Dependency cycleForgetting to include a module or set a variable è puppet run fails

Result

The host fails to enforce the expected state

@d2si

State of the art

Writing tests is a good way to verify that your modules are•  functional•  reusable

Ensure the module does what you want it to do

@d2si

State of the art

Automated testing is one of the key ways to ensure that:•  your libraries •  your manifests

are meeting your expectations !

@d2si

State of the art

There are differents ways to test puppet manifests

Puppet-lint

•  Quick way to ensure that everybody is following a common set of conventions

•  Analyze your manifests and look for deviations from the Puppet style guide

@d2si

Puppet-lint

@d2si

State of the art

Puppet-rspec

•  Manifests and modules compile

•  Manifests contain the expected values

•  Specific types, classes or definitions are in the compiled catalog

•  Parameters match the expectations

@d2si

Puppet-Rspec

@d2si

State of the art

Puppet apply

•  Applies a standalone Puppet manifest to the local system

•  « Simulates » a catalog compilation with modulepath option

$ puppet apply -l /tmp/manifest.log manifest.pp $ puppet apply --modulepath=/root/dev/modules -e "include ntpd::server" $ puppet apply --catalog catalog.json

@d2si

It’s cool but, beakerful

You want to be sure that running Puppet on a host will: •  build the host the way you want •  have the behavior you expect That means, verify services like: •  SSH •  Postgres •  Nginx

are running and serving resources !

@d2si

What's missing ?

Testing entire systems: 1. Create virtual machines 2. Configure virtual machines 3. Deploy applications 4. Execute tests 5. Report on tests 6. Destroy virtual machines

6 5

4

3

1

2

@d2si

Enters Beaker

•  Open source acceptance testing tool •  Built by Alice Nodelman (Puppet Labs) •  Build test environments by vm provisionning •  Multi-Cloud Providers (AWS, Google, Openstack) •  Multi Virtualisation support (Docker, vSphere, Vagrant, Virtualbox) •  Lifecycle management •  First step for Continuous Integration

@d2si

Principles and common use cases

•  Tests are just Ruby files

•  Tests passed if no errors/exceptions

•  Tests use asserts to explicitly enforce a state

@d2si

Principles and common use cases

Let’s look at tests !

@d2si

Principles and common use cases

More examples !

@d2si

Simulate a true puppet deployment on AWS

Tasks: 1. Create 5 Virtual machines 2. Configure 4 Virtual machines 3. Deploy applications 4. Execute tests 5. Report tests 6. Destroy 5 Virtual machines

@d2si

TASK 1: Create VMs

How to reproduce this use case with beaker ?

@d2si

TASK 1: Create VMs

Create a file and describes your nodes:

@d2si

Beaker Roles Each host in a host configuration file has one or more roles

Beaker natively supports the following roles:•  master•  agent•  frictionless•  dashboard•  database

These roles indicate what Puppet responsibilities the host will assume.

If puppet is installed as part of the Beaker test execution then the roles will be honored (ie, the host defined as master will become the puppet master node).

Other than puppet installation, the roles provide shortcuts to access nodes

@d2si

TASK 2: Configure VMs

@d2si

TASK 2: Configure VMs

Step 1: Bootstrap master node•  Update package list•  Install puppetmaster, ruby packages •  Install/Configure Hiera •  Install/Configure/Deploy r10k •  Run puppet apply Step 2: Bootstrap agent nodes •  Update package list •  Install curl, augeas packages •  Push hostname and public IP to Consul KV •  Install puppet agent •  Set Default Puppet Environment •  Run puppet agent

@d2si

r10k is a tool to help manage dynamic environments. •  Creates new environments from git branches •  Deploy modules to that environment.

R10K

@d2si

@d2si

beaker_helper.rb

@d2si

TASK 3: Deploy applications

@d2si

Puppet Stack – Roles/Profiles

From Craig Dunn: http://www.craigdunn.org/2012/05/239/

•  Nodes get a single role •  Roles use several profiles •  Profiles and Roles are Puppet modules

@d2si

Puppet Stack - Hiera

From Craig Dunn

•  Top down hierarchy for overriding configuration values based on roles, environments…

•  Puppet modules without hard-coded data are easily shared and more re-usable

•  Infrastructure configuration can be managed without needing to edit Puppet code

Pluggable Backends: •  Source data from multiple locations •  Data source is abstracted from code

@d2si

Then it’s puppet job !

•  Puppet get the node’s role with the node’s name

•  Puppet apply the manifest according to the role

For front-server role, puppet need backends server infos like: •  hostname •  public IP

To get these infos, puppet call hiera consul backend.

TASK 3: Deploy applications

@d2si

front_server.pp

@d2si

apache_server.pp

@d2si

TASK 4: Execute Tests

@d2si

app_server.rb

@d2si

front_server.rb

@d2si

client.rb

@d2si

Task 5: Reporting

@d2si

@d2si

@d2si

Task 6: Destroy VMs

@d2si

Integration in the dev workflow

Developper update a module Trigger a hook

Run puppet-lint Run puppet-rspec

Run integration tests Merge dev to prod

(pipeline)

Run tests Play configuration

@d2si

Questions ?

@d2si

Go further

https://github.com/fgouteroux/demo-puppet-beaker

http://blog.d2-si.fr/2015/03/18/technique-puppet-beaker/

@d2si

Thank you !