test driven infrastructure development

Test driven Infrastructure development

Tomas (t0m) Doran<tomas.doran@timgroup.com>@bobtfishhttps://github.com/bobtfishhttps://github.com/youdevise

Thursday, 14 March 13

‘Real men’ develop in production!

Repeat again and again. Development cycle SLOOOW.

• Edit / Commit / Push

• Update puppetmaster

• puppet agent -t

• Repeat

This is insane!

CHOAS and FAIL result when you break each other. Or, MORE likely (this happens twice a day!)

This is insane!

• Try it on an 8 person team.

This is insane!

• ‘LOL - I broke puppet’

OI t0m!!!!

You broke

puppet!

OI t0m!!!!

You broke

puppet!

AARRRGGH!!!

Lets fix this!

• First, a glossary:

Lets fix this!

• mco - mcollective

Lets fix this!

• mco - mcollective

• ENC - External node classifier

We can do better

This at least lets you develop things independently. Everyone can do dev in their own branch and merge once they have something that doesn’t break _everything_. You can also rebase -i (squash) all the ARGH PUPPET SYNTAX commits.

We can do better

• Branch == environment

We can do better

• Branch / Commit / Push

We can do better

• mco puppetupdate

We can do better

• mco puppetupdate

• puppet agent -t --environment xxx

Sounds good?

•Then you’ll be wanting:

•https://github.com/youdevise/puppetupdate

It’s a bit basic, but then I ripped it out of work internal code at 8am ;)

So we fixed it?

Refactoring

Sorry Chris, but when you say ‘refactoring’ - it’s not refactoring unless you have tests.The problem is that you can’t always remember to run the right branch on all the right nodes. Or rather, how do you even know what all the right nodes are? And if you’re hacking on custom functions, or anything using exported resource - WOE

Refactoring

• We change things to be consistent across codebase:

• Why did puppet just delete all the firewall rules on the production database?

Refactoring

• We change things to be consistent across codebase:

• Why did puppet just delete all the firewall rules on the production database?

• We don’t refactor:

• Add bugs all the time due to inconsistency

• Hard coded IPs in 10 places

Unfortunate reality:

So, despite our best efforts, our puppet code was SHIIIIT.Exported resources IS NOT a good fit for non-trivial things (like generating load balancer configs). Ergo lots of hard coded IPs in multiple places. Ergo puppet code per site.

• role::oy_lb

• hiera data split by domain (colo)

• role::oy_lb

• mco puppet

• role::oy_lb

• mco puppet

• 4 weeks per app per environment

The state of the art

• It’s certainly in a state

Nobody does automatic runsPuppet becomes an auditing tool (automatic noop runs + reports)

• Automatic runs dangerous

• cron --noop runs

• puppet becomes an auditing system

• This isn’t what I signed up for!

Business says no!

• Launching new products has a long lead time

• This is unhelpful if your company is trying to branch out into new markets

Business says no!

• Launching new products has a long lead time

• This is unhelpful if your company is trying to branch out into new markets

• CI / stage environments unlike prod

• Issues when new functionality goes live

• Developers think you’re incompetent

What is wrong with this picture?

You just don’t know the answer to any of these questions in any reliable way...But, generally, the answers are NO, YES, NO, NO

What is wrong with this picture?•Did you run it everywhere?

•Does it affect anything you’re not expecting?

•Can you rebuild cleanly?

•Does the code even make things reflect current state?

‘We use puppet’

Hint - you don’t!

‘We use puppet’

• Means nothing

Hint - you don’t!

‘We use puppet’

• Means nothing

• State of your system is the sum of all changes

Hint - you don’t!

‘We use puppet’

• Means nothing

• State of your system is the sum of all changes

• How do you know your code can rebuild things?

Hint - you don’t!

It’s all mierda

We need to grow up, and raise the level of the conversation..

It’s all mierda

•Development communities are 10 years ahead

It’s all mierda

•We don’t integration test

• (repeatably)

It’s all mierda

•We don’t integration test

• (repeatably)

•We can’t build / rebuild

• (reliably)

Infra is hard

Sure - it’s much much harder to get a standalone testable system in infra than it is in development.

Infra is hard

• Infrastructure is inherently more complex

Infra is hard

• Less control

Infra is hard

• Less control

• More moving parts

Infra is hard

• Less control

• ‘End to end’ testing

Infra is hard

• Less control

• ‘End to end’ testing

• Persistent data

No excuses:Scientific method

I do not consider this an excuse to abandon sanity.

The solution?

• Re-provision everything in tests

• N.B. Not perfect (but better!)

The solution?

• Proper software engineering

• Unit and integration tests

• Build pipeline + promotion

Openstack

• Our tests spinning up 12 machines => VMs

So, we should use openstack, right? As of December, when we looked - 2 networks max, inflexible. lvs not possible.

Openstack

• Openstack going to be awesome, right now:

Openstack

• Networking sucks

Openstack

• Load balancing is a shambles

Openstack

• Load balancing is a shambles

• lvs / vlans / metal / bonding - nope

My desires:

My desires:• Reuse as much code as possible! (e.g. load

balancers)

• No per colo/environment puppet code

balancers)

• No IPs anywhere

balancers)

• No IPs anywhere

• ‘DRY’

balancers)

• No IPs anywhere

• ‘DRY’

• CI pipeline to promote to production

balancers)

• No IPs anywhere

• ‘DRY’

• 1 puppet run from provisioned to working

balancers)

• No IPs anywhere

• ‘DRY’

• 1 puppet run from provisioned to working

• Repeatable and testable!

Orc• Continuous (zero downtime) deployment

• Development / infrastructure application contract

• Model driven

• https://github.com/youdevise/orc/

Puppetroll

• Rolls out a consistent sha1 from the puppetmaster to an entire environment

Puppetroll

• Fails if any puppet run fails

Puppetroll

• Fails if any puppet run fails

• https://github.com/youdevise/puppetroll

Provisioning tools

• debootstrap custom gold images

Provisioning tools

• mcollective ‘computenode’ agent for kvm

Provisioning tools

• ‘provision me a machine called X, on networks Y and Z’

Provisioning tools

• ‘provision me a machine called X, on networks Y and Z’

• Dynamic IP allocation (dnsmasq locally, DDNS for real)

stacks

• Model driven deployment

stacks

• DSL for describing groups of systems + dependencies

stacks

• rake tasks to provision / test / clean up stack + deps

stacks

• rake tasks to provision / test / clean up stack + deps

• Can provision a full environment, run E2E tests, tear it down - in CI.

I want to hack on load balancers

= 4 new, independent machines

How it works?

• DSL creates model of systems

How it works?

• rake task ‘launch’:

How it works?

• mco provisions boxes on compute nodes

How it works?

• each box runs puppet --waitforcert

How it works?

• mco signs cert

How it works?

• mco signs cert

• puppet runs for each box

mco computenode

Puppetmaster

• Uses the same model

Puppetmaster

• Generates an ENC for each node

Puppetmaster

• Puppet code:

Puppetmaster

• Puppet code:

• Just installs things / starts services

Puppetmaster

• Puppet code:

• Just installs things / starts services

• I.E. what it’s good at!

External node classifier

Putting it together

So, what do we have? Well - everything I showed you already...Building proxy server layer (by refactoring puppet code) right now. Databases to follow!

Putting it together

• Still ongoing - live production apps ETA two weeks.

Putting it together

• Still haven’t solved re-provisioning problem for live environments!

Putting it together

• Still haven’t solved re-provisioning problem for live environments!

• Do have repeatable and testable / tested infrastructure building in CI!

The top table is our test overview - we have two types of tests, those which are for a specific machine (i.e. a VM) and those which are for a virtual service (backed by multiple machines)‘behaves like’ is an rspec thing we haven’t overridden.For each machine, we test that it’s pingable, then run every nrpe (nagios) agent and check them - if all the things nagios monitors are OK, machine is OK.

In the (near) future?

• Live application stack in production

• Automated ‘promotion’ of good changes to production

• Integrated environment support for dev stacks on dev branches/environments

• Open source all the things!

Thanks!

• puppet is an awesome tool.

• It doesn’t solve higher level system modeling problems

• It shouldn’t try to!

Thanks!

• puppet is an awesome tool.

• It doesn’t solve higher level system modeling problems

• It shouldn’t try to!

• sysadmins need to level up

• It’s not done till you can test it still works

Photo Credits• Escher's "Relativity" in LEGO - Andrew Lipson (http://www.andrewlipson.com)

• Manure - Flickr - chesbayprogram

• Provisions - Flickr - quinn.anya

• Stacked - Flickr - andrewrennie

• Dilbert - Flickr - osde-info

• Stacking wood - fickr - arthuserea

• Square wheels - Flickr - vrogy

• Puppets - Flickr - SkipSteuart

• Light bulb - Flickr - bazik

• This-is-not-art - Wikimedia commons - Loran Davis

• Danger of death - Flickr - zigazou76

• Bob the Builder - Flickr - jamesclay

• Swiss roll - Flickr - add1sun

• Orc - Flickr - photo_munki

• Danger! Danger - Flickr - donsolo

• Cow of the future - Flickr - thewamphyri

• SCIENCE - Flickr - chasblackman

Links!

• http://github.com/youdevise

• http://github.com/bobtfish

• https://devblog.timgroup.com/

• (Yes, we are hiring)

test driven infrastructure development

right branch

right nodes

puppet syntax commits

better branch

doesnt break

development cycle slooow

branchand merge

real men

Technology

se 2015 devops risk mitigation - test driven infrastructure

test driven development_for_php

osdc 2014 test driven infrastructure

event driven infrastructure

collaborative infrastructure for test-driven scientific...

test driven infrastructure development - puppetconf 2013

test driven infrastructure development

enterprise model driven infrastructure

test driven product: applying test driven thinking to the...

test-driven development of infrastructure code

test-driven infrastructure with cloudformation and cucumber

test driven infrastructure development (2 - puppetconf 2013...

data-driven library infrastructure

applicability of acceptance test driven development in...

obstacle driven development: extending test driven...

test driven

test driven documentation

infrastructure...

test-driven development with qt and kde · practice...

containercon - test driven infrastructure