Quick Prototyping+CI with LXC and Puppet

Ben Kero

2014-05-04

Introduction

I Ben Kero

I Release Engineer

I Responsible for version control systems: CVS, SVN, BZR,Darcs, RCS, Git, Mercurial

Before at Mozilla

I Dawn of time (2001) to early 2014

I All infrastructure deployed with Puppet

I No documented environment or target for setup

I Wild-wild west. Write the code. Does it work? Ship it!

Framing the problem

I Problem: Developers and upstreams need a way to replicateour production environment

I Problem: Needed a way to make development environmentcloser to production

I Requirement: Quick to bring up

I Requirement: Public, shareable, infinitely reproducible

I Requirement: Dev environment should be as close as possibleto production

Ideas (1/3)

I Virtual machines (on workstation)I KVM/VMWare WorkstationI Could be closest match to productionI Expensive to generate up-to-date imagesI Slow, can only run a few at a time

I Virtual machines (in cloud)I Requires internet connection for development/testingI Requires IT to manage/audit accountsI No/separate public availability

Ideas (2/3)I Raw cgroups + CoW (Copy-on-Write) images

I CoW mechanisms require out-of-tree modules or filesystem ofdubious stability

I Raw cgroups are difficult to use. Bring-up time for devs wouldbe catastrophic

I Linux-only (Win/Mac devs need to virtualize)

I DockerI Doesn’t clean images up after itselfI Dockerfiles vs Puppet codeI Fast moving targetI Linux-only (Win/Mac devs need to virtualize)

I VMWare Vsphere (Vcloud)I ExpensiveI Requires specialized hardwareI Not free softwareI No offline developmentI Beta statusI Limited dev access

Ideas (3/3)

I Amazon Web Services EC2I Can be expensive (or slow)I Still a viable alternative for devsI No offline developmentI Volunteers can’t have access (but could provide AMIs)

I LXC + PuppetI Devs will need access to a Linux hostI Closely matches our production environmentI Linux-only (Win/Mac devs need to virtualize)

What are containers?

I Operating system level hypervisor

I ”One kernel, many userlands”

I In Linux these are:

I CGroups

I OpenVZ

CGroups (Explained)

I C(ontrol) groups

I Feature of the Linux kernel since 2.6.24

I Allows resource isolation (CPU, memory, network, devices)

I Can run a single process or entire system in isolationI Similar to chroot, but has other advantages

I Virtual network interfacesI Resource constraints (memory, CPU, more)I Separate process tableI Separate user/group tables

LXC (Explained)

I Set of convenience commands to facilitate use of CGroupsI Basic operations:

I CreateI DestroyI StartI StopI Others: Clone, console, (un)freeze, execute

I Creation: Uses ’template’ scripts for creating containers

I Can handle complex resource setups

Comparison to other dev environment strategies

I Keeps cost down due to lower resource requirements vs VMs

I Supported upstream by our vendor and good communitysupport (ML + IRC)

I Less flexibility (Linux-only, single-kernel, less security)

I Entirely in vanilla Linux kernel, no out-of-tree patches

Reasons you might choose another container-basedsolution (Docker)

I Desire the portability of Dockerfiles

I Can accept the maintenance of cleaning old instances

I Developers aren’t using Linux as their primary OS

Other container hypervisors

I OpenVZI Not many active contributorsI Has better isolationI Out-of-kernel patches required to useI Some large-scale VPS hosting companies use this

Configuration Management explanation (1/2)

I Many flavors including Puppet, Chef, Ansible, Salt, etc

I Approach used in this talk is agnostic

I For example:

Basic puppet pattern

1 class apache {

2 package { ’httpd’:

3 ensure => installed }

4 file { ’/etc/httpd/httpd.conf’:

5 source => "puppet :/// modules/httpd/service" }

6 service { ’httpd’:

7 ensure => running ,

8 enable => true }

9 Package[’httpd’] -> File[’httpd’] -> Service[’httpd ’]

10 }

= Configuration Management explanation (2/2)

I Higher-level classes (bricks, metaclasses, etc)

I Used to set up ’machine types’ (such as a web server)

Machine types

1 class webserver {

2 include apache

3 include nagios ::nrpe:: webserver

4 include logstash :: webserver }

5

6 node /^web .*\. dc1\. example \.com$/ {

7 include webserver

8 }

Initial impressions and roadbumps

I LXC CentOS template only added very recently

I Still no template for RHEL6

I Kernel audit grumbles

I Things are fast

Our dev environment offering

I Custom LXC templates

I Vagrant scripts to give to devs who would like to replicate

I Masterless puppet

I Templates on our public wiki

Custom LXC Templates (1/3)

I Regular templates create a ’golden image’ copy that lives in/var/cache/lxc/

I Written in shell, executed on lxc-createI Wrote custom LXC templates

I http://github.com/bkero/lxc-templates

Custom LXC Templates (2/3)

I Try:I Adding extra packagesI Apply a puppet base class (sometimes bad idea)I Pre-install puppet certsI Creating multiple containers (!)

Custom LXC Templates (3/3)

lxc-centos-hgweb

1 #!/bin/bash

2 download_centos () {

3 CACHE_DIR =/var/cache/lxc/lxc -centos -hgweb

4 INSTALL_ROOT=$1

5 PKG_LIST="yum initscripts rootfiles \

6 puppet librarian -puppet"

7

8 mkdir $INSTALL_ROOT

9 if [ -d $CACHE_DIR ]; then

10 cp -rv $CACHE_DIR $INSTALL_ROOT/rootfs; exit 0; fi

11 add_yum_repos

12 yum --installroot $INSTALL_ROOT -y \

13 --nogpgcheck $PKG_LIST

14 }

15 apply_puppet () {

16 cp -rv /data/incoming/hgweb /etc/puppet/modules/

17 librarian -puppet init

18 puppet apply -e ’include hgweb ’

19 }

How LXC improves CI

I Clean vanilla systems every time

I Allows multiple isolated systems to interact per CI-run

I Much faster turn-around time

I Allows devs to easily reproduce CI environment

How LXC improves prototyping

I Allows testing new code in production OS natively

I Eases testing of new code in multiple environments

I More than one or two won’t overload the host

I Programmatic creation of faux-production environment isdifficult in VMs, much easier with LXC

Productivity enhancements

I Things are fast

I We can write tooling to let developers have (almost) exactreplicas of entire production environments*

I * (Footnote: It’s RHEL, so we can’t without scary licensingproblems)

I Faster turnaround time

I Can attract more (external) contributors due to the ease ofconstructing a dev environment

Holistic Testing

I Able to spin up multiple containers at once

I Each container can run a single infrastructure component

I Containers can interact on a virtual network (Linux Bridge)

I Can be used to test the end result of a system

Holistic Testing (Example)

I loadbalancer1.test tcp/80 to webserver1.test

I webserver1.test tcp/3306 to mysql1.test

I webserver1.test tcp/6379 to redis1.test

I webserver1.test tcp/11211 to memcache1.test

Case Study: Socorro

I ”Firefox Crash Reporter”I Infrastructure

I ProcessorsI CollectorsI Admin hosts

I Deployment is handled through subclasses of a single puppetmodule

I Continuous integration working on codebase

I Failed CI runs can be replicated by devs

Adoption inside Mozilla

In Review

I Before: Little to no testing

I Before: Getting devs a production-esque environment was alot of manual work and cost

I Now: Some teams are rolling it out (master documentationon wiki, invitation to copy wiki template)

I Now: Deployed alongside devs using personal EC2 instances(personal preference)

Demo time!