dev to delivery with puppet, vagrant and aws
DESCRIPTION
"Dev to Delivery with Puppet, Vagrant and AWS" by Sam Bashton of Bashton Ltd. at Puppet Camp London 2013. Find the video here: http://puppetlabs.com/community/puppet-campTRANSCRIPT
DEV TO DELIVERY WITHPUPPET, VAGRANT AND AWS
SAM BASHTON, BASHTON LTD
ABOUT MELinux guy since Slackware, floppy disks and root + bootUsing Puppet since 2007Run a company in Manchester, North West EnglandWe provide outsourced ops for other companies
TOOLS FOR THE DAY
WHAT IS THE POINT OF THISTALK?
WHAT WE HAVEDevIntegrationQAStageLive
WHICH ENVIRONMENTS AREMANAGED BY PUPPET?
DevIntegrationQAStageLive
WHAT WE'RE AFTERConfidence that everything will work correctly inproductionConsistency between environments
OPS AND DEVS CO-OPERATING
Previously:Devs built stuffLater, Ops came and built production infrastructure
This caused many IT problemsThe solution?
OPSVELOPMENT
OPS AND DEVS WORKINGTOGETHER
Ops need to be involved in development planning processPuppet modules and manifests should be selected/built aspart of the development process
DEVELOP ON PUPPETPROVISIONED
ENVIRONMENTSAs early as possible, all dev should be done on systemsbuilt from PuppetPuppet manifests get tested as part of the developmentprocess
VAGRANTBuilds virtual machines, optionally from Puppet manifestsMakes it easy to spin up short-lived dev instancesQuick to get working
Avoid ops being a blocker for dev
A WORKFLOWDevelopment happens on Vagrant VM(s)Deployment to all shared environments happens viaJenkins
PUPPET CONFIGThere should be only one set of Puppetmanifests/modules
Tested deployed and merged through software testenvironments
ONE SET OF MANIFESTS,MANY ENVIRONMENTS
Different environments need different configResource locationsSettings
DEALING WITH DIFFERENTENVIRONMENTS
HieraRemoves the need for ugly if/else blocksPut anything that differers by environment in a separatefileCan encrypt with hiera-gpg if data sensitive
HIERA.YAML:hierarchy: - %{environment} - common
VAGRANTFILEVagrant.configure(VAGRANTFILE_API_VERSION) do |config| config.vm.box = "centos64-lxc" config.vm.hostname = "puppetconf-example" config.vm.provision :puppet do |puppet| puppet.manifests_path = "puppet/manifests" puppet.manifest_file = "site.pp" puppet.module_path = "puppet/modules" puppet.hiera_config_path = 'puppet/hiera.yaml' puppet.options => ["--environment", "localdev"] endend
TO DEVELOP:Start of the day, dev runs vagrant up and gets the latestenvironmentCode/objects sit in a shared vagrant volumeEnd of the day, or when new Puppet manifests/modulesare available, vagrant destroy is run
VAGRANT PROVISIONERSAvoid VirtualBox wherever possibleSlow, prone to taking down host machineOn Linux, vagrant-lxc is speedyVMWare Fusion for non-free fruit-based Unix
VAGRANT AND AWSUse Vagrant to bring up machines in AWS usingvagrant-aws plugin
Makes it easy to share work in progressMeans VirtualBox doesn't crash your laptopHas cost implications
QA/STAGINGENVIRONMENTS IN AWS
Merge to appropriate branch in gitJenkins takes over
ADVANTAGE OF AWSGreat thing about AWS - we don't need to run our testenvironments all the timeHave the environments only when you need them
TESTING VS LIVEUse the money saved to build better environmentsMinimise differences between testing and liveIn particular, test on environments with relevant HA asearly as possible
SPEEDING UP THE PROCESSSome resources, in particular DBs can be slow to provision(30 mins plus)Could just run 24/7One approach: pilot light provisioning
PILOT LIGHT PROVISIONINGTiers built using autoscaling groupsMinimum instance count is 0Jenkins sets desired capacity appropriately on deployReset to 0 via a recurring scheduled operation on ASGand/or Jenkins job
CONCLUSIONSInfrastructure development should run in parallel tosoftware dev
This means devs + ops must co-operate
Minimise differences from production at all stagesIf a dev can't see the problem in their environment,you're much more likely to get woken up by it
QUESTIONS? COMMENTS?Sam Bashton
Twitter: @bashtoni
(Psst.. )http://www.bashton.com/jobs/
REFERENCES, LINKSVagrantvagrant-lxchiera-gpgMasterless Puppet + AWS