releasing puppet: automating packaging for many platforms or 'make all the things' -...
DESCRIPTION
"Releasing Puppet: Automating Packaging for Many Platforms or 'Make all the things'" by Moses Mendoza, Release Engineer, Puppet Labs. Presentation Overview: A year ago, Puppet Labs had fewer than five fully packaged projects with disparate, difficult packaging workflows that took hours of manual work to accomplish identical results, with varying degrees of success. Since then, we have standardized workflow, added new products, expanded platform support, and removed manual steps (no more hand-crafted builds). This talk/workshop will describe how we have automated entire this process down to a single command from within the source code repo of most of our projects, enabling repeatable, automated packaging, and made packaging available to anyone using our projects’ source on github. This enables us to scale. Once upon a time adding new products to our release process was a major strain on the team. Speaker Bio: Moses Mendoza grew up near Portland, Oregon. Prior to working with Puppet Labs, Moses was primarily a System Administrator, most recently for the Art Institute of Portland as a Mac OSX System Administrator. At Puppet Labs, Moses Mendoza is working to build and scale the automated pipeline and improve continuous delivery at Puppet Labs.TRANSCRIPT
Packaging: Or how to make all the thingsMoses Mendoza and Matthaus OwensRelease Team | Puppet Labs @MosesMendoza@mlitteken
Friday, August 23, 13
puppetconf.com #puppetconf
Introduction
• Why package at all?
• How would I do this?
• Architecting for abstraction
• There be dragons - danger!
• How the F* we did this
Friday, August 23, 13
puppetconf.com #puppetconf
You can deliver more than the source.
Friday, August 23, 13
puppetconf.com #puppetconf
Why bother?
Friday, August 23, 13
puppetconf.com #puppetconf
Why?
• Ease adoption
• Format familiarity
• MOAR power!
• Easier to deploy
• Consistent user experience
• End build tool proliferation
Friday, August 23, 13
puppetconf.com #puppetconf
If it’s so awesome, why isn’t everyone doing it?
• Expense!
• Learning stuff is hard!
• Maintenance
• NO demand (or so you think)
• Somebody’s already doing it for me!
Friday, August 23, 13
puppetconf.com #puppetconf
But wait!
• Expense (start small! start free!)
• Learning stuff is hard! (Learning is fun!)
• Maintenance (you’re right! That is hard!)
• NO demand (or so you think)
• Somebody’s already doing it for me!
• control your release
Friday, August 23, 13
puppetconf.com #puppetconf
Create demand
Friday, August 23, 13
puppetconf.com #puppetconf
Alright, I’m in. Now what?
Friday, August 23, 13
puppetconf.com #puppetconf
Level up! packaging++
• Pick your distro
• Do some research about the distro
• Read the Maintainers Manual
• http://www.debian.org/doc/manuals/maint-guide/
• https://fedoraproject.org/wiki/Packaging:Guidelines
Friday, August 23, 13
puppetconf.com #puppetconf
Answer the following:
• Where do I put stuff? (FS pathing)
• How does the system handle services?
• What are the package management tools?
• Can they express dependencies?
• What does the metadata look like?
Friday, August 23, 13
puppetconf.com #puppetconf
And these also:
• How does versioning work?
• Can I uninstall?
• Upgrade?
• How are configuration files handled?
• Can I do pre-install configuration? post-?
Friday, August 23, 13
puppetconf.com #puppetconf
Package and Sign
• Fire up a VM and play with the tools
• Start small
• Package your project
• Who made this package, anyway?
• Make a gpg key and publish it
• Sign your package!
Friday, August 23, 13
puppetconf.com #puppetconf
Potential Next Steps...
• Get your package to your users!
• Become a distro maintainer
• Host basic downloads
• Host package repositories (yum, apt...)
Friday, August 23, 13
puppetconf.com #puppetconf
Pitfallsask us how we know.
• Sprawl...... . .
• Variation in workflows
• Historically preserved cruft
• Shipping without testing
• Over-abstraction
Friday, August 23, 13
puppetconf.com #puppetconf
Architect for Abstractiontools on tools on tools.
• Snowflakes Are Bad (tm)
• Cross-platform tools
• Start small but lay expansion groundwork
• Don’t hard code paths
• Work in a workspace
• Make infrastructure assumptions configurable
Friday, August 23, 13
puppetconf.com #puppetconf
Architect for Abstractiontools on tools on tools.
• Define a user interface API for automation
• Try a layered approach
• Entry points call to Abstractions call to System tools
Friday, August 23, 13
puppetconf.com #puppetconf
how Puppet Labs does it
el fedora debian
ubuntuandmore
windows
rubygems tarball osx
build ship
Friday, August 23, 13
puppetconf.com #puppetconf
the base layer
rake ruby make - tool for automation in ruby
ruby methods for utilities and system tool wrappers
system tools to create packages
Friday, August 23, 13
puppetconf.com #puppetconf
rake ruby make
Friday, August 23, 13
puppetconf.com #puppetconf
More than in the middle
• Ruby utilities to wrap system tools
• Data lives in projects, not in tools
• Packaging automation in a separate repo
• Automation completely project agnostic
Friday, August 23, 13
puppetconf.com #puppetconf
Setup workflow
puppet/ext/packaging
github.com/puppetlabs/packaging
puppet/Rakefile
rake package:apple
Friday, August 23, 13
puppetconf.com #puppetconf
so far so good!
• Wherefrom cometh the build hosts?
• How do I get from here to them?
but...
Friday, August 23, 13
puppetconf.com #puppetconf
System configuration?github.com/puppetlabs/puppetlabs-{deb,rpm}-builder
Friday, August 23, 13
puppetconf.com #puppetconf
• Jenkins has: master/slave, queuing, triggers
• Builders become Jenkins slaves
• Builders categorized by capability (label)
Friday, August 23, 13
puppetconf.com #puppetconf
Right place, right time
• Create per-build jobs from templates
• Builds are matrix jobs
• Groovy Label Assignment Plugin to assign
• Use labels to assign jobs
Friday, August 23, 13
puppetconf.com #puppetconf
Friday, August 23, 13
puppetconf.com #puppetconf
• On-demand capacity
• Parallelized
• Automatically triggered
• Jobs chained into a pipeline on the fly
• Open source: github.com/puppetlabs/packaging
Where we are today
Friday, August 23, 13
Thank YouMoses Mendoza and Matthaus OwensRelease Engineering | Puppet Labs @MosesMendoza@mlitteken
Collaborate. Automate. Ship.
Friday, August 23, 13
Follow us on Twitter @puppetlabs
youtube.com/puppetlabsinc
slideshare.net/puppetlabs
Collaborate. Automate. Ship.
Friday, August 23, 13