puppet & perforce: versioning everything for deployments
DESCRIPTION
We all want faster deployments – but how? Learn how to use Perforce to manage everything that goes into deployments, how to use Puppet to manage the state of nodes with Perforce, and how integrating Perforce and Puppet speeds up the adoption of DevOps practices.TRANSCRIPT
#
Christoph Leithner, Founder at celix
Puppet & Perforce: Versioning Everything for Deployments
#
• What is the starting Point of our Journey to Continuous Delivery?
• Continuous Integration– Source Code Checkout– Static Code Analysis– Automated Tests– Compilation– Packaging
Continuous Delivery Gap Analysis
#
• Database Structure• Database Content• Installable Software
– Libraries– Executables– OS Packages– Installers
• Some Type of Deployment
Continuous Delivery Gap Analysis
#
Types of Deployments
PersonalOnly a Guru understands the system and is able to deploy it
Manual
Defined
There enough Documentation so we do not always have to ask the Guru if we want to deploy
ScriptedScripts are available to setup most of the Components of our System
We use IT-Automation to deploy and enforce our Deployment – no manual changes.
Defined Deployments are not possible without IT-Automation, Traceability and Discipline
#
• Not all Changes to Dependencies trigger a Build• DEV-Database is populated with Snapshots • DEV-Database Changes are not tracked• Semi-automated DEV-Deployments cause Delays• Only some aspects of the System are managed by
Continuous Integration – no full Stack Deployment• Builds/Deployments are not reproducible
Common Pain Points
#
Version Everything for Deployments
#
• Developer Code• Deployment Code• Configuration Data• Database Structure• Database Content• Installable Software• Documentation
Version Everything for Deployments
#
Version Puppet Code and Configuration Data
#
Puppet Module
module_name
manifests
files
templates
lib
spec
tests
Modulefile
#
Puppet Environment
puppet module install puppetlabs-ntp --version 0.2.0puppet module install puppetlabs-apache --version 0.6.0
firewall
ntp
stdlib
apache
#
Reusing Modules from Puppet Forge
forge
puppetlabs-apache
puppetlabs-stdlib
puppetlabs-firewall
puppetlabs-apache
puppetlabs-stdlib
puppetlabs-firewall
#
• Module Reuse is the fastest way of Automation• But it is not as easy as it might seem to write a
reusable Puppet Module• “We have this Module managing …, but I cannot
share it. It contains too many Site specific Parameters”
• Separating Module Code from Configuration Data is key for Module Reuse
Module Reuse
#
• Puppet 3.0 integrates Hiera for Configuration Data• Hiera defines a list of Backends• Hiera defines a Lookup Hierarchy
:hierarchy - “%{clientcert}” - “%{environment}” - “%{osfamily}”
Versioning Configuration Data
#
Hiera Site Configururation Data
firewall
stdlib
apache
hiera
#
Hiera in Module Configuration Data
module_name
manifests
files
templates
lib
spec
tests
Modulefile
data
#
Version Database Structure and Content
#
• DB Structure– DB Dump– DB Snapshot– SQL Scripts
• Creating DB Structure with SQL Scripts1. Create all Tables
2. Fill all Tables
3. Create/Enable all Relational Constraints, Triggers etc.
Version DB Structure
#
• DB Content Types– System Data– Configuration Data– Business Object Data
• DB Content Tools– SQL Insert Scripts– Schema or Table Exports/Dumps– Tool Specific Files (e.g. XML)
Version DB Content
#
• DB Migration Tools– OR Mappers – built into the Application
• Rails: Active Records• Java: Flyway
– Evolutionary/Agile Database Tools • dbdeploy• DBmaestro
Version DB Changes
#
Versioning Configuration Data1_create.sql
2_fill.sql
3_enable.sql
create
tables
indexes
...
views
functions
procedures
fill
system
configuration
business_objects
migrate
changelist_A
changelist_B
#
Version Packages and Installers
#
• OS Packages– Different for each Platform and Distribution– Multiple Repositories required– Packages and Metadata
• Installers– Simplify complicated installations– Same Installation on multiple Platforms– Manual Installations not supported
Version Packages and Installers
#
• Puppet Package Resource:
• Ensure Attribute Values– present: Package is installed– latest: Package is upgraded to the latest available – ‘2.4.6’: specified Version of the Package is installed
Managing OS Packages
package {‘apache2’: ensure => present,}
#
• New Package Versions are available online• Can we apply them automatically?
– Is the Package managed by Puppet?– Is the Package a Dependency of a Package managed
by Puppet?– Is there a Requirement for a particular Version of an
updated Package?– Is there a Requirement for a particular Version of a
Dependency of an updated Package?
Managing OS Upgrades
#
• We need to know which is the latest Version of a Package in a certain Environment
• We need to create a local Repository and manage all Changes to it.
• On Ubuntu we use aptly to manage our local Repositories
Managing OS Upgrades
#
Managing Repositories with aptly
Source: http://www.aptly.info/doc/overview/
#
Managing Repositories with aptly
#
• We check in the published Repository into Perforce to get full Traceability
• We can use Perforce to propagate Changes between Environments
• What do we use for Puppet ensure: • latest: Puppet actively tries to upgrade• present: Puppet trusts the Package Manger• ´2.4.6´: if we need to pin an older Version (apt::pin)
Managing Repositories with Perforce
#
• We use a Perforce Stream to track the Work of an Installer in particular Directory
• This way can use a Task Stream to track the work of the Installer on each machine
• Currently we only use this Feature when we automate deployments
• The same concept can be used to tack changes of Directories on managed nodes
Version Installers
#
• Developer Code• Deployment Code• Configuration Data• Database Structure• Database Content• Installable Software• Documentation
Version Everything for Deployments
#
• Full Traceability• Code Reviews in Swarm for all Changes• Ability to make consistent changes across all
Artifacts• Revision Graph for Insights on a particular Artifact• Easy Tracking of the Work of Installers
Benefits of Versioning Everything
##
Thank you!Christoph [email protected]