ChefSmart infrastructure automation

Who am I

• Johannes Skov Frandsen

• Works primarily with Open Source

• Open Source enthusiast since 2000

• Mostly into web development og process automation.

What is devops• Your software product is not

only the application itself but also the platform it is running on

• Methods used for software development that can be valuable in the field of operations

• “Missing link” between developers and sysadmins

What are we trying to solve?• Differences in configuration of each environment

Famous: „Works for me”

• Big amount of time required to configure new environment

• Manual configuration changes are prone to errors

• Lack of local development environment encapsulation

• Lack of version control for configuration

How are we trying to solve it• Make tasks repeatable

• No manual steps and idempotent.

• Make tasks rapid

• Fast to build, deploy and restore

• Make systems resilient

• Automated reconfiguration

Devops working areas

• Configuration management

• Deployment automation (not todays topic)

• Build automation (not todays topic)

Configuration management• The two biggest contenders are Puppet and

Chef

• Both a written in Ruby

• Chef used Ruby as a DSL, Puppet use resource declaration files.

• If you are more “Dev” than “Ops”, Chef is probably your best fit and vice versa.

Chef : http://www.getchef.com/chef/ Puppet : http://puppetlabs.com/

• A systems and cloud infrastructure automation framework

• Makes it easy to deploy servers and applications to any physical, virtual, or cloud location

• No matter the size of the infrastructure

How to use Chef

• Use it to configure a single machine (chef-solo)

• Or your entire infrastructure (chef client-server)

• Use it on-site or in the cloud (build in to amazon and Rackspace)

• Use in you local development environment.

Chef in general

Provisioning Configuring Integration

Chef is used to describe abstract definitions as code, defining how you want individual parts of

you infrastructure constructed.

Chef provisioning• Chef can administrate machines via a REST API. • Chef supports Kickstart on Linux, Jumpstart on Solaris and

NIM on AIX. • In virtualised environments, Chef integrates with libvirt and

hypervisors like XEN, KVM, VMware. Chef works well with VirtualBox.

Provisioning

Configuration• Chef is a complete configuration handling tool where recipes

and roles are used to describe how servers are configured.

• You can describe which packages must be installed, what services that needs to run, and which configuration files that needs to be edited.

• Chef can ensure that all resources are correct installed and will only make changes to the system if needed.

• Chef works well in tandem with existing configurations scripts like shell or perl scripts.

Configuring

Integration• Chef can handle separation of configuration logic and

configuration data.

• As an example, with Chef, when you install a new load balancer, you can search for installed http servers and automatically add them to you configuration.

• Likewise, if you install a new memcached server, you can advertise this to services that need memcached and automatically add the new server to their configuration.

Integration

Chef terms• Cookbooks

• Cookbooks describes how to install an individual pieces of software in a generic way across any number of nodes. Configuration options and settings are specified as attributes with sensible defaults.

• Environments

• Different environments can be specifies to distinguish groups of node from others. When a cookbook is provisioned in a environments, the attributes specified in the cookbook is overridden by those specified in the environment.

• Roles

• Roles work much the same way as environments, but instead defines a node role. This allows a cookbook to be used on different nodes with different configurations. When a cookbook is provisioned in a role, the attributes specified in the cookbook is overridden by those specified in the role.

• Nodes

• Nodes are the finest level of granularity in Chef. The node names a specific instance in the setup and its configuration can override any attribute define either cookbook, environment or role. Chef server uses node configurations for provisioning Chef clients.

• Data Bags

• A global variable that is stored as JSON data and is accessible from a Chef Server. The contents of a data bag include sensitive information and is encrypted.

Solo or Client/Server• Chef Solo

• In cases where you can't use the client server model, Chef solo can be used to provision the nodes locally. This is handy for provisioning the chef server itself or for testing new recipes before they are deployed to the Chef server.

• Chef Server

• The Chef server manages a repository of all the cookbooks, environments, roles and nodes in your setup. The Chef server monitors all the node it manages.

• Chef Client

• The Chef client request its configuration from the Chef server, download the required software and configures it self.

Show me some codeChef “Hello World” recipe

package "logrotate" do action :installend

...if platform?("redhat") node[:php5][:packages][:redhat].each do |pkg| package pkg do action :install end endendif platform?("suse") node[:php5][:packages][:suse].each do |pkg| package pkg do action :install end endend...

default.php5.packages.redhat = [ "php", "php-gd", "php-mysql", "php-odbc", "php-pdo", "php-soap", "php-xml", "php-xmlrpc", "php-mbstring", "php-mcrypt"]!default.php5.packages.suse = [ "php5", "apache2-mod_php5", "php5-calendar", "php5-ctype", "php5-curl", "php5-dom", "php5-exif"]

Chef php cookbookRecipe Attributes

Templates and scripts...template "/etc/php5/conf.d/memcache.ini" do source "extension" mode 0644 owner "root" group "root" variables({:extension => "memcache.so"}) notifies :restart, "service[apache2]"end...

extension=<%= @extension %>

Recipe Template

...cookbook_file "/tmp/install_memcache.exp" do source "install_memcache.exp" mode 0600 owner "root" group "root"endscript "install_pecl_memcache" do interpreter "bash" user "root" cwd "/tmp" code <<-EOH cat /tmp/install_memcache.exp | expect -- rm /tmp/install_memcache.exp EOHend...

#!/usr/bin/expectspawn pecl install memcache!set timeout -1!expect "Enable memcache session handler support?"send "yes\r"!expect eof

Recipe Script

Providers

action :replace do execute "sed -e \"s|#{new_resource.search}|#{new_resource.replace}|g\" -i #{new_resource.file}"end...

...service "apache2" do action :stopend!package "apache2" do action :installend!# make backup of /etc/apache2/listen.confbackup "/etc/apache2/listen.conf"!# change listening portsed "/etc/apache2/listen.conf" do action :replace search "^Listen [0-9]\\{1,5\\}" replace "Listen #{node[:apache2][:port]}" end!#Allow named virtual hostssed "/etc/apache2/listen.conf" do action :replace search "^#NameVirtualHost \\*:[0-9]\\{1,5\\}" replace "NameVirtualHost *:#{node[:apache2][:port]}"end...

action :create do execute "cp #{new_resource.file} #{new_resource.file}.#{new_resource.extension}" do not_if {::File.exists?("#{new_resource.file}.#{new_resource.extension}")} only_if {::File.exists?("#{new_resource.file}")} endend

Recipe Providers

Roles{ "name": "skeleton", "default_attributes": {}, "override_attributes": {}, "json_class": "Chef::Role", "description": "This installs a skeleton server.", "chef_type": "role", "run_list": [ "recipe[networking]", "recipe[base]" ]}

{ "name": "alfresco", "default_attributes": {}, "override_attributes": {}, "json_class": "Chef::Role", "description": "This installs a alfresco server.", "chef_type": "role", "run_list": [ "recipe[networking]", “recipe[base]", "recipe[alfresco]", "recipe[alfresco::ssh]", "recipe[alfresco::backup]" ]}

Role skeleton Role alfresco

Environments{ "name": "_default", "description": "The default Chef environment", "cookbook_versions": { }, "json_class": "Chef::Environment", "chef_type": "environment", "default_attributes": { }, "override_attributes": { }}

{ "name": "production", "description": “Production environment", "cookbook_versions": { "app-master" : "1.1.3", "app-slave" : "1.1.3", "db-master" : "1.1.3", "db-slave" : "1.1.3" }, "json_class": "Chef::Environment", "chef_type": "environment", "default_attributes": { "postfix": { "aliases": { "root": "someone@example.com" } } }, "override_attributes": { "apache2": { "admin": "someone@example.com"" }, "mysql": { "config": { "innodb_buffer_pool_size": "6144M" }, "replication": { "master": "db-master" } }, "backup": { "server": “files.example.com" } }}

Default Production

Because you can version your cookbooks, different environments can run different versions.

Structure

Lets try it with VirtualBox/Vagrant

VAGRANTFILE_API_VERSION = "2" !Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| config.vm.box = "ubuntu" config.vm.box_url = "https://ubuntu-server13.10.box" config.vm.network :forwarded_port, guest: 80, host: 8080 config.ssh.forward_agent = true config.ssh.shell = "bash -c 'BASH_ENV=/etc/profile exec bash'" config.vm.synced_folder "www/", "/var/www", :create => true ! config.vm.provision :chef_solo do |chef| chef.recipe_url = "https://cookbooks.tar.gz" chef.add_recipe "apache2" chef.add_recipe "php5" end end

VirtualBox : https://www.virtualbox.org/ Vagrant : http://www.vagrantup.com/

Experience• Latest project was running ~50 servers with Chef.

• All developer was using vagrant to get a local development environment auto configured.

• Provisioning and configuration of servers takes minutes… not days.

• There are tons of free cookbooks available online but in our experience you will mostly use them for inspiration and write your own.

Questions

ThanksGet the slide at http://www.slideshare.net/localgod

Anything that is in the world when you're born is normal and ordinary

and is just natural part of the way the world works.

Anything that's invented between when you're fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it.