multi node deployments - viewdeck€¦ · multi node deployments chefconf 2015 . kennon kwok...
TRANSCRIPT
Multi Node Deployments
ChefConf 2015
Kennon Kwok Solutions Engineer - Chef Software, Inc.
“mini-me”
Mars
Zack Zondlo
Previously: Senior Infrastructure Engineer, CareCloud
Previously: Senior Infrastructure Engineer, Selling Source
Support Engineer
5
Kevin Dickerson
Kevin is a former sandwich artisan from Alaska. He's previously worked at places like Carnegie Mellon University, Salesforce, and Chef. Interests include machine learning algorithms, super big web applications, and software automation.
Solutions Engineer - Chef Software, Inc.
Sean Carolan
Previously: Unix Systems Administrator at Electronic Arts Worked with CFEngine, Puppet, and now Chef Now: Sips martinis with the sales team
Solutions Architect - Chef Software, Inc.
Introduce Yourselves Name
Current job role
Previous job roles/background
Experience with Chef and/or config management
Favorite Text Editor
8
Agenda
Introduction to Chef Provisioning Emerging usage patterns - cluster cookbook - provisioning node - state of cookbook patterns Chef Provisioning walkthrough - WordPress cluster
Choose your own adventure Provision a stack of your own We’ll provide resources and help!
Morning Afternoon
Other logistics
• Morning Break: 10:30-11:00 • Lunch: 12:30-1:45 • Afternoon Break: 3:15-3:45 • All food is on the 2nd level of Hyatt • Coffee outside of classroom all day • Restrooms are 2 right turns when exiting classroom
Course style
Training is a discussion • Hands-on labs • Lots of typing • Ask questions when they come to you • Ask for help when you need it • We will troubleshoot issues on the spot
Chef Provisioning An Introduction
Chef Provisioning • Allows programatic creation of cloud resources
in Chef Recipes • Allows for multiple tiers to be created in one
shot • Moves more towards “Infrastructure as Code” • Test and repair also applies to infrastructure • Previously known as “chef-metal”
Chef Provisioning • Plugin model lets you write bootstrappers for
your own infrastructure (e.g. VirtualBox, EC2, LXC, bare metal, etc)
• Today we are going to use the AWS driver for Chef Provisioning
https://github.com/chef/chef-provisioning-aws
Provisioning workflows • Chef Server with “provisioning node” • Chef Client local-mode
Amazon Web Servies • This isn’t a course on AWS (that’s next door) but
we’ll be touching on some AWS concepts. • We’ve taken some steps before class to set up
your provisioning workstations for use with AWS.
Provisioning workstation • EC2 instance, CentOS 6 • Contains pre-installed software: ChefDK 0.4.0,
chef-provisioining, chef-provisioning-aws, awscli • AWS credentials • user: chef • password: chef.io
Chef Provisioning WordPress walkthrough
Success Problem - Fully provision WordPress
cluster - Soup-to-nuts - 3 tiers:
- database (mysql) - application (php) - load balancer (ELB)
- github.com/kennonkwok/wordpress-simple
- Build a cookbook that describes the cluster
- One chef-client run to provision the entire cluster
Wordpress cluster steps 1. Build database machine 2. Build application server(s) 3. Build load balancer
About those playing cards… http://git.io/jR9L
REMOTE
SSH to provisioning host
[email protected]'s password:
Last login: Mon Mar 16 22:26:01 2015 from 107.3.143.174
[chef@ip-10-221-14-71 ~]$
$ ssh YOUR_HOST
REMOTE
Chef generate cookbook
Usage: chef generate repo NAME [options]
-C, --copyright COPYRIGHT Name of the copyright holder - defaults to 'The Authors'
-m, --email EMAIL Email address of the author - defaults to '[email protected]'
-a, --generator-arg KEY=VALUE Use to set arbitrary attribute KEY to VALUE in the code_generator cookbook
-I, --license LICENSE all_rights, apache2, mit, gplv2, gplv3 - defaults to all_rights
-p, --policy-only Create a repository for policy only, not cookbooks
-g GENERATOR_COOKBOOK_PATH, Use GENERATOR_COOKBOOK_PATH for the code_generator cookbook --generator-cookbook
$ chef generate cookbook --help
REMOTE
Chef generate cluster
…
* directory[/home/chef/wordpress-cluster/recipes] action create
- create new directory /home/chef/wordpress-cluster/recipes
* template[/home/chef/wordpress-cluster/recipes/default.rb] action create_if_missing
- create new file /home/chef/wordpress-cluster/recipes/default.rb
- update content in file /home/chef/wordpress-cluster/recipes/default.rb from none to 9f3d82
(diff output suppressed by config)
* execute[initialize-git] action run
- execute git init .
* cookbook_file[/home/chef/wordpress-cluster/.gitignore] action create
- create new file /home/chef/wordpress-cluster/.gitignore
- update content in file /home/chef/wordpress-cluster/.gitignore from none to dd37b2 (diff output suppressed by config)
$ chef generate cookbook wordpress-cluster
REMOTE
Navigate to new cookbook
(no output)
$ cd wordpress-cluster
Anatomy of a cluster cookbook wordpress-cluster
+---.chef
\---knife.rb
+---attributes
+---cookbooks
+---recipes
\---_aws_settings.rb
\---database.rb
\---app.rb
\---lb.rb
• .chef directory with knife.rb • attributes work like other
cookbooks • cookbooks directory is
in .gitignore • recipes only contain
provisioning code • recipes can contain
application tiers or driver specific code
REMOTE
Create .chef directory
(no output)
$ mkdir .chef
Create File
current_dir = File.dirname(__FILE__)
chef_repo_path "#{current_dir}/.."
.chef/knife.rb
Best Practice De-couple provider specific code
Create file
require 'chef/provisioning/aws_driver' with_driver 'aws' with_machine_options( bootstrap_options: { instance_type: node['wordpress-cluster']['aws']['flavor'], key_name: node['wordpress-cluster']['aws']['key_name'], security_group_ids: node['wordpress-cluster']['aws']['security_group_ids'] }, ssh_username: node['wordpress-cluster']['aws']['ssh_username'], image_id: node['wordpress-cluster']['aws']['image_id'] ) aws_key_pair node['wordpress-cluster']['aws']['key_name']
recipes/_aws_settings.rb
REMOTE
Make attributes directory
(no output)
$ mkdir attributes
Create file
default['wordpress-cluster']['aws']['flavor'] = 't1.micro'
default['wordpress-cluster']['aws']['key_name'] = YOUR_KEY_NAME_HERE
default['wordpress-cluster']['aws']['security_group_ids'] = 'sg-c49e0ff4'
default['wordpress-cluster']['aws']['ssh_username'] = 'root'
default['wordpress-cluster']['aws']['image_id'] = 'ami-37361807'
attributes/default.rb
• Centos6 image with chef account • default security group is open to world on 22,80,443 and
open internally • use a unique key_name (e.g. kennon-wp-cluster)
Create file
require 'chef/provisioning' include_recipe 'wordpress-cluster::_aws_settings'
machine 'wordpress-database' do
tag 'wordpress-database' recipe 'wordpress-simple::database’ action :converge
end
recipes/database.rb
• The machine resource defines one (or more) machines and can be combined into clusters of machines.
• This allows clusters to be maintained in a version control system and to be defined using multi-machine orchestration scenarios.
Best Practice Create a destroy recipe
Create File
include_recipe 'wordpress-cluster::_aws_settings' machine_batch do machines search(:node, '*:*').map { |n| n.name } action :destroy end
recipes/destroy_all.rb
• Just a quick clean up recipe. • :destroy action terminates instance, cleans up node and
client data
Modify file
name 'wordpress-cluster' maintainer 'The Authors' maintainer_email '[email protected]' license 'all_rights' description 'Installs/Configures wordpress-cluster' long_description 'Installs/Configures wordpress-cluster' version '0.1.0’ depends 'wordpress-simple'
metadata.rb
Berkshelf • Cookbook dependency manager • Included in ChefDK • We’ll use it here to download a WordPress
cookbook and its dependencies
Modify file
source 'https://supermarket.chef.io'
metadata
cookbook 'wordpress-simple', git: 'https://github.com/kennonkwok/wordpress-simple.git'
Berksfile
Helpful one-liners $ rm -fr cookbooks; berks vendor cookbooks
$ chef-client -z -o wordpress-cluster::destroy_all
- First line: re-vendor cookbooks - Second line: destroys cluster!
REMOTE
Gather cookbook dependecies
…
Vendoring selinux (0.9.0) to cookbooks/selinux
Vendoring smf (2.2.6) to cookbooks/smf
Vendoring tar (0.6.0) to cookbooks/tar
Vendoring wordpress-cluster (0.1.0) to cookbooks/wordpress-cluster
Vendoring wordpress-simple (0.1.0) to cookbooks/wordpress-simple
Vendoring yum (3.5.3) to cookbooks/yum
Vendoring yum-epel (0.6.0) to cookbooks/yum-epel
Vendoring yum-mysql-community (0.1.14) to cookbooks/yum-mysql-community
[chef@ip-10-157-113-135 wordpress-cluster]$
$ berks vendor cookbooks
REMOTE
Converge!
... * mysql_database_user[wordpressuser] action grant
- Granting privs for 'wordpressuser'@'%'
Running handlers:
Running handlers complete
Chef Client finished, 34/44 resources updated in 305.018417375 seconds
- run 'chef-client -l auto' on wordpress-database
[2015-03-23T23:58:43+00:00] WARN: Skipping final node save because override_runlist was given
Running handlers:
Running handlers complete
Chef Client finished, 1/1 resources updated in 429.529162915 seconds
$ chef-client -z -o wordpress-cluster::database
Wordpress cluster steps 1. Build database machine ✓ 2. Build application server(s) 3. Build load balancer
Create app cookbook
include_recipe 'wordpress-cluster::_aws_settings' machine 'wordpress-app' do tag 'wordpress-app' recipe 'wordpress-simple::app' action :converge converge true attributes lazy {{ 'wordpress-simple' => { 'dbhost' => search(:node, 'tags:wordpress-database').first['ipaddress'] } }} end
recipes/app.rb
REMOTE
Converge!
…
* service[httpd] action restart
- restart service service[httpd]
Running handlers:
Running handlers complete
Chef Client finished, 16/21 resources updated in 78.8732913 seconds
- run 'chef-client -l auto' on wordpress-app
[2015-03-24T00:31:53+00:00] WARN: Skipping final node save because override_runlist was given
Running handlers:
Running handlers complete
Chef Client finished, 1/1 resources updated in 199.882265438 seconds
$ rm -fr cookbooks; berks vendor cookbooks
$ chef-client –z –o wordpress-cluster::app
Wordpress cluster steps 1. Build database machine ✓ 2. Build application server(s) ✓ 3. Build load balancer
Create load balancer recipe (using ELB)
include_recipe 'wordpress-cluster::_aws_settings'
load_balancer 'your-elb-name' do
load_balancer_options :availability_zones => ['us-west-2a', 'us-west-2b', 'us-west-2c'],
:listeners => [{
:port => 80,
:protocol => :http,
:instance_port => 80,
:instance_protocol => :http
}]
machines ['wordpress-app']
end
recipes/lb.rb
Load balancer name must be unique in class/account
REMOTE
Converge!
…
{:port=>80, :protocol=>:http, :instance_port=>80, :instance_protocol=>:http}
- with security groups
- add machines wordpress-app
- create data bag loadbalancers at http://localhost:8889
- create data bag item wordpress-elb at http://localhost:8889
- add location = {"driver_url"=>"aws", "driver_version"=>"0.4.0", "allocated_at"=>"2015-03-24 02:55:31 UTC"}
[2015-03-24T02:55:31+00:00] WARN: Skipping final node save because override_runlist was given
Running handlers:
Running handlers complete
Chef Client finished, 1/1 resources updated in 21.639462242 seconds
$ rm -fr cookbooks; berks vendor cookbooks
$ chef-client –z –o wordpress-cluster::lb
Wordpress cluster steps 1. Build database machine ✓ 2. Build application server(s) ✓ 3. Build load balancer ✓
Tie it all together!
#
# Cookbook Name:: wordpress-cluster
# Recipe:: default
#
# Copyright (c) 2015 The Authors, All Rights Reserved.
include_recipe 'wordpress-cluster::database'
include_recipe 'wordpress-cluster::app'
include_recipe 'wordpress-cluster::lb'
recipes/default.rb
Add lb to destroy_all recipe
include_recipe 'wordpress-cluster::_aws_settings' machine_batch do machines search(:node, '*:*').map { |n| n.name } action :destroy end load_balancer 'your-elb-name' do action :destroy end
recipes/destroy_all.rb
Wordpress cluster steps 1. Build database machine ✓ 2. Build application server(s) ✓ 3. Build load balancer ✓ 4. Scale out application layer!!
Scale out app servers
TODO
recipes/app.rb
REMOTE
Converge!
…
* service[httpd] action restart
- restart service service[httpd]
Running handlers:
Running handlers complete
Chef Client finished, 16/21 resources updated in 78.8732913 seconds
- run 'chef-client -l auto' on wordpress-app
[2015-03-24T00:31:53+00:00] WARN: Skipping final node save because override_runlist was given
Running handlers:
Running handlers complete
Chef Client finished, 1/1 resources updated in 199.882265438 seconds
$ rm -fr cookbooks; berks vendor cookbooks
$ chef-client –z –o wordpress-cluster::app
REMOTE
Soup to nuts!
- create data bag item wordpress-elb at http://localhost:8889
- add location = {"driver_url"=>"aws", "driver_version"=>"0.4.0", "allocated_at"=>"2015-03-24 03:34:09 UTC"}
[2015-03-24T03:34:09+00:00] WARN: Skipping final node save because override_runlist was given
Running handlers:
Running handlers complete
Chef Client finished, 3/3 resources updated in 629.509273264 seconds
$ chef-client –z –o wordpress-cluster::default
Discussion What questions can I answer for you? • Chef Provisioning • machine resource
Introduction to Chef Provisioning Emerging usage patterns - cluster cookbook - provisioning node - state of cookbook patterns Chef Provisioning walkthrough - WordPress cluster
Choose your own adventure Provision a stack of your own We’ll provide resources and help!
Morning Afternoon
Choose your own Adventure • Scale out the app layer • LXC or Docker provisioning • Vagrant provisioning (on your laptop) • AWS allthethings! (VPC, RDS…) • Try a different application stack
Break