an introduction to devops with chef

Rethinking IT: An Introduction to DevOps with ChefJulian C. DunnSenior Consultant, Opscode<jdunn@opscode.com>

Wednesday, October 2, 13

Topics• Rethinking IT with DevOps• What Chef is and how it helps• Flavors of Chef• Use cases and demo

I came to hear about Chef.

Why are you throwing DevOps into this?

It’s how Chef works best• Chef is a tool.• It works best when developers and operations

work well together• You don’t need “DevOps” for Chef to be successful.• But it helps.

What is this DevOps? Sounds like a buzzword.

DevOps: Aligning Objectives• Cultural and Professional

movement• Development and

Operations working together

• Leveraging ideas & processes from other industries

• To enable the businessWednesday, October 2, 13

Technology: Business Engaging Customers

Speed of Globalization• 40 years for container

ships to move 70% of seaborne trade

• 22 years for internet access to reach 78% penetration in North America

Globalization• Online retail sales are 7%

of all retail sales• 75% of 2011 Thanksgiving

shoppers did so online• 42% of all retail

purchases were influenced by online research – accounting for ~50% of total retail spending.

WTO Trends in Globalization http://www.wto.org/english/res_e/booksp_e/anrep_e/wtr08-2b_e.pdf

http://www.flickr.com/photos/duke_raoul/2261478794/sizes/l/in/photostream/

Mobile Devices• 42% are smartphones• 58% will be on next

purchase• 4.2 Billion phones

globally for 7.09 Billion people (USCB)

http://ssiknowledgewatch.com/2012/05/09/cell-phones-approach-total-penetration-globally-with-smartphones-moving-toward-market-dominance-2/

http://www.brightsideofnews.com/news/2011/1/26/digital-divide-global-household-penetration-rates-for-technology.aspx?pageid=1

Software: The Interface for Consumtpion

Drivers of IT Innovation

The Result: The Coded Business

How: Redefinition of how to use technology to create business value

Why: To rapidly deliver experiences, goods and services to customers

What: Consumer-facing businesses

The Rise of the Coded Business• Speed of change > speed of skills development• IT is moving from the back office to the front office• Digital consumption• Technology directly supports customer interactions• Companies must move faster to compete

Patterns of the Coded BusinessBusiness

Agility

Development Velocity and Consistency

Continuous Delivery

IT enables Business Agility and becomes a strategic advantage rather than a cost center.

The Coded Business: Coming to Every Enterprise

Manufacturing Financial Services

Retail

Media and Entertainment

High Technology

Healthcare

Scale x Complexity > Skills

DevOps...• Cultural and professional movement• Grew directly from people who have experienced

this transition• Applies directly to traditional IT• Applies directly to other systems of survival in

our culture

DevOps: CAMS• Culture• Automation• Measurement• Sharing

Open Communication• Developers &

Operations talk and listen to one another

• Production & build metrics are available to all

• Current infrastructure is documented

Incentive & Responsibility Align• Create awesome

customer experiences• Responsibility &

accountability, not authority

• Responsible for your own uptime

Respect• You don’t have to be

everyone’s best friend• But you do need to

recognize contributions and treat each other well

• No asshole rule

Culture: Trust• Trust that everyone is

competent & working toward the common goals

• Without trust, the tools don’t matter

DevOps: CAMS• Culture• Automation• Measurement• Sharing

• Chef is (part of) “A”. Works best with C, M and S.

Infrastructure Automation with Chef

http://www.flickr.com/photos/steffenz/337700069/http://www.flickr.com/photos/kky/704056791/

Applications

Infrastructure

http://www.flickr.com/photos/sbh/462754460/

http://www.flickr.com/photos/philliecasablanca/3354734116/

• Networking

• Files

• Directories

• Symlinks

• Mounts

• Routes

• Users

• Groups

• Tasks

• Packages

• Software

• Services

• Configuration

• Other Stuff

Collection of Resources

Acting in Concert

http://www.flickr.com/photos/glowjangles/4081048126/Wednesday, October 2, 13

To Provide a Service

http://www.flickr.com/photos/28309157@N08/3743455858/

And it Evolves

http://www.flickr.com/photos/16339684@N00/2681435235/

Application Server

See Node

Application Server

Application Database

See Nodes

Application Server

Application Databases

See Nodes Grow

Application Servers

See Nodes Grow

Application Servers

Load Balancer

See Nodes Grow

Application Servers

Load Balancers

See Nodes Grow

Application Servers

Application Database Cache

Load Balancers

See Nodes Grow

Application Servers

Load Balancers

Tied Together with Configuration

Application Servers

Load Balancers

Floating IP?

Infrastructure is a Snowflake

Load Balancers

Application Servers

Database Slaves

Application Cache

Database Cache

Database

Evolving Complexity

Complexity Increases Very Quickly

Configuration Management

http://www.flickr.com/photos/philliecasablanca/3354734116/Wednesday, October 2, 13

Golden Images are not the answer

• Gold is heavy

• Hard to transport

• Hard to mold

• Easy to lose configuration detail

http://www.flickr.com/photos/garysoup/2977173063/

JBoss App

Memcache

PostgreSQL Slaves

PostgreSQL Master

NagiosGraphite

Typical Infrastructure

JBoss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite

• Move SSH off port 22

• Let’s put it on 2022

New Compliance Mandate!

JBoss App

Memcache

PostgreSQL Slaves

PostgreSQL Master

NagiosGraphite

• edit /etc/ssh/sshd_config

6 Golden Image Updates

JBoss App

Memcache

PostgreSQL Slaves

PostgreSQL Master

NagiosGraphite

• Delete, launch

3 4 5 6 7

• Repeat

• Typically manually

12 Instance Replacements

• Don’t break anything!

• Bob just got fired =(

JBoss App

Memcache

PostgreSQL Slaves

PostgreSQL Master

NagiosGraphite 1 2

4 5 6 7

Done in Maintenance Windows

JBoss App

Memcache

PostgreSQL Slaves

PostgreSQL Master

NagiosGraphite

• Invalid configs!

Different IP Addresses?

Configuration Desperation

http://www.flickr.com/photos/francoforeshock/5716969942/Wednesday, October 2, 13

Chef Solves This Problem• But you already

guessed that, didn’t you?

Chef is Infrastructure as Code• Programmatically

provision and configure

• Treat like any other code base

• Reconstruct business from code repository, data backup, and bare metal resources.http://www.flickr.com/photos/louisb/4555295187/

Programs• Chef generates

configurations directly on nodes from their run list

• Reduce management complexity through abstraction

• Store the configuration of your programs in version control

http://www.flickr.com/photos/ssoosay/5126146763/

Declarative Interface to Resources• Define Policy• Say what, not how• Pull not Push

http://www.flickr.com/photos/bixentro/2591838509/

That Looks Like Thispackage "apache2"

template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode "0644" variables(:allow_override => "All") notifies :reload, "service[apache2]"end

service "apache2" do action [:enable,:start] supports :reload => trueend

Ohai"languages": { "ruby": {

}, "perl": { "version": "5.14.2", "archname": "x86_64-linux-gnu-thread-multi" }, "python": { "version": "2.7.3", "builddate": "Aug 1 2012, 05:14:39" }, "php": { "version": "5.3.10-1ubuntu3.6", "builddate": "(cli) (built: Mar" }},

"network": { "interfaces": { "lo": { "mtu": "16436", "flags": [ "LOOPBACK", “UP","LOWER_UP" ], "encapsulation": "Loopback", "addresses": { "127.0.0.1": { "family": "inet", "netmask": "255.0.0.0", "scope": "Node" }, "::1": { "family": "inet6", "scope": "Node" } }, }, "eth0": { "type": "eth", "number": "0",

"kernel": { "name": "Linux", "release": "3.2.0-32-virtual", "version": "#51-Ubuntu SMP Wed Sep 26 21:53:42 UTC 2012", "machine": "x86_64", "modules": { "isofs": { "size": "40257", "refcount": "0" }, "acpiphp": { "size": "24231", "refcount": "0" } }, "os": "GNU/Linux"},"os": "linux","os_version": "3.2.0-32-virtual","ohai_time": 1369328621.3456137,

execute "load sysctl" do command "/sbin/sysctl -p" action :nothingend

bytes = node['memory']['total'].split("kB")[0].to_i * 1024 / 3,pages = node['memory']['total'].split("kB")[0].to_i * 1024 / 3 / 2048

# adjust shared memory and semaphorestemplate "/etc/sysctl.conf" do source "sysctl.conf.erb" variables( :shmmax_in_bytes => bytes, :shmall_in_pages => pages ) notifies :run, "execute[load sysctl]", :immediatelyend

Decide what to declare

Recipes and Cookbooks• Recipes are collections of

Resources• Cookbooks contain recipes,

templates, files, custom resources, etc

• Code re-use and modularity

http://www.flickr.com/photos/shutterhacks/4474421855/

Run ListsServerServerServerServer

chef-server

API chef-client “recipe[ntp::client]”

client.rb

chef-server

API chef-client “recipe[ntp::client]”, “recipe[openssh::server]”

client.rb

openssh

server.rb

chef-server

API chef-client “recipe[ntp::client]”, “recipe[openssh::server]”,

“recipe[apache]”,“recipe[php]”

client.rb

openssh

server.rb

apache

default.rb

default.rbWednesday, October 2, 13

name "base"description "base"run_list [ "recipe[selinux::disabled]", "recipe[etchosts]", "recipe[yum::epel]", "recipe[debugtools]"]

name "webserver"description "webserver server"run_list [ "role[base]", "recipe[nginx::server]"]

RecipeRecipeRecipe

RecipeRecipeRecipeRole

Recipe

ServerServerServerServer

chef-server

API chef-client “recipe[ntp::client]”, “recipe[openssh::server]”,

“recipe[apache]”,“recipe[php]”

client.rb

openssh

server.rb

apache

default.rb

RolesServerServerServerServer

chef-server

API chef-client

“role[webserver]”

client.rb

openssh

server.rb

apache

default.rb

RolesServerServerServerServer

chef-server

chef-client

“role[webserver]”

client.rb

openssh

server.rb

apache

default.rb

chef-client

“role[database]”node

client.rb

openssh

server.rb

http://www.flickr.com/photos/kathycsus/2686772625

• IP addresses

• Hostnames

• FQDNs

• Search for nodes with Roles

• Find configuration data

Search: Dynamic Infrastructure

Search for Nodespool_members = search("node","role:webserver")

template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]"end

Pass results into Templates# Set up application listeners here.

listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%><% if node["haproxy"]["enable_admin"] -%>listen admin 0.0.0.0:22002 mode http stats uri /<% end -%>

JBoss App

Memcache

PostgreSQL Slaves

PostgreSQL Master

NagiosGraphite

So when this...

JBoss App

Memcache

PostgreSQL Slaves

PostgreSQL Master

NagiosGraphite

... becomes this ...

JBoss App

Memcache

PostgreSQL Slaves

PostgreSQL Master

NagiosGraphite

...this can happen automatically

NagiosGraphite

JBoss App

Memcache

PostgreSQL Slaves

To Add a New Server…

• 2x Web Server Configurations

• 2 Web Server Restarts

• 4x Database Configurations

• 8x Firewall Configurations

• DNS Service

• Network Configuration

• Deployer

• 8x Monitoring Changes

20+ Changes

12+ New Infrastructure Dependencies

4+ Hours

Count the Resources

NagiosGraphite

JBoss App

Memcache

PostgreSQL Slaves

To Add a New Server…

• 2x Web Server Configurations

• 2 Web Server Restarts

• 4x Database Configurations

• 8x Firewall Configurations

• DNS Service

• Network Configuration

• Deployer

• 8x Monitoring Changes

20+ Changes

12+ New Infrastructure Dependencies

4+ Hours

Count the Resources

Add 1 server20+ Changes

Managing Complexity Later

Managing Complexity LaterWe added:

• Load Balancers

• MemCache

• Search Appliances

• Lots of VM’s

• More Scale

Exponential Increase In:

• Configuration Changes

• Infrastructure Dependencies

• Skills Needed

• Greater Risk

And at Greater Scale...

And at Greater Scale...How Do we Manage This at Cloud Scale?

• Thousands of infrastructure dependencies and configurations needed for each change.

• Huge Amounts of Time

• Increased Cost of Correction of Manual Errors

• Huge Need for Talent

• Risk of Critical Skills Shortage

Automation is a JourneyFull

Automation

Common Automation

Tasks:Scripts, OS Compliance, Updates, etc.

Configuration ManagementDiscovery and

Visibility

Application Management

Continuous Deployment

The Path to the Coded BusinessWednesday, October 2, 13

Flavors of Chef Server

Landscape of Chef-Managed Infrastructure

• SaaS, hosted by Opscode• Manage up to 50,000 servers• Industry-leading SLAs• 24x7x365 Support Options• Get up and running quickly• Pay/grow as you need

Hosted Enterprise Chef

• All the power of Hosted Enterprise, behind the firewall

• Delivered as enterprise software• Implementation consulting customized to

customer’s needs

Enterprise Chef

• Core components of Chef Server developed as open source

• No enterprise features (LDAP, HA, etc.) but many people use it successfully

• Community support

Open Source Chef Server

Learn Chef!

Let’s Learn Chef!

• https://learnchef.opscode.com/• Next meetup: Hands on. Bring laptops!• Feel free to get started early

Thanks• Julian Dunn• jdunn@opscode.com• @julian_dunn• www.juliandunn.net

an introduction to devops with chef

Technology

chef. кто на кухне хозяин....

achieving devops success with chef automate

when docker ends, chef begins ~ #idi2015 incontro devops...

devops and the cloud: chef and microsoft azure

from chef to saltstack on cloud providers - incontro devops...

chef devops and the cloud white paper [30 apr] download-3-2

devops for drupal: why we cook with chef

devops with chef

devops: next - goto conference · nicole forsgren, phd...

devops meets paas - ny meetup with chef (opscode)

devops introduction @cegeka

devops explained - niceideas.ch explained jerome kehrli /...

devops hackathon session 2: basics of chef

devops & chef - files.meetup.com and chef - choose your...

effective devops by using docker and chef together !

devops and the cloud: chef and amazon web services · pdf...

devops for etl processing at scale with mongodb, solr, aws...

devops 101 - an introduction to devops

making infrastructure s'mores with chef (kc devops edition)

introduction to devops