puppet and aws: getting the best of both worlds
DESCRIPTION
This talk will be a modified and updated version of the talk given at Puppet Camp Amsterdam. I will discuss some technical examples of how to use Puppet to manage large infrastructures in Amazon's cloud, as well as giving some background in to how Puppet fits in to the AWS ecosystem. Further topics include: - OpsWorks (Amazon's Chef-based configuration management offering) and what it means for Puppet - using Puppet in conjunction with CloudFormation - using Puppet to automate common AWS tasks, such as building AMIs - using Vagrant and Puppet to create an easy path from local development to production - common pitfalls and workarounds Mike Ryan Cloud Infrastructure Consultant, Epitech BV Mike Ryan is the founder of Epitech BV, a cloud technology consultancy based in Amsterdam, and author of the upcoming O'Reilly book AWS System Administration. He is also the founder of Cloudfrag, an internet gaming startup, and is focussed on using efficient engineering practices to help companies accelerate their growth.TRANSCRIPT
WELCOME
Puppet and AWSGetting the Best of Both Worlds
Mike Ryan - Epitech BV23
August2013
CONTACT:[email protected]
Saturday, August 24, 13
EPITECH BV
Hello, I’m Mike2
CONTACT:[email protected]
Sysadmin with a passion for automa2on
Londoner in Amsterdam
Epitech.nl -‐ sysadmin as a service
Saturday, August 24, 13
EPITECH BV
AWS - A very brief introduction
3
CONTACT:[email protected]
EC2 -‐ Elas2c Compute Cloud
AMI -‐ Amazon Machine Images
User Data
Saturday, August 24, 13
EPITECH BV
CloudFormation5
CONTACT:[email protected]
{ "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "EC2 instance", "Resources" : { "MyEC2Instance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "ImageId" : "ami-79fd7eee", "KeyName" : "my-ssh-key", } } }}
Saturday, August 24, 13
EPITECH BV
6
CONTACT:[email protected]
"Enabled" : "true", "Logging" : { "Bucket" : "webapplication.s3.amazonaws.com", "Prefix" : "webapp-logging/" } } } },
"asgMyAutoScalingGroup": { "Type": "AWS::AutoScaling::AutoScalingGroup", "Properties": { "AvailabilityZones": [ "us-east-1b", "us-east-1c" ], "Cooldown": "300", "DesiredCapacity": "1", "MaxSize": "1", "MinSize": "1", "LaunchConfigurationName": { "Ref": "lcMyLC" }, "LoadBalancerNames": [ { "Ref": "elbMyLB" } ] } }, "s3webapplication": { "Type": "AWS::S3::Bucket" }, "sgwebappsecuritygroup": { "Type": "AWS::EC2::SecurityGroup", "Properties": { "GroupDescription": "for web app", "SecurityGroupIngress": [
Saturday, August 24, 13
EPITECH BV
7
CONTACT:[email protected]
Bootstrapping an Infrastructure
www.infrastructures.org
Saturday, August 24, 13
EPITECH BV
9
CONTACT:[email protected]
Files, Services and Packages with cfn-init
"Metadata": { "AWS::CloudFormation::Init": { "config": { "sources" : { "/etc/puppet" : "http://example.com/puppet.tar.gz" }, "packages": { "yum": { "puppet-server": [], }, }, "services": { "sysvinit": { "puppetmaster": { "ensureRunning": "true", "enabled": "true"
Saturday, August 24, 13
EPITECH BV
10
CONTACT:[email protected]
Run a script at launch time with User Data
{ "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "EC2 instance", "Resources" : { "PuppetMasterInstance": { "Type": "AWS::EC2::Instance", "Metadata": { }, "Properties": { "UserData": { "Fn::Base64": { "Fn::Join": [ "", [ "#!/bin/bash\n", "/opt/aws/bin/cfn-init --region ", "\n", "/usr/bin/puppet apply site.pp, "\n" ]...
Saturday, August 24, 13
EPITECH BV
11
CONTACT:[email protected]
cfn-create-stack
cfn-create-stack puppetmaster \--template-file=puppetmaster.json
http://aws.amazon.com/developertools/2555753788650372
http://aws.amazon.com/cli/
Saturday, August 24, 13
EPITECH BV
13
CONTACT:[email protected]
The Hostname Issuemike@ip-10-32-34-116:~$ hostname -fip-10-32-34-116.eu-west-1.compute.internal
Saturday, August 24, 13
EPITECH BV
15
CONTACT:[email protected]
User Data - CloudFormation},"UserData": { "Fn::Base64": { "Fn::Join": [ "", [ “{\”role\”: \”web\”, ”, “\”env\”: \”staging\”} ” ] ] }}
Saturday, August 24, 13
EPITECH BV
16
CONTACT:[email protected]
include stdlibnode default { $userdata = parsejson($ec2_userdata) $role = userdata[‘role’] $environment = userdata[‘env’]
case $role { ‘web’: { include nginx } ‘db’: { include postgresql } }}
Saturday, August 24, 13
EPITECH BV
19
CONTACT:[email protected]
Vagrant::Config.run do |config| config.vm.provision :puppet do |puppet| puppet.manifests_path = "../puppet/manifests" puppet.module_path = "../puppet/modules" puppet.manifest_file = "site.pp" puppet.options = "--verbose --debug" puppet.facter = { :ec2_userdata => { :role => "database", :env => "vagrant", }.to_json, :vagrant => "true" } endend
Saturday, August 24, 13
EPITECH BV
21
CONTACT:[email protected]
Broken Puppet Master =Broken Auto Scaling
No
Saturday, August 24, 13
EPITECH BV
22
CONTACT:[email protected]
#!/bin/bash
/usr/local/bin/update.sh/usr/bin/puppet apply site.pp
/etc/rc.local
Saturday, August 24, 13
EPITECH BV
23
CONTACT:[email protected]
But you lose...
Puppet Dashboard
Exported Resources
Stored configs
Saturday, August 24, 13
EPITECH BV
24
CONTACT:[email protected]
env.roledefs = { 'web': ['www1', 'www2', 'www3'], 'db': ['db1', 'db1']}
@roles('db', ‘web’)def run_puppet(): sudo('puppet apply site.pp')
$fab run_puppet #run puppet everywhere$fab run_puppet --roles db # role-specific
Triggering Puppet with Fabric
Saturday, August 24, 13
EPITECH BV
25
CONTACT:[email protected]
def configure_roles():
tags = EC2TagManager(AWS_KEY, AWS_SECRET, regions=['eu-west-1'])
roles = {}
for role in [‘db’, ‘web’]: roles[role] = tags.get_instances(role=role) return roles
Triggering Puppet with Fabric
https://github.com/mikery/fabric-ec2
Saturday, August 24, 13
