aws presents: infrastructure as code on aws - chefconf 2015
TRANSCRIPT
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Scott McDonald, Sr. Consultant
4/2/2015
Infrastructure as Code on AWSExample of Using CloudFormation to Deploy Chef Server 12 and Automatically Bootstrap Clients in AWS
Infrastructure as Code is….
• A technical domain revolving around building and managing infrastructure programmatically.
• A way to enable the reconstruction of the business from nothing but a source code repository, an application data backup, and bare metal resources.
• Your primary constraint should be the amount of time it takes to restore your application data.
AWS CloudFormation: Infrastructure as Code
AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion
First released in 2010
Amazon CloudFormation
• Infrastructure as Code
• Integrates with version control
• JSON formatted documents
• Templates for repeatable infrastructure
• Stacks of resources
• Supports AWS resource typesAWS CloudFormation
AWS CloudFormation: Infrastructure as Code
Document, version control, and share your applications and infrastructure as a JSON document
Provision app and other AWS resources (VPC, DynamoDB, RDS< EC2, Security Groups,) from a template
Repeatable, reliable deployments for test/dev/prod in any AWS Region
Resource Property Types
• AutoScaling• CloudFront• CloudWatch• DynamoDB• EC2• Elastic Beanstalk
• Elastic Load Balancer• IAM• RDS• S3• SNS/SQS
AWS CloudFormation: Application stack example (continue)
Template File Defining Stack
GitSubversionMercurial
Dev
Test
Prod
The entire application can be represented in an AWS CloudFormation template.
Use the version control system of your choice to store and track changes to this template
Build out multiple environments, such as for Development, Test, and Production using the template
{ "AWSTemplateFormatVersion": "2010-09-09",
"Description": "Example CloudFormation to install Chef 12 Server using RHEL 6.5 ami in us-east-1. This template creates and starts a Chef 12 Server with the Web Management module (for up to 10 hosts), initializes knife in ec2-user account, and then uploads the aws cookbook to the running Chef 12 Server. Roles are used to create a private s3 bucket and upload a client validation key. A WaitCondition is used to pause the stack creation until the server is completely deployed. **WARNING** This template creates one or more Amazon EC2 instances. You will be billed for the AWS resources used if you create a stack from this template.",
"Parameters": { "KeyName": { "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the Chef Server", "Type": "String", "MinLength": "1", "MaxLength": "255", "AllowedPattern" : "[\\x20-\\x7E]*", "ConstraintDescription" : "can contain only ASCII characters." }, "InstanceType" : { "Description" : "Chef 12 Server EC2 instance type", "Type" : "String", "Default" : "m3.large", "AllowedValues" : [ "t2.micro","t2.medium","m3.medium","m3.large","m3.xlarge","m3.2xlarge"], "ConstraintDescription" : "must be a valid EC2 instance type." },
CloudFormation Chef Server 12 Example 1/7
"ChefServerRole" : { "Description" : "Pre-create a Role - it needs at least S3 put/get", "Type" : "String" },
"SourceLocation" : { "Description" : "Source IP address range allowed SSH/Web to the Chef Server", "Type": "String", "MinLength": "9", "MaxLength": "18", "Default": "0.0.0.0/0", "AllowedPattern": "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})", "ConstraintDescription": "must be a valid IP CIDR range of the form x.x.x.x/x." } },
"Mappings" : { "AWSRegion2AMI" : { "us-east-1" : { "id" : "ami-00a11e68" } } },
CloudFormation Chef Server 12 Example 2/7
"Resources" : {
"ChefServer": { "Type": "AWS::EC2::Instance", "Metadata" : { "AWS::CloudFormation::Init" : { "config" : { "files" : { "/root/.aws/config" : { "content" : { "Fn::Join" : ["", [ "[default]\n", "region = us-east-1\n" ]]}, "mode" : "000600", "owner" : "root", "group" : "root" } } } } },
CloudFormation Chef Server 12 Example 3/7
"Properties": { "SecurityGroups": [ { "Ref": "ChefServerSecurityGroup" } ], "IamInstanceProfile" : { "Ref" : "ChefServerRole" }, "ImageId": { "Fn::FindInMap": [ "AWSRegion2AMI", { "Ref": "AWS::Region" }, "id" ] }, "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [ "#!/bin/bash\n",
"export PATH=$PATH:/usr/local/bin:/opt/aws/bin\n",
"function error_exit\n", "{\n", " cfn-signal -e 1 -r \"$1\" '", { "Ref" : "ChefServerWaitHandle" }, "'\n", " exit 1\n", "}\n",
"curl https://bootstrap.pypa.io/ez_setup.py -o - | python\n", "easy_install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz\n", "cfn-init --region ", { "Ref" : "AWS::Region" }, " -s ", { "Ref" : "AWS::StackId" }, " -r ChefServer ", "|| error_exit 'Failed to run cfn-init'\n",
"# Bootstrap chef\n", "cd /home/ec2-user \n", "wget https://s3.amazonaws.com/awshat-chefcon2015/install-chef-aws.sh >> /tmp/install-chef-aws.log 2>&1 \n", "chmod +x /home/ec2-user/install-chef-aws.sh \n", "/home/ec2-user/install-chef-aws.sh >> /tmp/install-chef-aws.log 2>&1 \n",
"# Setup awscli on redhat\n", "curl https://bootstrap.pypa.io/get-pip.py -o - | python\n", "pip install awscli\n”,
CloudFormation Chef Server 12 Example 4/7
"# use awscli to copy validation key to S3 bucket\n", "/usr/bin/aws s3 cp /home/ec2-user/.chef/chef-validator.pem s3://", {"Ref" : "ChefKeyBucket" } ,"/chef-validator.pem\n",
"# If all went well, signal success\n", "cfn-signal -e $? -r 'Chef Server configuration' '", { "Ref" : "ChefServerWaitHandle" }, "'\n" ]]}}, "KeyName": { "Ref": "KeyName" }, "InstanceType": { "Ref": "InstanceType" } } },
"ChefServerSecurityGroup" : { "Type" : "AWS::EC2::SecurityGroup", "Properties" : { "GroupDescription" : "Open up SSH/Web access to Chef Server from allowed Group and Source IP range", "SecurityGroupIngress" : [ { "IpProtocol": "tcp", "FromPort": "22", "ToPort": "22", "CidrIp": { "Ref" : "SourceLocation"} }, { "IpProtocol": "tcp", "FromPort": "443", "ToPort": "443", "SourceSecurityGroupName": { "Ref" :"ChefClientSecurityGroup" }}, { "IpProtocol": "tcp", "FromPort": "443", "ToPort": "443", "CidrIp": { "Ref" : "SourceLocation"} } ] } },
CloudFormation Chef Server 12 Example 5/7
"ChefClientSecurityGroup" : { "Type" : "AWS::EC2::SecurityGroup", "Properties" : { "GroupDescription" : "Group with client access to Chef Server", "SecurityGroupIngress" : [ { "IpProtocol": "tcp", "FromPort": "22", "ToPort": "22", "CidrIp": { "Ref" : "SourceLocation"} }, { "IpProtocol": "tcp", "FromPort": "80", "ToPort": "80", "CidrIp": { "Ref" : "SourceLocation"} } ] } },
"ChefKeyBucket" : { "Type" : "AWS::S3::Bucket", "Properties" : { "AccessControl" : "Private" }, "DeletionPolicy" : "Delete" },
"ChefServerWaitHandle" : { "Type" : "AWS::CloudFormation::WaitConditionHandle" },
CloudFormation Chef Server 12 Example 6/7
"ChefServerWaitCondition" : { "Type" : "AWS::CloudFormation::WaitCondition", "DependsOn" : "ChefServer", "Properties" : { "Handle" : { "Ref" : "ChefServerWaitHandle" }, "Timeout" : "7200" } } }, "Outputs" : { "ServerURL" : { "Description" : "URL of newly created Chef 12 server - login and change password", "Value" : { "Fn::Join" : ["", ["https://", {"Fn::GetAtt" : [ "ChefServer", "PublicDnsName" ]}, ":443/organizations/chef"]]} }, "ChefKeyBucket" : { "Description" : "Private S3 bucket with validation key for client bootstrap automation:", "Value" : {"Ref" : "ChefKeyBucket" } }, "ChefSecurityGroup" : { "Description" : "EC2 Security Group for access to Chef 12 Server", "Value" : { "Ref" :"ChefClientSecurityGroup" } } } }
CloudFormation Chef Server 12 Example 7/7
Using CloudFormation from the CLI
685b358e3054:ChefCon2015 smcdon$ pwd/Users/smcdon/Documents/ChefCon2015
685b358e3054:ChefCon2015 smcdon$ cat add.server
aws cloudformation create-stack --stack-name chef12-server5 --template-body file://./server.cfn --disable-rollback --parameters \ParameterKey=InstanceType,ParameterValue=m3.2xlarge \ParameterKey=KeyName,ParameterValue=awshat_key01 \ParameterKey=ChefServerRole,ParameterValue=devops \
Deploying Chef Server 12 via CloudFormation
685b358e3054:ChefCon2015 smcdon$ ./add.server
{ "StackId": "arn:aws:cloudformation:us-east-1:162012790422:stack/chef12-server5/f30bbc20-d48c-11e4-a271-50443313686e"}
685b358e3054:ChefCon2015 smcdon$
Infrastructure as Code under Version Control!
685b358e3054:ChefCon2015 smcdon$ git commit -m "ChefCon2015 presentation commit example"[master ccf204b] ChefCon2015 presentation commit example 1 file changed, 1 insertion(+), 1 deletion(-)
685b358e3054:ChefCon2015 smcdon$ git push origin masterCounting objects: 5, done.Delta compression using up to 4 threads.Compressing objects: 100% (3/3), done.Writing objects: 100% (3/3), 316 bytes | 0 bytes/s, done.Total 3 (delta 2), reused 0 (delta 0)To https://github.com/awshat/chefcon2015.git 190668d..ccf204b master -> master
685b358e3054:ChefCon2015 smcdon$
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Sample template to bring up a redhat linux ec2 instance and bootstrap a client node to be managed by an existing Chef Server. **WARNING** This template creates an EC2 instance. You will be billed for the AWS resources used if you create a stack from this template.",
"Parameters": { "KeyName": { "Type": "String", "Description" : "EC2 KeyPair to enable SSH access to the client instance" }, "InstanceType": { "Default": "m3.medium", "Description" : "Type of EC2 instance for the client node", "Type": "String", "AllowedValues" : [ "t2.micro", "t2.medium", "m3.small", "m3.medium", "m3.large", "m3.xlarge", "m3.2xlarge"], "ConstraintDescription" : "must contain only alphanumeric characters." },
CloudFormation for Chef Clients in AWS 1/3
"ServerURL" : { "Description" : "Chef 12 Server URL", "Type": "String" }, "ChefSecurityGroup" : { "Description" : "Security group for clients to get access to Chef Server", "Type": "String" }, "S3Role" : { "Description" : "IAM S3 Role with Get access for chef client bootstrapping automation", "Type" : "String" }, "ChefKeyBucket" : { "Description" : "S3 bucket with validation key", "Type": "String" }, "ChefClientEnv" : { "Description" : "Environment setting for deployed instances", "Type": "String", "Default" : "_default" }},
CloudFormation for Chef Clients in AWS 2/3
"Mappings" : { "AWSRegion2AMI" : { "us-east-1" : { "id" : "ami-00a11e68" } }},"Resources" : {"ChefClient": { "Type": "AWS::EC2::Instance", "Properties": { "SecurityGroups": [ { "Ref": "ChefSecurityGroup" } ], "IamInstanceProfile" : { "Ref" : "S3Role" }, "ImageId": { "Fn::FindInMap": [ "AWSRegion2AMI", { "Ref": "AWS::Region" }, "id" ] }, "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [ "#!/bin/bash\n",
"# Bootstrap chef client\n", "cd /home/ec2-user \n", "wget https://s3.amazonaws.com/awshat-chefcon2015/install-chef-client.sh >> /tmp/install-chef-amzn.log 2>&1 \n", "chmod +x /home/ec2-user/install-chef-client.sh \n", "/home/ec2-user/install-chef-client.sh ", {"Ref" : "ChefKeyBucket" } ," ", {"Ref" : "ServerURL" } ," ", {"Ref" : "ChefClientEnv" } ," >> /tmp/install-chef-amzn.log 2>&1 \n" ]]}}, "KeyName": { "Ref": "KeyName" }, "InstanceType": { "Ref": "InstanceType" } } } }}
CloudFormation for Chef Clients in AWS 3/3
Create a Version Controlled Client Script
685b358e3054:ChefCon2015 smcdon$ cp add.client4 add.client5685b358e3054:ChefCon2015 smcdon$ vi add.client5 685b358e3054:ChefCon2015 smcdon$ cat add.client5aws cloudformation create-stack --stack-name $1 --template-body file://./client.cfn --disable-rollback --parameters \ParameterKey=KeyName,ParameterValue=awshat_key01 \ParameterKey=InstanceType,ParameterValue=m3.large \
ParameterKey=ServerURL,ParameterValue=https://ec2-52-1-228-71.compute-1.amazonaws.com:443/organizations/chef \ParameterKey=ChefKeyBucket,ParameterValue=chef12-server5-chefkeybucket-zut8xoz6apsn \ParameterKey=ChefSecurityGroup,ParameterValue=chef12-server5-ChefClientSecurityGroup-NIR4XTVU1POF \ParameterKey=S3Role,ParameterValue=s3access
685b358e3054:ChefCon2015 smcdon$ git add -A685b358e3054:ChefCon2015 smcdon$ git commit -m "ChefCon2015 example client script for presentation"[master f655195] ChefCon2015 example client script for presentation 1 file changed, 8 insertions(+) create mode 100755 add.client5685b358e3054:ChefCon2015 smcdon$ git push origin masterCounting objects: 4, done.Delta compression using up to 4 threads.Compressing objects: 100% (3/3), done.Writing objects: 100% (3/3), 599 bytes | 0 bytes/s, done.Total 3 (delta 1), reused 0 (delta 0)To https://github.com/awshat/chefcon2015.git ccf204b..f655195 master -> master
Use CloudFormation to Bootstrap Clients
685b358e3054:ChefCon2015 smcdon$ ./add.client5 server5-client1{ "StackId": "arn:aws:cloudformation:us-east-1:162012790422:stack/server5-client1/73936ff0-d497-11e4-8f0c-50e2416294a8"}685b358e3054:ChefCon2015 smcdon$ ./add.client5 server5-client2{ "StackId": "arn:aws:cloudformation:us-east-1:162012790422:stack/server5-client2/75a5fc40-d497-11e4-bf18-50e24162947c"}685b358e3054:ChefCon2015 smcdon$ ./add.client5 server5-client3{ "StackId": "arn:aws:cloudformation:us-east-1:162012790422:stack/server5-client3/77368750-d497-11e4-adea-50018ffe9e62"}
CloudFormation = Easy Teardown
• Fully automated
• Delete resources
• Complete teardown
• Entire stacks are disposable
• For this Chef 12 Example – Deleting clients in
CloudFormation automatically deregisters nodes
from Chef Server managementAWS CloudFormation