aws power tools: advanced aws cloudformation and cli
TRANSCRIPT
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Power Tools: Advanced AWS CloudFormation and CLI
Alex CorleySolutions Architect
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
“How do I plan my stacks?”
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Organize by layers & environmentsLayers of stacks
Environments
Dev1Dev2
IntegProd
Frontend Services
• Consumer website, seller website, mobile backend
Backend Services
• Search, payments, reviews, recommendations
Shared Services
• CRM DBs, common monitoring /alarms, subnets, security groups
Base Network
• VPCs, Internet gateways, VPNs, NATs
Identity • IAM users, groups, roles
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Apply service-oriented architecture
Food catalog website Ordering website
Customer DB service
Inventory service
Recommendations service
Analytics serviceFulfillment
service
Paymentservice
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Apply service-oriented architecture
Food catalog website Customer DB service
“Outputs” : {
“CustDBEndPoint”:””
}
“Parameters” : {
“CustDBEndPoint”:””
}
Depends on
“Wire”
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Nested stacks
Website1
“Resources” : { “ELB”, “AutoScaling”,
“RDS”}
Website2
“Resources” : { “ELB”, “AutoScaling”, “DynamoDB”}
ELB_AND_AS
“Resources” : { “ELB”, “AutoScaling”}
Website1
“Resources” : { “NestedStack”, “RDS”}
Website2
“Resources” : { “NestedStack”, “DynamoDB”}
for reusability & specialization
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
“How do I ensure error-free stack creation?”
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Validate your templates
ValidateTemplate API action validates:• JSON syntax• Absence of circular dependencies• Template structure
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
*NEW* Use parameter types"Parameters" : { “aVpcId" : { "Type" : "AWS::EC2::VPC::Id" }, “bSubnetIds" : { "Type" : "List<AWS::EC2::Subnet::Id>" }, “cSecurityGroups" : { "Type" : "List<AWS::EC2::SecurityGroup::Id>" }, “dKeyPair" : { "Type" : "AWS::EC2::KeyPair::KeyName" }
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Use parameter constraints "Parameters" : { "SourceCIDRForSSH" : { "Description" : "CIDR block to allow SSH from", "Type" : "String", "MinLength" : "9", "MaxLength" : "18", "AllowedPattern" : "^([0-9]+\\.){3}[0-9]+\\/[0-9]+$"}
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
“How do I deploy and bootstrap software and data?”
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
"MyInstance": { "Type": "AWS::EC2::Instance", "Metadata" : { "AWS::CloudFormation::Init" : { "webapp-config": { "packages" : {}, "sources" : {}, "files" : {}, "groups" : {}, "users" : {}, "commands" : {}, "services" : {}
Use AWS::CloudFormation::Init Declarative Reusable Grouping & ordering Debug-able Updatable Highly secure BIOT™ (Bring in other tools)
ow.ly/DiNCm
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Use AWS::CloudFormation::Init
"YourInstance": { "Metadata": { "AWS::CloudFormation::Authentication": { "S3AccessCreds": { "type": "S3", "roleName": { "Ref" : "InstanceRole"}, "buckets" : ["your-bucket"] } }, "AWS::CloudFormation::Init": {}
Supports role-based auth
Securely download
Choose auth type. IAM Role is recommended
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Use AWS::CloudFormation::Init
"AWS::CloudFormation::Init": {"webapp-config": {
"packages" : {}, "sources" : {}, "files" : {}, "groups" : {}, "users" : {}, "commands" : {}, "services" : {}
Declarative
"sources" : {"/etc/myapp" :
"https://s3.amazonaws.com/mybucket/myapp.tar.gz"
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Use AWS::CloudFormation::InitSupports updates
"packages" : {}, "sources" : {}, "files" : {}, "groups" : {}, "users" : {},"commands" : {}, "services" : {}
1. Update instance metadata in the template
2. UpdateStack
Instance
Metadata
cfn-hup
3. AWS CloudFormation daemon updates configuration
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Use AWS::CloudFormation::Init
"install_chef" : {},
"install_wordpress" : { "commands" : { "01_get_cookbook" : {}, ..., "05_configure_node_run_list" : { "command" : "knife node run_list add -z `knife node list -z` recipe[wordpress]", "cwd" : "/var/chef/chef-repo", "env" : { "HOME" : "/var/chef" }
Flexibility to bring in other tools such as Chef
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Use AWS::CloudFormation::Init"UserData": {
"# Get the latest CloudFormation helper scripts package\n","yum update -y aws-cfn-bootstrap\n",
"# Trigger CloudFormation::Init configuration \n","/opt/aws/bin/cfn-init --stack ", {"Ref": "AWS::StackId"}, " --resource WebServerInstance ", " --region ", {"Ref": "AWS::Region"}, "\n",
"# Signal completion\n","/opt/aws/bin/cfn-signal –e $? --stack ", {"Ref": "AWS::StackId"}, " --resource WebServerInstance ", " --region ", {"Ref": "AWS::Region"}, "\n"
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Use CloudWatch Logs for debugging"install_logs": { "packages" : { ... "awslogs" ... }, "services" : { ... "awslogs" ... } "files": { "/tmp/cwlogs/cfn-logs.conf": {}
file = /var/log/cfn-init.loglog_stream_name = {instance_id}/cfn-init.log
file = /var/log/cfn-hup.loglog_stream_name = {instance_id}/cfn-hup.log
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Use CloudWatch Logs for debugging
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
CloudFormation Demo
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
CloudFormation Template: Web Application
Availability Zone - A
DMZ – 10.0.0.0/24
Web – 10.0.3.0/24
Database – 10.0.6.0/24
Availability Zone - B
DMZ – 10.0.1.0/24
Web – 10.0.4.0/24
Database – 10.0.7.0/24
Availability Zone - C
DMZ – 10.0.2.0/24
Web – 10.0.5.0/24
Database – 10.0.8.0/24
AWS CloudFormatio
n
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
CloudFormation Template – Production, Test, Dev
Production Test Development
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
CloudFormation Template – Multiple Regions
us-east-1 us-west-1 us-west-2
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
CloudFormation Template – Second Environment
Availability Zone - A
DMZ – 10.1.0.0/24
Web – 10.1.3.0/24
Database – 10.1.6.0/24
Availability Zone - B
DMZ – 10.1.1.0/24
Web – 10.1.4.0/24
Database – 10.1.7.0/24
Availability Zone - C
DMZ – 10.1.2.0/24
Web – 10.1.5.0/24
Database – 10.1.8.0/24
AWS CloudFormatio
n
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
CloudFormation Parameters
"Parameters": {"Environment": { "Default": "Development", "Type": "String",
"AllowedValues": [ "Production", "Staging", "Development" ], "Description": "Name of Environment", "ConstraintDescription": "Production or Staging or Development"
}
}
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
CloudFormation Mappings
"Mappings": {
"AWSRegionAvailabilityZones": {
"us-east-1": { "1": "us-east-1a", "2": "us-east-1b", "3": "us-east-1d" },
"us-west-1": { "1": "us-west-1a", "2": "us-west-1b", "3": "us-west-1c" },
"us-west-2": { "1": "us-west-2a", "2": "us-west-2b", "3": "us-west-2c" }
}
}
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
CloudFormation Mappings"Environment2Vpc": {
"Production": {
"VpcCidr": "10.0.0.0/16",
"PublicSubnet1Cidr": "10.0.0.0/24",
"PublicSubnet2Cidr": "10.0.1.0/24",
"PublicSubnet3Cidr": "10.0.2.0/24",
"WebSubnet1Cidr": "10.0.3.0/24",
"WebSubnet2Cidr": "10.0.4.0/24",
"WebSubnet3Cidr": "10.0.5.0/24",
"PrivateSubnet1Cidr": "10.0.6.0/24",
"PrivateSubnet2Cidr": "10.0.7.0/24",
"PrivateSubnet3Cidr": "10.0.8.0/24"
}
}
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
CloudFormation Subnets"PublicSubnet1": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"AvailabilityZone": { "Fn::FindInMap": [ "AWSRegionAvailabilityZones", { "Ref": "AWS::Region" }, "1" ] },
"VpcId": { "Ref": "VPC" },
"CidrBlock": { "Fn::FindInMap": [ "Environment2Vpc", { "Ref": "Environment" }, "PublicSubnet1Cidr" ] },
"Tags": [
{ "Key": "Name", "Value": { "Fn::Join": [ "-", [ { "Ref": "Environment" }, "DMZ" ] ] } },
{ "Key": "Environment", "Value": { "Ref": "Environment" } },
{ "Key": "Application", "Value": { "Ref": "AWS::StackName" } },
{ "Key": "Network", "Value": "DMZ" }
]
}
}
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
When Should I Script?• Whenever possible:
– If it can be scripted, script it!– AWS services have API reference guides publicly available.– Scripts and AWS CloudFormation templates can be used to quickly
launch, start, and stop dev and test environments.
AWS CLI
AWS CloudFormation
template
stack
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Key Features of the AWS CLI• It’s a unified tool to manage your AWS services• It can be installed via MSI, a cross-platform bundle, or pip• It’s a Python package• Run aws configure to set access
keys and default region/output format• All commands in the AWS CLI
have a structure similar to:– $ aws <service> <operation>– ex: $ aws ec2 describe-instances
Details for commands can be found online and using the help subcommand.
AWS CLI
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
AWS CLI Configuration Values• The aws configure <subcommand>
operation gives you programmatic access to configuration variables– list provides you with all of the current configuration data.– get returns the value of a single configuration variable.– set sets the value of a single configuration variable.
• Example: aws configure get region prints the currently set region to standard output
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
EC2 Instance Profiles
• Assign an IAM role to an EC2 instance on launch
• EC2 securely delivers credentials to the instance
• EC2 rotates credentials
• AWS SDKs, AWS CLI, and Tools for PowerShell have support for instance profiles
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Handy AWS CLI Command Options:• --output is a useful way to make your returned information easier to read and
parse.• There are three output formats:
JSON Text Table
{ “Places”: [ { “City”: “Seattle”, “State”: “WA” }, { “City”: “Las Vegas”, “State”: “NV” } [}
PLACES Seattle WA PLACES Las Vegas NV
----------------------------| SomeOperationName |+--------------------------+|| Places |||+------------+-----------+||| City | State |||+------------+-----------+||| Seattle | WA |||| Las Vegas | NV |||+------------+-----------+|
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Handy AWS CLI Command Options:
• In the AWS CLI, a profile is a group of configuration values
• Since profiles store access key information, you can use them to manage access to resources; i.e., you can have separate dev and test profiles
• aws configure --profile <profile name> creates a new profile with that name
• Can be combined with the list, get, and set subcommands– To set the default region setting for the “dev2” profile to eu-west-1:
aws configure set region eu-west-1 --profile dev2– Alternatively, that command can be constructed this way:
aws configure set profile.dev2.region eu-west-1
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Handy AWS CLI Command Options:• --query uses JMESPath to filter response data down to include only
the things you want– JMESPath is a query language for JSON– jmespath.org has web-based tutorials to help learn the language
• If a command generates a response, the response body is parsed into JSON
• If a --query option was used, this data is then filtered based on the query and converted into the requested output format (JSON, text, or table)
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Using• --query '<keyname>[*].<key>' will return all values in keyname
with that key:--query 'Capitals[*].City'
Input JSON Query Result
{ "Capitals": [ { "Nation": "USA", "City": "Washington" }, { "Nation": "Japan", "City": "Tokyo" }, { "Nation": "Brazil", "City": "Brasília" } ],}
[ "Washington", "Tokyo", "Brasília"]
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Using Waiters: Polling Instance States• What's the best way to make sure the AWS CLI waits for an EC2 instance to reach
the running state before performing any other operations?
Script it, right?
#!/bin/bashinstance_id=$(aws ec2 run-instances --image-id ami-12345 \ --query 'Reservations[].Instances[].InstanceID' \ --output text)instance_state=$(aws ec2 describe-instances –instance-ids $instance_id \ --query 'Reservations[].Instances[].State.Name')while [ "$instance_state" != "running" ]do sleep 1 instance_state=$(aws ec2 describe-instances --instance-ids $instance_id \ --query 'Reservations[].Instances[].State.Name')done
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Using Waiters: Polling Instance States• What's the best way to make sure the AWS CLI waits for an EC2 instance to
reach the running state before performing any other operations?
Script it, right?
#!/bin/bashinstance_id=$(aws ec2 run-instances --image-id ami-12345 \ --query 'Reservations[].Instances[].InstanceID' \ --output text)instance_state=$(aws ec2 describe-instances –instance-ids $instance_id \ --query 'Reservations[].Instances[].State.Name')while [ "$instance_state" != "running" ]do sleep 1 instance_state=$(aws ec2 describe-instances --instance-ids $instance_id \ --query 'Reservations[].Instances[].State.Name')done
Problems with this?
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Using Waiters: Polling Instance States• Let's rewrite this script using a waiter:
#!/bin/bashinstance_id=$(aws ec2 run-instances --image-id ami-12345 \ --query 'Reservations[].Instances[].InstanceID' \ --output text)aws ec2 wait instance-running --instance-ids $instance_id
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015
Thank You.This presentation will be loaded to SlideShare the week following the Symposium.
http://www.slideshare.net/AmazonWebServices
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015