(web307) scalable site management using aws opsworks | aws re:invent 2014
DESCRIPTION
Migrating from a hosted environment to AWS is a good opportunity to streamline deployment and site operations. This session shows how FillZ used AWS OpsWorks with other tools to automate site operations and deliver a highly available site that is used by large numbers of customers. Through code and examples, this session shows you how to automate deployments across an entire fleet, configure a patching strategy, use common tools to create useful alarms and monitor system performance, and employ security best-practices in AWS.TRANSCRIPT
![Page 1: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/1.jpg)
November 13, 2014 | Las Vegas, NV
Jonathan Quail / Cliff McCollum, FillZ.com
![Page 2: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/2.jpg)
![Page 3: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/3.jpg)
![Page 4: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/4.jpg)
![Page 5: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/5.jpg)
Story background…
![Page 6: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/6.jpg)
![Page 7: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/7.jpg)
https://www.flickr.com/photos/stevecorey/12844122413
![Page 8: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/8.jpg)
![Page 9: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/9.jpg)
Debian CentOS Amazon
Linux
https://www.flickr.com/photos/bdesham/2432400623
![Page 10: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/10.jpg)
![Page 11: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/11.jpg)
![Page 12: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/12.jpg)
https://www.flickr.com/photos/beigephotos/5334612
https://www.flickr.com/photos/colorhome/4033895230
![Page 13: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/13.jpg)
![Page 14: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/14.jpg)
![Page 15: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/15.jpg)
![Page 16: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/16.jpg)
![Page 17: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/17.jpg)
Monitoring
Provisioning
Logging
Deployment Modeling
Configuration management Auto Scaling Access management
Resource management
![Page 18: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/18.jpg)
![Page 19: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/19.jpg)
![Page 20: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/20.jpg)
github.com/aws/opsworks-cookbooks
supermarket.getchef.com
![Page 21: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/21.jpg)
![Page 22: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/22.jpg)
Hot
tip!
![Page 23: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/23.jpg)
Hot
tip!
![Page 24: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/24.jpg)
• Application servers
• Monitoring servers
• Graphite/dashboards
• Splunk cluster
![Page 25: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/25.jpg)
![Page 26: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/26.jpg)
![Page 27: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/27.jpg)
![Page 28: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/28.jpg)
![Page 29: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/29.jpg)
![Page 30: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/30.jpg)
![Page 31: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/31.jpg)
![Page 32: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/32.jpg)
– fillz-app::setup, fillz-app::configure, fillz-app::deploy
– fillz-base::setup, fillz-base::configure
Setup Configure Deploy Undeploy Shutdown
![Page 33: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/33.jpg)
![Page 34: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/34.jpg)
![Page 35: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/35.jpg)
![Page 36: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/36.jpg)
$ easy_deploy.py deploy –-application=myapp instances –-hosts=host1,host2 \
-–stack-name=my-test-stack --comment=“Deploying myapp to host1/host2”
$ easy_deploy.py deploy –-application=myapp all –-exclude-hosts=host1,host2 \
-–layer-name=backend-servers -–stack-name=my-test-stack \
--comment=“Deploying myapp to rest of the stack”
$ easy_deploy.py deploy –-application=myapp rolling –-layer-name=apiservers \
-–stack-name=my-api-stack --comment=“Deploying myapp behind ELB servers”
![Page 37: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/37.jpg)
Code is available from the Github repo
![Page 38: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/38.jpg)
![Page 39: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/39.jpg)
By permission of Leigh Rubin and Creators Syndicate, Inc.
![Page 40: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/40.jpg)
![Page 41: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/41.jpg)
![Page 42: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/42.jpg)
all_hosts = {}
all_hostgroups = {}
# Gets all (online only) instances in OpsWorks stack
all_instances = search(:node, 'role:*')
![Page 43: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/43.jpg)
all_instances.each do |instance|
node_hostgroups = {}
# add a hostgroup for each layer the instance is in
instance['opsworks']['layers'].each do |layer_id, layer_info|
node_hostgroups[layer_id] = layer_info['name']
end
# add a hostgroup for each availability zone
node_hostgroups[instance['availability_zone']] =
instance['availability_zone']
all_hosts[instance['hostname']] = {
:hostgroups => node_hostgroups,
:private_ip => instance['private_ip']
}
all_hostgroups = all_hostgroups.merge(node_hostgroups)
end
![Page 44: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/44.jpg)
template "/etc/nagios/conf.d/hostgroups.cfg" do
source 'hostgroups.cfg.erb'
owner 'nagios'
group 'nagios'
mode 00640
variables(
:hostgroups => all_hostgroups
)
notifies :reload, 'service[nagios]'
backup 0
end
![Page 45: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/45.jpg)
template "/etc/nagios/conf.d/hosts.cfg" do
source 'hosts.cfg.erb'
owner 'nagios'
group 'nagios'
mode 00640
variables(
:hosts => all_hosts
)
notifies :reload, 'service[nagios]'
backup 0
end
![Page 46: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/46.jpg)
![Page 47: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/47.jpg)
![Page 48: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/48.jpg)
![Page 49: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/49.jpg)
![Page 50: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/50.jpg)
![Page 51: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/51.jpg)
![Page 52: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/52.jpg)
Hot
tip!
![Page 53: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/53.jpg)
<infomercial>
</infomercial>
Wait…Remember that deployment tool? If you call in
the next 20 minutes…it does patching too.
![Page 54: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/54.jpg)
No “4 easy payments of $29.95” this part is open source too
![Page 55: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/55.jpg)
$ easy_deploy.py update --amazon-linux-release=2014.09 --allow-reboots \
instances –-hosts=host1,host2 -–stack-name=my-test-stack \
--comment=“Upgrading host1/host2 to 2014.09 release”
$ easy_deploy.py update all –-exclude-hosts=host1 \
-–layer-name=backend-servers -–stack-name=my-test-stack \
--comment=“Patching layer for ++openssl vulnerability”
$ easy_deploy.py update --allow-reboots rolling \
-–layer-name=apiservers –-stack-name=my-api-stack \
--comment=“Patching api behind ELB servers and reboot for kernel”
![Page 56: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/56.jpg)
![Page 57: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/57.jpg)
![Page 58: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/58.jpg)
“<stack> <name>”
![Page 59: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/59.jpg)
“ bastion-dev fillz-graphite”
“ bastion-dev localhost:8000:fillz-graphite:80”
![Page 60: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/60.jpg)
![Page 61: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/61.jpg)
![Page 62: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/62.jpg)
https://www.flickr.com/photos/wfryer/716965391
![Page 63: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/63.jpg)
![Page 64: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/64.jpg)
![Page 65: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/65.jpg)
![Page 66: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/66.jpg)
/var/log/aws/opsworks/opsworks-agent.log
/var/log/aws/opsworks/opsworks-agent.keep_alive.log
/var/log/aws/opsworks/opsworks-agent.process_command.log
http://blogs.aws.amazon.com/application-
management/post/TxTX72HFKVS9W9/Using-Amazon-CloudWatch-
Logs-with-AWS-OpsWorks
![Page 67: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/67.jpg)
![Page 68: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/68.jpg)
BEFORE AFTER
Time to deploy a new
server2 days + 2 hours 25 minutes
Time to deploy a new
software release2 hours 2 minutes
After-hours alerts 6 / week < 1 / week
![Page 69: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/69.jpg)
BEFORE AFTER
Scheduled downtime per
month5 hours 0 minutes
Software deployment rate 2 / month 2 / week
![Page 70: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/70.jpg)
![Page 71: (WEB307) Scalable Site Management Using AWS OpsWorks | AWS re:Invent 2014](https://reader031.vdocuments.us/reader031/viewer/2022020105/559445ff1a28ab02738b45f7/html5/thumbnails/71.jpg)
http://github.com/aws/opsworks-cookbooks
http://supermarket.getchef.com
http://blogs.aws.amazon.com/application-
management/post/TxZX2UI4YSBW3T/Customizing-AWS-OpsWorks-with-Attributes
http://blogs.aws.amazon.com/application-
management/post/TxTX72HFKVS9W9/Using-Amazon-CloudWatch-Logs-with-AWS-
OpsWorks
https://github.com/awslabs/reinvent2014-scalable-site-management