my cloudconnect 2013 talk: infrastructure automation in the cloud
DESCRIPTION
This session talks about the challenges and complexity both large and small organizations face while managing their IT infrastructure in the cloud, how complex infrastructure can be managed easily through automation, and the various solutions available in the market which with the combination of configuration management and service oriented architecture makes it easy to create an elegant fully automated infrastructure. Some of the main aspects covered in this talk are: - Challenges with managing Infrastructure - What is "Agile" Infrastructure and why is it important - Best practices in setting up an agile infrastructure using automation tools - Continuous integration, delivery and deployment - what is DevOps ?TRANSCRIPT
![Page 1: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/1.jpg)
Innova'on → Execu'on → Solu'on → Delivered
Infrastructure Automation for the Cloud
Chirag Jog, CTO Clogeny [email protected]
Clogeny Confidential
![Page 2: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/2.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
About Me
• Strong Experience in DevOps, Infrastructure Automation
• Built cutting edge products: • Hybrid Cloud Migration Appliances • Enterprise Java PaaS • Multi-cloud single pane of glass Dashboard
• Experience building highly scalable automated cloud applications
• Leading engineering teams
Co-founder/CTO at Clogeny
![Page 3: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/3.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
What is my talk about?
Infrastructure Automation for the Cloud
![Page 4: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/4.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
What is my talk about?
Infrastructure Automation for the Cloud
![Page 5: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/5.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
What is Infrastructure?
App LBs
App Servers
NoSQL
DB slaves
Cache
DB Cache
DBs
![Page 6: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/6.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
What is Infrastructure?
DC1
DC3
DC2
![Page 7: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/7.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Problems in Managing Infrastructure
" Configuration Drift
" Changes causes Outages
" Wall of Confusion – Dev and Ops
" Non Functional Requirements: • Backups, security, monitoring, synchronization,
performance, capacity planning
" Dev, Test and Prod – All Different Environments
" Mystery Machine: On But don’t know why?
" Code Complete != Production Ready
![Page 8: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/8.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Infrastructure is Hard !!
![Page 9: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/9.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Old Problem: Infrastructure is Hard
![Page 10: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/10.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Old Problem: Infrastructure is Hard
" Proprietary Solutions
![Page 11: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/11.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Old Problem: Infrastructure is Hard
" Solve very little of the problem " Enterprise Focused " Large Professional Services Bills " On-premise " Acquired by companies with large consulting
organizations " Google, Amazon, Microsoft built their own tools
but its secret sauce
" Proprietary Solutions
![Page 12: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/12.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Infrastructure is Changing !!
![Page 13: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/13.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Cloud Computing
![Page 14: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/14.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
New Problem: Infrastructure is Changing
" Easier to get .. but harder to manage
" Demand is dynamic
" Developers are crucial to Operations
" Web/Cloud services are proliferating
" Manual Configuration no longer a crutch
" Few tools solve all the problems
" No Shrink Wrap software
![Page 15: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/15.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Infrastructure is Code
" API Driven Abstractions " Infrastructure is now an Application
App LBs
App Servers
NoSQL
DB slaves
Cache
DB Cache
DBs
![Page 16: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/16.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Agile Infrastructure: The Final Frontier for Agile
![Page 17: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/17.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Agile Infrastructure
" Infrastructure is Code: Apply the lessons learnt from Software Development
" Build more Flexible systems and faster feedback
" Beyond Continuous Integration: Continuous Delivery
" 10+ deploys per Day – Flickr, 2009
![Page 18: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/18.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Amazon.com
" Mean time between deployments: 11.6s
" Max # of deploys per hour : 1079
" Mean # of host simultaneously receiving a
deployment: 10,000
" Max # of hosts receiving a deployment: 30,000
![Page 19: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/19.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
What is my talk about?
Infrastructure Automation for the Cloud
![Page 20: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/20.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Agile Infrastructure Techniques
u Configuration Management
u Version Control Everything
u Build from source
u One step deploy
u Monitoring
u Continuous Delivery
u Work Together – Dev and Ops
![Page 21: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/21.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
What is Infrastructure (again) ?
• Nodes • Networking • Files • Directories • Symlinks • Mounts • DB Schema
Infrastructure is a collection of resources:
• Routes • Users • Groups • Tasks • Packages • Software • Services • Configurations • Stuff
![Page 22: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/22.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Infrastructure as Code (really?)
package "ntp" do ! action :install !end !
service "ntpd" do ! action [:enable,:start] !end !
template "/etc/ntpd.conf" do ! source "ntpd.conf.erb" ! owner "root" ! group "root" ! mode 0644 ! action :create ! variables(:time_server => “time.example.com”) ! notifies :restart, “service[ntpd]”!end !
![Page 23: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/23.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Configuration Management
" Infrastructure as Code " Put systems into a known state " Audit and enforce consistency " Manage server lifecycle " Reason about services, instead of systems " Apply dev-test-prod cycle to infrastructure
![Page 24: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/24.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Configuration Management Tools
![Page 25: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/25.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Version Control Everything
" Network Configurations " System Configurations " Application Configurations " Application Code " Database Schema " Documentation " Anything else that matters!
![Page 26: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/26.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Build from Source
" Build Application and Infrastructure from source
" Dev, Test and Prod in sync
" Test from a known state
" Scaling
" Disaster Recovery
![Page 27: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/27.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
One Step Deploy
" One automated process from version control to live services
" Lower the time and cost to deploy
![Page 28: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/28.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Monitoring
" Define ‘Normal’ Conditions " Monitor Baselines, Charts, trends " Monitor, Monitor and Monitor !
Ref: http://consultingblogs.emc.com/blogs/gracemollison/image_thumb_123A5775.png
![Page 29: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/29.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Continuous Integration
" Check in regularly " Create an automated and comprehensive test
suite. " Test New builds " Assert Services are running " Deploy early and often " No manual steps! " Never go home on a broken build " Build a pipeline!
![Page 30: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/30.jpg)
Poll Master and
Compile
Run Unit Tests
Package
Deploy
Run Smoke/
Integration Tests
Release
![Page 31: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/31.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Continuous Delivery Pipeline
SCM Build CI Analyze Deploy Config
Private Clouds
Public Clouds
Repo Infrastructure
Continuous Integration
Continuous Deployment
![Page 32: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/32.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Work Together – Dev and Ops
" Dev and Ops see the same thing, in the same place
" Share Metrics " Share the Repository • Configs in sync with Application Code
" Minimize Surprise " Always ship trunk/master
![Page 33: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/33.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
How often do we see this?
![Page 34: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/34.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Work Together – Dev and Ops
" Devs need to understand the infrastructure " Ops needs to understand the Application " Manage Flows – Avoid Fires " Build a culture – Avoid the blame game " Break the silos and work together! " Collaboration: not only when things go wrong " No Rock star mentality
![Page 35: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/35.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
What is DevOps ?
DevOps: is a software development method that stresses communication, collaboration and integration between Dev and Ops.
![Page 36: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/36.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
What is DevOps ?
DevOps really is: " Processes " Tools " Culture
![Page 37: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/37.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
DevOps
" Processes and Tools • Automated infrastructure • Shared version control • One step build and deploy • Shared metrics
" Culture • Respect • Trust • Healthy attitude about failure • Avoiding Blame • Superhero Culture
![Page 38: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/38.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Conclusion
ü Agile Infrastructure ü Embrace DevOps ü Business Agility ü Improved Time to Market ü Ahead of the competition
![Page 39: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/39.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
Questions ?
![Page 40: My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b642e24a795940648b4649/html5/thumbnails/40.jpg)
http://www.clogeny.com © 2013 Clogeny Technologies
References
" Dev2Ops - http://dev2ops.org
" Flickr’s 10 deploys per day - http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
" Agile Infrastructure - http://www.slideshare.net/littleidea/agile-infrastructure-agile-2009
" DevOps – The war is over - http://www.slideshare.net/jedi4ever/devops-the-war-is-over-if-you-want-it
" DevOps, the future is here – http://www.slideshare.net/KrisBuytaert/devops-the-future-is-here-its-just-not-evenly-distributed-yet
" Jesse Robbins’s DevOps talk - http://www.slideshare.net/jesserobbins/devops-interop-las-vegas-jesse-robbins-opscode