enterprise devops at scale with aws | aws public sector summit 2016
TRANSCRIPT
![Page 1: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/1.jpg)
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Jason Shawn, Senior Director of DevOps, EllucianJesse Gigler, DevOps Engineer, Ellucian
June 21, 2016
Enterprise DevOps at Scale with AWS
![Page 2: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/2.jpg)
Who are we?
Jesse Gigler Jason Shawn
DevOps Engineer
Sr. Director of DevOps@jsin@jessegigler
![Page 3: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/3.jpg)
Ellucian: Enabling Student Success
![Page 4: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/4.jpg)
How we define DevOps
People working together with a common set of tools & goals to achieve the best customer experience
![Page 5: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/5.jpg)
Did we mention DevOps is culture?
![Page 6: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/6.jpg)
![Page 7: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/7.jpg)
Our DevOps mission statement
![Page 8: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/8.jpg)
Continuous delivery
Source: https://github.com/red-gate/continuous-delivery-periodic-table
![Page 9: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/9.jpg)
Prior to DevOpsMostly lift-and-shift into AWSVery little test coverageSecurity scans ad-hocSparse CI, no real CD processesNew node deployments took man-weeks
Comparison
Current stateRefactoring into cloud-native appsImproved test coverageSecurity scans in DevOps pipeline1500+ Jenkins jobs running daily New node deployments took ~4 hours automated
![Page 10: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/10.jpg)
Our DevOps toolchain
![Page 11: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/11.jpg)
![Page 12: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/12.jpg)
Jenkins – orchestration layer
• Amazon EC2 Plugin allows Jenkins to spin up slaves dynamically as needed.
• Folder per product team. Product teams restricted to their folder based on AD Group.
• CloudBees Folder Plus Plugin allows us to constrain projects/folders to specific slave pools. Separate slave pools for different instance profiles for assume-role access.
![Page 13: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/13.jpg)
Packer to create immutable AMIs
• Immutable AMIs ensure all tools and components are included in the development lifecycle.
• AMIs can be spun up in a different account to audit software and licensing without direct access to the product environment.
• New AMIs are rolled out by updating the launch configuration in terraform.
![Page 14: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/14.jpg)
Terraform for infrastructure as code
• Remote Amazon S3 state-file allows sharing of resource values across modules and teams.
• Count, split, element design pattern to scale resources.
• Jenkins serves as a middleware wrapper to handle dynamic variables and configuration across AWS accounts.
• Allows us to remain “cloud agnostic”.
![Page 15: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/15.jpg)
Ansible for configuration management layer
• Ansible serves as standard format to write and share server-level automation.
• Playbooks are pushed to S3 from Jenkins, then downloaded from S3 and executed in local mode in user_data to provision the instance.
• PowerShell Desired State Configuration for Windows platforms to adhere to the same Ansible principles (push, pull, local).
![Page 16: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/16.jpg)
Automated tests and scans
• Unit, smoke, and functional tests ensure environment is operating as expected.
• ServerSpec tests to validate infrastructure is configured properly.
• Results proxied through bastion server back to Jenkins for reporting and tracking.
![Page 17: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/17.jpg)
Challenges and pain points
![Page 18: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/18.jpg)
Cross-account deployments
![Page 19: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/19.jpg)
AWS resource tag management
• Lots and lots of teams.
• One central “billing” account.
• Defined “required” tags – but how do you enforce this?
![Page 20: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/20.jpg)
Some select child AWS resources do not support a “tag flow down”.
Requirement: Automate a way to flow the resource tags down from the parents to the children.
Answer: Leverage AWS Lambda to regularly scan the environment and copy tags from parent resources to the appropriate child resources.
Resource tag “flow down”
![Page 21: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/21.jpg)
AWS resource soft limitsEach resource class in each region of all of our accounts has a soft limit but:• How can we monitor our soft limits?• How can we automatically request an increase?• How can we ensure production isn’t affected?
![Page 22: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/22.jpg)
AWS scheduling for cost optimizations
"ScheduledActionUp": { "Type": "AWS::AutoScaling::ScheduledAction", "Condition": "DEV", "Properties": { "AutoScalingGroupName": { "Ref": ”us-east-1-DevApp" }, "MaxSize": "1", "MinSize": "1", "DesiredCapacity": "1", "Recurrence": "0 12 * * 1-5" }},"ScheduledActionDown": { "Type": "AWS::AutoScaling::ScheduledAction", "Condition": "DEV", "Properties": { "AutoScalingGroupName": { "Ref": "us-east-1-DevApp" }, "MaxSize": "0", "MinSize": "0", "DesiredCapacity": "0", "Recurrence": "0 22 * * 2-6" }}
![Page 23: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/23.jpg)
Emerging patterns
![Page 24: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/24.jpg)
Blue/Green deployment
![Page 25: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/25.jpg)
Self-healing CI/CD environment
• Deployment and configuration of Jenkins pipeline is fully automated.
• Can seamlessly deploy to new regions or recreate an existing environment. EBS volume snapshot is taken and reattached after recreating.
• Self-healing - Jenkins will recreate itself based on certain Amazon CloudWatch alarms.
Events
![Page 26: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/26.jpg)
DevSecOps
![Page 27: Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016](https://reader036.vdocuments.us/reader036/viewer/2022062310/58733d551a28abf21b8b5e35/html5/thumbnails/27.jpg)
Thank you!