death to manual deployments
DESCRIPTION
Manual deployments are terrible because they they ask people to be as repeatable as machines. This presentation looks at why manual deployments fail, what automation brings to the table, and how to build on top of automated deployments.TRANSCRIPT
Death to Manual DeploymentsImplementing repeatable, reliable releases
Eric MinickDevOps EvangelistAug, 2013
@EricMinick
Who’s that guy?
Eric Minick
DevOps Evangelist
[email protected] as a
developer, tester and tools guy
Helping clients implement build, deploy & release tooling for the past 10 years
Confession: I hate manual deployments
They’re slow
They aren’t consistent across environments
You fail in production
You don’t know who deployed what, where or when
Why do people like manual deployments?
I feel more in control from the command line
It’s how I’ve always done it
I know what I did and see the output of the commands I ran
When something goes wrong, I can be the hero and fix it
Scripts to the rescue!?
I don’t have a manual deployment: I have scripts!–How many scripts?
–What order?
–Who has permissions to run them?
–Are they the same across environments?
–What else do you need to do?
Scripts are only part of a fully automated deployment.
I love automated deployments
They’re NOT slow
They ARE consistent across environments
You DON’T fail in production
You DO know who deployed what, where or when
Automated deployments provide benefits
Provide a platform you can build on– continuous deploys, smoke tests, other goodies
Empower people while increasing control– stop waiting on other people
– stop being harassed by other people
Better, Faster, Cheaper (Pick any 3)
The Program
Today’s Manual Deployments
Elements of an Automated Solution
Best Practices with Automated Deployments
The Program
Today’s Manual Deployments
Elements of an Automated Solution
Best Practices with Automated Deployments
… you might have a manual deployment
Hand-offs are through email (or file shares)
Developers, QA, and Ops deploy differently
Tweaks communicated on an as-broken basis
Deployments steps are a huge document
The term “deployment weekend” is used
Developers have production passwords
Hand-offs are through email (or file shares)
They’re slow
They aren’t consistent across environments
You fail in production
You don’t know who deployed what, where or when
Developers, QA, and Ops deploy differently
They’re slow
They aren’t consistent across environments
You fail in production
You don’t know who deployed what, where or when
Tweaks communicated as-broken
They’re slow
They aren’t consistent across environments
You fail in production
You don’t know who deployed what, where or whenThe main page won’t
load?Oh… Right, you needed
to add another data source.
Deployments steps are a huge document
They’re slow
They aren’t consistent across environments
You fail in production
You don’t know who deployed what, where or when
The term “deployment weekend” is used
They’re slow
They aren’t consistent across environments
You fail in production
You don’t know who deployed what, where or when
That’s ok, I don’t really
like my kids
anyway
Developers have production passwords
They’re slow
They aren’t consistent across environments
You fail in production
You don’t know who deployed what, where or when
If only people were more disciplined…
Deployment docs–could capture everything
–always be updated
–could be followed them, perfectly, every time.
Audit spreadsheets–could capture everything
–could update them
Image from: http://www.ultimatebasictraining.com/blog/2010/02/what-area-drill-sergeants-like/
People aren’t machines
People: good at using creative judgment
Machines: good at consistency
Automated deployments leverage the strengths of people and machines.
Alistar Cockburn, “Characterizing People as Non-Linear, First-Order Components in Software Development”
DevOps isn’t natural
Conway’s Law: organizations are constrained to produce solutions that copy the structure of their organization
DevOps isn’t natural
Conway’s Law: organizations are constrained to produce solutions that copy the structure of their organization
Ops wants to maintain control
–doesn’t care about DEV, can’t keep up with the frequent changes
Development doesn’t want responsibility for production deployments
QA lacks influence in DEV & PROD
The Program
Today’s Manual Deployments
Elements of an Automated Solution
Best Practices with Automated Deployments
Elements of an automated solution
Automated process
Coordinating tiers
Environment definitions
Artifact repository
Access control
Audit trail
Automated process
Status QuoDon’t exist
Incomplete
Multiple order-dependent scripts
Multiple versions per environment (or machine)
Uncontrolled
What we needSingle script reused across
environments
Controlled access to deployment steps
Auto record what happened
Fast, consistent and repeatable
Coordinating Tiers / Components
Status QuoBased on oral history
In a Release Manager’s spreadsheet
Total plan based on documented changes for each release
–Redesign every time
What we needA model for whole systems
Standard deployment based on changes to all pieces and parts
Actual deployment runs the needed subset.
Move all tiers, services, config etc together
Environment definitions
Status Quo Configured manually
Details embedded in binaries, scripts, property files, ini files
Environment information either uncontrolled or in source control
What we need Environments a first-level
entity
Properties controlled at the environment level
Secure storage of sensitive values (passwords)
Per-environment configuration and passwords
Artifact repository
Status Quo Is email a repository
File share on the network
Source control for binary control
Uncontrolled
Crude control over what to keep
What we need Definitive Software Library
Consistent set of artifacts across environments
Tamper-proof artifacts
Automated retention policies based on deployments
Know what you’re getting
Access control
Status QuoControlled by who can log
onto the machine
Access tends to grow over time in response to crisis
Access control distributed across machines
What we needRole-based access control
Access control by environment
Single source of access control
LDAP / Active Directory authentication (or authorization)
Who can deploy to which environment
Audit trail
Status QuoWhat audit trail?
I have the spreadsheet
Our Change Management System has what we think we did
Audit trail starts at PROD
What we needWin at Clue:
–Who
–What
–Where
–When / How (better than Clue!)
Require no additional work
End-to-end traceability
Know what happened
Solution Benefits
We have confidence that our production deployments will succeed
QA can take new builds faster
DEV deployments fast enough to keep up with our rate of change
We have an audit trail from day one
We have separation of duties
The Program
Today’s Manual Deployments
Elements of an Automated Solution
Best Practices with Automated Deployments
Automated Deployments Best Practices
Deferred deployments
Self-service deployments
Automatic deployment
Automatic smoke-test
Automated rollbacks
Continuous production deployment
Deferred deployments
Story: I want to do my deployment at 11 pm on Friday after my users are offline.
Solution: Schedule deployment for desired time, go home. Wait for notification of success or failure.
Self-service deployments
Story: QA wants to be able to take a new build when they need one.
Solution: Use per-environment access control to allow the QA people to deploy on-demand, but only to the test environment.
Automatic deployment
Story: Want to keep the development test server in sync with the latest changes in source control.
Solution: After building, automatically deploy to the development server.
Automatic smoke-test
Story: I want to know if something breaks immediately.
Solution: After deploying, automatically trigger a suite of automated tests and report the results.
Automated rollback
Story: If my deployment fails I want to restore the previous build.
Solution: When a deployment step fails, trigger the restoration of the last successfully deployed build.
Continuous production deployments
Story: When a change is committed it should be deployed to production if it doesn’t break anything.
Solution: After deployment to test environment run automated test suites. If they all pass, trigger a deployment to production, then test that production deploy was successful. Rollback on failure.
Summary
Manual deployments are slow and broken because they require people to act like machines
Computers make good machines
Automated deployments provide a solid foundation for further improvement
References
http://urbancode.com/resources
Deployment Automation Basics
Lean Build & Deployment Automation
ITIL Release Management and Automation
Blogs.urbancode.com
Twitter.com/UrbanCode
Deployment and Release Products
IBM UrbanCode Deploy
–Application Release Automation
IBM UrbanCode Release
–Release management planning and
release weekend execution
41
Slideshare.net/UrbancodeUrbancode.com/resources/Blogs.urbancode.com/[email protected] @EricMinick