death to manual deployments

Death to Manual DeploymentsImplementing repeatable, reliable releases

Eric MinickDevOps EvangelistAug, 2013

@EricMinick

Who’s that guy?

Eric Minick

DevOps Evangelist

eminick@us.ibm.comBackground as a

developer, tester and tools guy

Helping clients implement build, deploy & release tooling for the past 10 years

Confession: I hate manual deployments

They’re slow

They aren’t consistent across environments

You fail in production

You don’t know who deployed what, where or when

Why do people like manual deployments?

I feel more in control from the command line

It’s how I’ve always done it

I know what I did and see the output of the commands I ran

When something goes wrong, I can be the hero and fix it

Scripts to the rescue!?

I don’t have a manual deployment: I have scripts!–How many scripts?

–What order?

–Who has permissions to run them?

–Are they the same across environments?

–What else do you need to do?

Scripts are only part of a fully automated deployment.

I love automated deployments

They’re NOT slow

They ARE consistent across environments

You DON’T fail in production

You DO know who deployed what, where or when

Automated deployments provide benefits

Provide a platform you can build on– continuous deploys, smoke tests, other goodies

Empower people while increasing control– stop waiting on other people

– stop being harassed by other people

Better, Faster, Cheaper (Pick any 3)

The Program

Today’s Manual Deployments

Elements of an Automated Solution

Best Practices with Automated Deployments

The Program

Today’s Manual Deployments

Elements of an Automated Solution

Best Practices with Automated Deployments

… you might have a manual deployment

Hand-offs are through email (or file shares)

Developers, QA, and Ops deploy differently

Tweaks communicated on an as-broken basis

Deployments steps are a huge document

The term “deployment weekend” is used

Developers have production passwords

Hand-offs are through email (or file shares)

They’re slow

They aren’t consistent across environments

You fail in production

You don’t know who deployed what, where or when

Developers, QA, and Ops deploy differently

They’re slow

They aren’t consistent across environments

You fail in production

You don’t know who deployed what, where or when

Tweaks communicated as-broken

They’re slow

They aren’t consistent across environments

You fail in production

You don’t know who deployed what, where or whenThe main page won’t

load?Oh… Right, you needed

to add another data source.

Deployments steps are a huge document

They’re slow

They aren’t consistent across environments

You fail in production

You don’t know who deployed what, where or when

The term “deployment weekend” is used

They’re slow

They aren’t consistent across environments

You fail in production

You don’t know who deployed what, where or when

That’s ok, I don’t really

like my kids

anyway

Developers have production passwords

They’re slow

They aren’t consistent across environments

You fail in production

You don’t know who deployed what, where or when

If only people were more disciplined…

Deployment docs–could capture everything

–always be updated

–could be followed them, perfectly, every time.

Audit spreadsheets–could capture everything

–could update them

Image from: http://www.ultimatebasictraining.com/blog/2010/02/what-area-drill-sergeants-like/

People aren’t machines

People: good at using creative judgment

Machines: good at consistency

Automated deployments leverage the strengths of people and machines.

Alistar Cockburn, “Characterizing People as Non-Linear, First-Order Components in Software Development”

DevOps isn’t natural

Conway’s Law: organizations are constrained to produce solutions that copy the structure of their organization

DevOps isn’t natural

Conway’s Law: organizations are constrained to produce solutions that copy the structure of their organization

Ops wants to maintain control

–doesn’t care about DEV, can’t keep up with the frequent changes

Development doesn’t want responsibility for production deployments

QA lacks influence in DEV & PROD

The Program

Today’s Manual Deployments

Elements of an Automated Solution

Best Practices with Automated Deployments

Elements of an automated solution

Automated process

Coordinating tiers

Environment definitions

Artifact repository

Access control

Audit trail

Automated process

Status QuoDon’t exist

Incomplete

Multiple order-dependent scripts

Multiple versions per environment (or machine)

Uncontrolled

What we needSingle script reused across

environments

Controlled access to deployment steps

Auto record what happened

Fast, consistent and repeatable

Coordinating Tiers / Components

Status QuoBased on oral history

In a Release Manager’s spreadsheet

Total plan based on documented changes for each release

–Redesign every time

What we needA model for whole systems

Standard deployment based on changes to all pieces and parts

Actual deployment runs the needed subset.

Move all tiers, services, config etc together

Environment definitions

Status Quo Configured manually

Details embedded in binaries, scripts, property files, ini files

Environment information either uncontrolled or in source control

What we need Environments a first-level

entity

Properties controlled at the environment level

Secure storage of sensitive values (passwords)

Per-environment configuration and passwords

Artifact repository

Status Quo Is email a repository

File share on the network

Source control for binary control

Uncontrolled

Crude control over what to keep

What we need Definitive Software Library

Consistent set of artifacts across environments

Tamper-proof artifacts

Automated retention policies based on deployments

Know what you’re getting

Access control

Status QuoControlled by who can log

onto the machine

Access tends to grow over time in response to crisis

Access control distributed across machines

What we needRole-based access control

Access control by environment

Single source of access control

LDAP / Active Directory authentication (or authorization)

Who can deploy to which environment

Audit trail

Status QuoWhat audit trail?

I have the spreadsheet

Our Change Management System has what we think we did

Audit trail starts at PROD

What we needWin at Clue:

–Who

–What

–Where

–When / How (better than Clue!)

Require no additional work

End-to-end traceability

Know what happened

Solution Benefits

We have confidence that our production deployments will succeed

QA can take new builds faster

DEV deployments fast enough to keep up with our rate of change

We have an audit trail from day one

We have separation of duties

The Program

Today’s Manual Deployments

Elements of an Automated Solution

Best Practices with Automated Deployments

Automated Deployments Best Practices

Deferred deployments

Self-service deployments

Automatic deployment

Automatic smoke-test

Automated rollbacks

Continuous production deployment

Deferred deployments

Story: I want to do my deployment at 11 pm on Friday after my users are offline.

Solution: Schedule deployment for desired time, go home. Wait for notification of success or failure.

Self-service deployments

Story: QA wants to be able to take a new build when they need one.

Solution: Use per-environment access control to allow the QA people to deploy on-demand, but only to the test environment.

Automatic deployment

Story: Want to keep the development test server in sync with the latest changes in source control.

Solution: After building, automatically deploy to the development server.

Automatic smoke-test

Story: I want to know if something breaks immediately.

Solution: After deploying, automatically trigger a suite of automated tests and report the results.

Automated rollback

Story: If my deployment fails I want to restore the previous build.

Solution: When a deployment step fails, trigger the restoration of the last successfully deployed build.

Continuous production deployments

Story: When a change is committed it should be deployed to production if it doesn’t break anything.

Solution: After deployment to test environment run automated test suites. If they all pass, trigger a deployment to production, then test that production deploy was successful. Rollback on failure.

Summary

Manual deployments are slow and broken because they require people to act like machines

Computers make good machines

Automated deployments provide a solid foundation for further improvement

References

http://urbancode.com/resources

Deployment Automation Basics

Lean Build & Deployment Automation

ITIL Release Management and Automation

Blogs.urbancode.com

Twitter.com/UrbanCode

Deployment and Release Products

IBM UrbanCode Deploy

–Application Release Automation

IBM UrbanCode Release

–Release management planning and

release weekend execution

41

Slideshare.net/UrbancodeUrbancode.com/resources/Blogs.urbancode.com/eMinick@us.ibm.com @EricMinick