dev ops for software architects

NICTA Copyright 2012 From imagination to impact

DevOps for

Software Architects

Len Bass


Outline

• What is DevOps?

• What are the implications of DevOps practices

on system structure?

– Team practices.

– Deployment practices.

2


What is DevOps?

• “DevOps is a software development method that stresses

communication, collaboration, and integration between software

developers and IT professionals” – Wikipedia

• “DevOps is a new term describing what has also been called “agile

system administration” or “agile operations” joined together with the

values of agile collaboration between development and operations

staff.

Effectively, you can define DevOps as system administrators

participating in an agile development process alongside developers

and using a many of the same agile techniques for their systems

work. “ - http://theagileadmin.com/what-is-devops/

3

http://theagileadmin.com/what-is-devops/







What is DevOps - 2

• DevOps is accompanied by a certain amount of

mysticism.

– “Be Self-Aware

– Be aware of a project’s maturity

– Be aware of others” http://architects.dzone.com/articles/zen-and-art-collaborative

• Similar to the early days of agile.

4

http://architects.dzone.com/articles/zen-and-art-collaborative









What problem is DevOps trying to solve?

• Poor communication between developers and

operations personnel

• Slow release schedule

• Limited capacity of operations staff

• Limited organizational insight into operations

5


My Take on DevOps

• DevOps is a set of practices intended to – Reduce management overhead

– Speed up deployment

– Move some (formerly) IT responsibilities to developers

– Increase communication between developers and operations

– Reduce operations costs

• These practices have implications on – Team size, communication, and responsibilities.

– Deployment

• In turn, there are implications for

– System structure

6


Outline

• What is DevOps?



– Team practices.


7


Teams

• Size/coordination

• Responsibilities

• Processes/tooling

• Structural implications

8


Team Size

• Teams are small. Amazon has a “two pizza”

rule.

• It is easy for small teams to have good internal

coordination.

• Small teams mean

– A lot of teams

– Small scope for each team

– Short delivery times

– Coordination among teams

becomes an issue

9


Coordination among teams

• Asynchronous rather than synchronous

– Allows team members to respond when it is

convenient for them

– Avoids time zone coordination

• Persistent and visible

– E-mail is not generally visible to all of the team

– Chat boards, Wikis, issue trackers, comments in code

are all persistent and visible

– Connect a message to something – issue, feature,

person.

10


Team Responsibilities with respect to Services

• Requirements are sliced thinly both horizontally (breadth

of requirement) and vertically (decomposition of service

into utilities)

• Each service has an owner (a developer)

• Service owner decides when to deploy service to

production. Deployment done with tooling.

• Deployment may involve use of canaries (discussed with

deployment)

• When a service is deployed, service owner examines

monitoring data and decides when/if to roll back.

• Service owner is called if there is a problem.

11


Structural Implications of Team Practices

• Conway’s Law (1968)

– The structure of a system reflects the structure of the

organization that constructed the system.

• DevOps advocates

– Small teams

– Mostly independent teams

• Conway’s Law & many small, mostly

independent teams => Service Oriented

Architecture with

– Many services with small scope of each service

– Loose coupling between services

12


Outline

• What is DevOps?



– Team practices.


13


Deployment Practices

• Deployment can be either an initial deployment

or an upgrade of an existing system.

• We will discuss

– Upgrade

– Continuous deployment

– Roll back

14


Deployment tools

• Have “recipes” for standard configurations

• Moving outside standard configurations may

introduce errors

• Recipes managed by DevOps group

• Configuration specification is version controlled

• Leads to “scripts are code too” mentality

– Development

– Staging

– Deployment

• Goal is to support developer’s ability to

automatically deploy

15


Upgrade

• How many at once?

– One at a time (rolling upgrade)

– Groups at a time (staged upgrade, e.g. canaries)

– All at once (big flip)

• Big flip requires double the number of resources.

Infeasible in environment that uses large

numbers of resources.

• Standard practice is rolling upgrade, possibly

with canaries.

16


Rolling Upgrade Process

17

Update Auto Scaling

Group

Sort Instances

Remove & Deregister

Old Instance from ELB

Confirm Upgrade Spec

Terminate Old

Instance

Wait for ASG to Start

New Instance

Register New Instance

with ELB

• Suppose there are 100s or 1000s of

instances of an application running in the

cloud.

• Then it is too expensive to make a copy

of a new version while leaving the old

version running with all of its instances.

• The solution is to install the new version

on one server at a time – called a rolling

upgrade

• Figure on the right is an example of a

process for a rolling upgrade.

• This process is implemented by a

deployment tool.


Upgrading a service within the service hierarchy

Service level N

Service level N+1

(A)

Service level N+2

Service level N+2

Service level N+1

(B)

Service level N+2

Service level N+1

(B)

Service level N+2

18

Suppose we are doing a rolling upgrade at Service

level N+1

Version B assumes new features from Service level

N+2


Staging Upgrades

• Service level N+2 must be activated before

activating service level N+1.

• Distinction between upgrading and activating.

Upgrades can occur at any time as long as they

are not activated.

• Structural implication

– Upgrades can be activated through software

switches. Could use Zookeeper for coordinating

active versions.

– Activates all of the instances at (essentially) same

time.

19


Upgrades can fail

• Functionally incorrect

• Incompatibility with other portions of the

application or infrastructure

• Resource limitations

• Configuration inconsistency

20


After failure is detected

• Turn off new features in level N+1 and its clients

(level N).

• May need to propagate to top of hierarchy.

• Structural implications • Features are software switchable

• Require all versions to be backward compatible with previous

versions. If new version on level N+1 is switched off, do not need

to worry about level N+2.

21


Canaries

• Canaries are a small number of new versions in order to

perform live testing in a production environment.

• A/B testing is similar to canaries in that A/B testing

compares two proposed new interfaces. Different in that

canaries test new functionality.

22


Canary Issues

• Canaries are a form of live testing. Put a new

version into limited production to test its

correctness.

• Issues

– How long are new versions tested

to determine correctness?

• Period based – for some period of time

• Load based – under some utilization assumptions

• Result based – until some criteria is met

– How are clients of new version chosen and how is

this choice enforced?

23


Continuous Deployment

• Puts deployment decisions into developers hands.

• May mean simultaneous deployment from independent

teams. Some organizations report dozens of

deployments a day.

• Deployment tool must set configuration information e.g.,

in Zookeeper, so that services know what features are

currently active.

24


Structural Implications of Continuous

Deployment

• Packaging

• Maintaining Backward Compatibility

25


Packaging

• Two dimensions

– Flat vs deep service hierarchy

– One service per virtual machine vs many services per

virtual machine

26


Flat vs Deep Service Hierarchy

• Trading off independence of teams and

possibilities for reuse.

• Flat Service Hierarchy

– Limited dependence among services & limited

coordination needed among teams

– Difficult to reuse services

• Deep Service Hierarchy

– Provides possibility for reusing services

– Requires coordination among teams to discover

reuse possibilities.

27


Services per VM Image

28

Service

1

Service

2

VM image

Develop

Develop

Embed

Embed

One service per VM

Service VM image

Develop Embed

Multiple services per VM


One Possible Race Condition with Multiple

Services per VM

29

TIME

Initial State: VM image with Version N of Service 1 and Version N of Service 2

Developer 1

Build new image with VN+1|VN

Begin provisioning

process with new image

Developer 2

Build new image with VN|VN+1

Begin provisioning


without new version of

Service 1

Results in Version N+1 of Service 1 not being

updated until next build of VM image

Could be prevented by VM image build tool


Another Possible Race Condition with Multiple

Services per VM

30

TIME

Initial State: VM image with Version N of Service 1 and Version N of Service 2

Developer 1

Build new image with VN+1|VN

Begin provisioning


overwrites image

created by developer 2

Developer 2

Build new image with VN+1|VN+1

Begin provisioning


Results in Version N+1 of Service 2 not being

updated until next build of VM image

Could be prevented by provisioning tool


Trade offs

• One service per VM

– Message from one service to another must go

through inter VM communication mechanism – adds

latency

– No possibility of race condition

• Multiple Services per VM

– Inter VM communication requirements reduced –

reduces latency

– Adds possibility of race condition caused by

simultaneous deployment

31


Motivating Backward Compatibility

• New version of a service may be introduced at

any time

• Existing clients of that service should not have to

be changed

• Require APIs and DB schemas to be backward

compatible.

32


Achieving Backwards Compatibility

• APIs and DB schemas can be extended but

must always be backward compatible.

• Leads to a translation layer

External APIs (unchanging but with ability to extend

or add new ones)

Translation to internal APIs

Client Client

Internal APIs (changes require changes to

translation layer but do not propagate further)


Summary

• DevOps is a collection of practices that have

implications on system structure.

– These practices can be categorized into

• Team practices

• Deployment practices

• Some structural implications are

– Loosely coupled systems with deep hierarchy of

services

– Version aware

– Backward compatible

– Packaging services per VM

34

dev ops for software architects

Technology

devops devops

services nicta copyright

devops advocates small

operationsnicta copyright

issuenicta copyright

devopsnicta copyright

deployment tools

initial deployment